You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ko...@apache.org on 2022/10/27 09:14:07 UTC
[arrow-site] branch asf-site updated: Ballista 0.9.0 User Guide (#262)
This is an automated email from the ASF dual-hosted git repository.
kou pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 459e750842 Ballista 0.9.0 User Guide (#262)
459e750842 is described below
commit 459e75084206c8efb8938f6c32a6f53cf5694031
Author: Andy Grove <an...@gmail.com>
AuthorDate: Thu Oct 27 03:14:02 2022 -0600
Ballista 0.9.0 User Guide (#262)
---
ballista/_images/ballista-web-ui.png | Bin 0 -> 142890 bytes
ballista/_images/example-query-plan.png | Bin 0 -> 192455 bytes
ballista/_sources/community/communication.md.txt | 84 ----
ballista/_sources/index.rst.txt | 57 ---
ballista/_sources/user-guide/cli.md.txt | 139 ------
ballista/_sources/user-guide/configs.md.txt | 59 ---
.../user-guide/deployment/cargo-install.md.txt | 50 --
.../user-guide/deployment/configuration.md.txt | 34 --
.../user-guide/deployment/docker-compose.md.txt | 97 ----
.../_sources/user-guide/deployment/docker.md.txt | 106 -----
.../_sources/user-guide/deployment/index.rst.txt | 28 --
.../user-guide/deployment/kubernetes.md.txt | 308 ------------
ballista/_sources/user-guide/faq.md.txt | 31 --
ballista/_sources/user-guide/introduction.md.txt | 50 --
ballista/_sources/user-guide/python.md.txt | 90 ----
ballista/_sources/user-guide/rust.md.txt | 99 ----
ballista/_sources/user-guide/tuning-guide.md.txt | 71 ---
ballista/community/communication.html | 54 ++-
ballista/genindex.html | 54 ++-
ballista/index.html | 81 +++-
ballista/objects.inv | Bin 614 -> 648 bytes
ballista/search.html | 54 ++-
ballista/searchindex.js | 2 +-
ballista/user-guide/cli.html | 155 +++---
ballista/user-guide/configs.html | 81 +++-
ballista/user-guide/deployment/cargo-install.html | 67 ++-
ballista/user-guide/deployment/docker-compose.html | 133 +++---
ballista/user-guide/deployment/docker.html | 141 +++---
ballista/user-guide/deployment/index.html | 84 ++--
ballista/user-guide/deployment/kubernetes.html | 149 +++---
ballista/user-guide/faq.html | 56 ++-
ballista/user-guide/flightsql.html | 518 +++++++++++++++++++++
ballista/user-guide/introduction.html | 85 ++--
ballista/user-guide/python.html | 140 ++++--
ballista/user-guide/rust.html | 66 ++-
ballista/user-guide/{faq.html => scheduler.html} | 131 ++++--
ballista/user-guide/tuning-guide.html | 76 ++-
37 files changed, 1592 insertions(+), 1838 deletions(-)
diff --git a/ballista/_images/ballista-web-ui.png b/ballista/_images/ballista-web-ui.png
new file mode 100644
index 0000000000..389cd1cb29
Binary files /dev/null and b/ballista/_images/ballista-web-ui.png differ
diff --git a/ballista/_images/example-query-plan.png b/ballista/_images/example-query-plan.png
new file mode 100644
index 0000000000..6ead4e62bc
Binary files /dev/null and b/ballista/_images/example-query-plan.png differ
diff --git a/ballista/_sources/community/communication.md.txt b/ballista/_sources/community/communication.md.txt
deleted file mode 100644
index d7fe3e03ee..0000000000
--- a/ballista/_sources/community/communication.md.txt
+++ /dev/null
@@ -1,84 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Communication
-
-We welcome participation from everyone and encourage you to join us, ask
-questions, and get involved.
-
-All participation in the Apache Arrow DataFusion project is governed by the
-Apache Software Foundation's [code of
-conduct](https://www.apache.org/foundation/policies/conduct.html).
-
-## Questions?
-
-### Mailing list
-
-We use arrow.apache.org's `dev@` mailing list for project management, release
-coorindation and design discussions
-([subscribe](mailto:dev-subscribe@arrow.apache.org),
-[unsubscribe](mailto:dev-unsubscribe@arrow.apache.org),
-[archives](https://lists.apache.org/list.html?dev@arrow.apache.org)).
-
-When emailing the dev list, please make sure to prefix the subject line with a
-`[DataFusion]` tag, e.g. `"[DataFusion] New API for remote data sources"`, so
-that the appropriate people in the Apache Arrow community notice the message.
-
-### Slack and Discord
-
-We use the official [ASF](https://s.apache.org/slack-invite) Slack workspace
-for informal discussions and coordination. This is a great place to meet other
-contributors and get guidance on where to contribute. Join us in the
-`#arrow-rust` channel.
-
-We also have a backup Arrow Rust Discord
-server ([invite link](https://discord.gg/Qw5gKqHxUM)) in case you are not able
-to join the Slack workspace. If you need an invite to the Slack workspace, you
-can also ask for one in our Discord server.
-
-### Sync up video calls
-
-We have biweekly sync calls every other Thursdays at both 04:00 UTC
-and 16:00 UTC (starting September 30, 2021) depending on if there are
-items on the agenda to discuss and someone being willing to host.
-
-Please see the [agenda](https://docs.google.com/document/d/1atCVnoff5SR4eM4Lwf2M1BBJTY6g3_HUNR6qswYJW_U/edit)
-for the video call link, add topics and to see what others plan to discuss.
-
-The goals of these calls are:
-
-1. Help "put a face to the name" of some of other contributors we are working with
-2. Discuss / synchronize on the goals and major initiatives from different stakeholders to identify areas where more alignment is needed
-
-No decisions are made on the call and anything of substance will be discussed on this mailing list or in github issues / google docs.
-
-We will send a summary of all sync ups to the dev@arrow.apache.org mailing list.
-
-## Contributing
-
-Our source code is hosted on
-[GitHub](https://github.com/apache/arrow-datafusion). More information on contributing is in
-the [Contribution Guide](https://github.com/apache/arrow-datafusion/blob/master/CONTRIBUTING.md)
-, and we have curated a [good-first-issue]
-(https://github.com/apache/arrow-datafusion/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)
-list to help you get started. You can find datafusion's major designs in docs/source/specification.
-
-We use GitHub issues for maintaining a queue of development work and as the
-public record. We often use Google docs, Github issues and pull requests for
-quick and small design discussions. For major design change proposals, we encourage you to write a rfc.
diff --git a/ballista/_sources/index.rst.txt b/ballista/_sources/index.rst.txt
deleted file mode 100644
index a9343b1da9..0000000000
--- a/ballista/_sources/index.rst.txt
+++ /dev/null
@@ -1,57 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-=====================
-Apache Arrow Ballista
-=====================
-
-Table of content
-================
-
-
-.. _toc.guide:
-
-.. toctree::
- :maxdepth: 1
- :caption: User Guide
-
- user-guide/introduction
- user-guide/deployment/index
- user-guide/python
- user-guide/rust
- user-guide/cli
- user-guide/configs
- user-guide/tuning-guide
- user-guide/faq
-
-.. _toc.source:
-
-.. toctree::
- :maxdepth: 1
- :caption: Source Code
-
- Ballista <https://github.com/apache/arrow-ballista/>
-
-.. _toc.community:
-
-.. toctree::
- :maxdepth: 1
- :caption: Community
-
- community/communication
- Issue tracker <https://github.com/apache/arrow-ballista/issues>
- Code of conduct <https://github.com/apache/arrow-ballista/blob/master/CODE_OF_CONDUCT.md>
diff --git a/ballista/_sources/user-guide/cli.md.txt b/ballista/_sources/user-guide/cli.md.txt
deleted file mode 100644
index 2ec9c8d499..0000000000
--- a/ballista/_sources/user-guide/cli.md.txt
+++ /dev/null
@@ -1,139 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# DataFusion Command-line Interface
-
-The DataFusion CLI allows SQL queries to be executed by an in-process DataFusion context, or by a distributed
-Ballista context.
-
-```
-USAGE:
- datafusion-cli [FLAGS] [OPTIONS]
-
-FLAGS:
- -h, --help Prints help information
- -q, --quiet Reduce printing other than the results and work quietly
- -V, --version Prints version information
-
-OPTIONS:
- -c, --batch-size <batch-size> The batch size of each query, or use DataFusion default
- -p, --data-path <data-path> Path to your data, default to current directory
- -f, --file <file>... Execute commands from file(s), then exit
- --format <format> Output format [default: table] [possible values: csv, tsv, table, json, ndjson]
- --host <host> Ballista scheduler host
- --port <port> Ballista scheduler port
-```
-
-## Example
-
-Create a CSV file to query.
-
-```bash
-$ echo "1,2" > data.csv
-```
-
-```bash
-$ datafusion-cli
-
-DataFusion CLI v8.0.0
-
-> CREATE EXTERNAL TABLE foo (a INT, b INT) STORED AS CSV LOCATION 'data.csv';
-0 rows in set. Query took 0.001 seconds.
-
-> SELECT * FROM foo;
-+---+---+
-| a | b |
-+---+---+
-| 1 | 2 |
-+---+---+
-1 row in set. Query took 0.017 seconds.
-```
-
-## DataFusion-Cli
-
-Build the `datafusion-cli` without the feature of ballista.
-
-```bash
-cd arrow-datafusion/datafusion-cli
-cargo build
-```
-
-## Ballista
-
-The DataFusion CLI can also connect to a Ballista scheduler for query execution.
-
-Before you use the `datafusion-cli` to connect the Ballista scheduler, you should build/compile
-the `datafusion-cli` with feature of "ballista" first.
-
-```bash
-cd arrow-datafusion/datafusion-cli
-cargo build --features ballista
-```
-
-Then, you can connect the Ballista by below command.
-
-```bash
-datafusion-cli --host localhost --port 50050
-```
-
-## Cli commands
-
-Available commands inside DataFusion CLI are:
-
-- Quit
-
-```bash
-> \q
-```
-
-- Help
-
-```bash
-> \?
-```
-
-- ListTables
-
-```bash
-> \d
-```
-
-- DescribeTable
-
-```bash
-> \d table_name
-```
-
-- QuietMode
-
-```
-> \quiet [true|false]
-```
-
-- list function
-
-```bash
-> \h
-```
-
-- Search and describe function
-
-```bash
-> \h function_table
-```
diff --git a/ballista/_sources/user-guide/configs.md.txt b/ballista/_sources/user-guide/configs.md.txt
deleted file mode 100644
index dc7fe7da70..0000000000
--- a/ballista/_sources/user-guide/configs.md.txt
+++ /dev/null
@@ -1,59 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Configuration
-
-Ballista has a number of configuration settings that can be specified when creating a BallistaContext.
-
-_Example: Specifying configuration options when creating a context_
-
-```rust
-let config = BallistaConfig::builder()
- .set("ballista.shuffle.partitions", "200")
- .set("ballista.batch.size", "16384")
- .build()?;
-
-let ctx = BallistaContext::remote("localhost", 50050, &config).await?;
-```
-
-## Ballista Configuration Settings
-
-| key | type | default | description |
-| --------------------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| ballista.shuffle.partitions | UInt16 | 16 | Sets the default number of partitions to create when repartitioning query stages. |
-| ballista.batch.size | UInt16 | 8192 | Sets the default batch size. |
-| ballista.repartition.joins | Boolean | true | When set to true, Ballista will repartition data using the join keys to execute joins in parallel using the provided `ballista.shuffle.partitions` level. |
-| ballista.repartition.aggregations | Boolean | true | When set to true, Ballista will repartition data using the aggregate keys to execute aggregates in parallel using the provided `ballista.shuffle.partitions` level. |
-| ballista.repartition.windows | Boolean | true | When set to true, Ballista will repartition data using the partition keys to execute window functions in parallel using the provided `ballista.shuffle.partitions` level. |
-| ballista.parquet.pruning | Boolean | true | Determines whether Parquet pruning should be enabled or not. |
-| ballista.with_information_schema | Boolean | true | Determines whether the `information_schema` should be created in the context. This is necessary for supporting DDL commands such as `SHOW TABLES`. |
-| ballista.plugin_dir | Boolean | true | Specified a path for plugin files. Dynamic library files in this directory will be loaded when scheduler state initializes. |
-
-## DataFusion Configuration Settings
-
-In addition to Ballista-specific configuration settings, the following DataFusion settings can also be specified.
-
-| key | type | default | description |
-| ----------------------------------------------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| datafusion.execution.coalesce_batches | Boolean | true | When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting 'datafusion.execution.coalesce_target_batch_size'. |
-| datafusion.execution.coalesce_target_batch_size | UInt64 | 4096 | Target batch size when coalescing batches. Uses in conjunction with the configuration setting 'datafusion.execution.coalesce_batches'. |
-| datafusion.explain.logical_plan_only | Boolean | false | When set to true, the explain statement will only print logical plans. |
-| datafusion.explain.physical_plan_only | Boolean | false | When set to true, the explain statement will only print physical plans. |
-| datafusion.optimizer.filter_null_join_keys | Boolean | false | When set to true, the optimizer will insert filters before a join between a nullable and non-nullable column to filter out nulls on the nullable side. This filter can add additional overhead when the file format does not fully support predicate push down. |
-| datafusion.optimizer.skip_failed_rules | Boolean | true | When set to true, the logical plan optimizer will produce warning messages if any optimization rules produce errors and then proceed to the next rule. When set to false, any rules that produce errors will cause the query to fail. |
diff --git a/ballista/_sources/user-guide/deployment/cargo-install.md.txt b/ballista/_sources/user-guide/deployment/cargo-install.md.txt
deleted file mode 100644
index 8e68278fd6..0000000000
--- a/ballista/_sources/user-guide/deployment/cargo-install.md.txt
+++ /dev/null
@@ -1,50 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Deploying a standalone Ballista cluster using cargo install
-
-A simple way to start a local cluster for testing purposes is to use cargo to install
-the scheduler and executor crates.
-
-```bash
-cargo install --locked ballista-scheduler
-cargo install --locked ballista-executor
-```
-
-With these crates installed, it is now possible to start a scheduler process.
-
-```bash
-RUST_LOG=info ballista-scheduler
-```
-
-The scheduler will bind to port 50050 by default.
-
-Next, start an executor processes in a new terminal session with the specified concurrency
-level.
-
-```bash
-RUST_LOG=info ballista-executor -c 4
-```
-
-The executor will bind to port 50051 by default. Additional executors can be started by
-manually specifying a bind port. For example:
-
-```bash
-RUST_LOG=info ballista-executor --bind-port 50052 -c 4
-```
diff --git a/ballista/_sources/user-guide/deployment/configuration.md.txt b/ballista/_sources/user-guide/deployment/configuration.md.txt
deleted file mode 100644
index 56ca928951..0000000000
--- a/ballista/_sources/user-guide/deployment/configuration.md.txt
+++ /dev/null
@@ -1,34 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Configuration
-
-The rust executor and scheduler can be configured using toml files, environment variables and command line arguments. The specification for config options can be found in `rust/ballista/src/bin/[executor|scheduler]_config_spec.toml`.
-
-Those files fully define Ballista's configuration. If there is a discrepancy between this documentation and the files, assume those files are correct.
-
-To get a list of command line arguments, run the binary with `--help`
-
-There is an example config file at `ballista/rust/ballista/examples/example_executor_config.toml`
-
-The order of precedence for arguments is: default config file < environment variables < specified config file < command line arguments.
-
-The executor and scheduler will look for the default config file at `/etc/ballista/[executor|scheduler].toml` To specify a config file use the `--config-file` argument.
-
-Environment variables are prefixed by `BALLISTA_EXECUTOR` or `BALLISTA_SCHEDULER` for the executor and scheduler respectively. Hyphens in command line arguments become underscores. For example, the `--scheduler-host` argument for the executor becomes `BALLISTA_EXECUTOR_SCHEDULER_HOST`
diff --git a/ballista/_sources/user-guide/deployment/docker-compose.md.txt b/ballista/_sources/user-guide/deployment/docker-compose.md.txt
deleted file mode 100644
index 1cd841765d..0000000000
--- a/ballista/_sources/user-guide/deployment/docker-compose.md.txt
+++ /dev/null
@@ -1,97 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Starting a Ballista cluster using Docker Compose
-
-Docker Compose is a convenient way to launch a cluster when testing locally.
-
-## Build Docker image
-
-There is no officially published Docker image so it is currently necessary to build the image from source instead.
-
-Run the following commands to clone the source repository and build the Docker image.
-
-```bash
-git clone git@github.com:apache/arrow-datafusion.git -b 8.0.0
-cd arrow-datafusion
-./dev/build-ballista-docker.sh
-```
-
-This will create an image with the tag `ballista:0.7.0`.
-
-## Start a cluster
-
-The following Docker Compose example demonstrates how to start a cluster using one scheduler process and one
-executor process, with the scheduler using etcd as a backing store. A data volume is mounted into each container
-so that Ballista can access the host file system.
-
-```yaml
-version: "2.2"
-services:
- etcd:
- image: quay.io/coreos/etcd:v3.4.9
- command: "etcd -advertise-client-urls http://etcd:2379 -listen-client-urls http://0.0.0.0:2379"
- ballista-scheduler:
- image: ballista:0.8.0
- command: "/scheduler --config-backend etcd --etcd-urls etcd:2379 --bind-host 0.0.0.0 --bind-port 50050"
- ports:
- - "50050:50050"
- environment:
- - RUST_LOG=info
- volumes:
- - ./data:/data
- depends_on:
- - etcd
- ballista-executor:
- image: ballista:0.8.0
- command: "/executor --bind-host 0.0.0.0 --bind-port 50051 --scheduler-host ballista-scheduler"
- ports:
- - "50051:50051"
- environment:
- - RUST_LOG=info
- volumes:
- - ./data:/data
- depends_on:
- - ballista-scheduler
-```
-
-With the above content saved to a `docker-compose.yaml` file, the following command can be used to start the single
-node cluster.
-
-```bash
-docker-compose up
-```
-
-This should show output similar to the following:
-
-```bash
-$ docker-compose up
-Creating network "ballista-benchmarks_default" with the default driver
-Creating ballista-benchmarks_etcd_1 ... done
-Creating ballista-benchmarks_ballista-scheduler_1 ... done
-Creating ballista-benchmarks_ballista-executor_1 ... done
-Attaching to ballista-benchmarks_etcd_1, ballista-benchmarks_ballista-scheduler_1, ballista-benchmarks_ballista-executor_1
-ballista-executor_1 | [2021-08-28T15:55:22Z INFO ballista_executor] Running with config:
-ballista-executor_1 | [2021-08-28T15:55:22Z INFO ballista_executor] work_dir: /tmp/.tmpLVx39c
-ballista-executor_1 | [2021-08-28T15:55:22Z INFO ballista_executor] concurrent_tasks: 4
-ballista-scheduler_1 | [2021-08-28T15:55:22Z INFO ballista_scheduler] Ballista v0.7.0 Scheduler listening on 0.0.0.0:50050
-ballista-executor_1 | [2021-08-28T15:55:22Z INFO ballista_executor] Ballista v0.7.0 Rust Executor listening on 0.0.0.0:50051
-```
-
-The scheduler listens on port 50050 and this is the port that clients will need to connect to.
diff --git a/ballista/_sources/user-guide/deployment/docker.md.txt b/ballista/_sources/user-guide/deployment/docker.md.txt
deleted file mode 100644
index e00acac98c..0000000000
--- a/ballista/_sources/user-guide/deployment/docker.md.txt
+++ /dev/null
@@ -1,106 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Starting a Ballista cluster using Docker
-
-## Build Docker image
-
-There is no officially published Docker image so it is currently necessary to build the image from source instead.
-
-Run the following commands to clone the source repository and build the Docker image.
-
-```bash
-git clone git@github.com:apache/arrow-datafusion.git -b 8.0.0
-cd arrow-datafusion
-./dev/build-ballista-docker.sh
-```
-
-This will create an image with the tag `ballista:0.7.0`.
-
-### Start a Scheduler
-
-Start a scheduler using the following syntax:
-
-```bash
-docker run --network=host \
- -d ballista:0.7.0 \
- /scheduler --bind-port 50050
-```
-
-Run `docker ps` to check that the process is running:
-
-```
-$ docker ps
-CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
-1f3f8b5ed93a ballista:0.7.0 "/scheduler --bind-p…" 2 seconds ago Up 1 second tender_archimedes
-```
-
-Run `docker logs CONTAINER_ID` to check the output from the process:
-
-```
-$ docker logs 1f3f8b5ed93a
-[2021-08-28T15:45:11Z INFO ballista_scheduler] Ballista v0.7.0 Scheduler listening on 0.0.0.0:50050
-```
-
-### Start executors
-
-Start one or more executor processes. Each executor process will need to listen on a different port.
-
-```bash
-docker run --network=host \
- -d ballista:0.7.0 \
- /executor --external-host localhost --bind-port 50051
-```
-
-Use `docker ps` to check that both the scheduer and executor(s) are now running:
-
-```
-$ docker ps
-CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
-7c6941bb8dc0 ballista:0.7.0 "/executor --externa…" 3 seconds ago Up 2 seconds tender_goldberg
-1f3f8b5ed93a ballista:0.7.0 "/scheduler --bind-p…" 50 seconds ago Up 49 seconds tender_archimedes
-```
-
-Use `docker logs CONTAINER_ID` to check the output from the executor(s):
-
-```
-$ docker logs 7c6941bb8dc0
-[2021-08-28T15:45:58Z INFO ballista_executor] Running with config:
-[2021-08-28T15:45:58Z INFO ballista_executor] work_dir: /tmp/.tmpeyEM76
-[2021-08-28T15:45:58Z INFO ballista_executor] concurrent_tasks: 4
-[2021-08-28T15:45:58Z INFO ballista_executor] Ballista v0.7.0 Rust Executor listening on 0.0.0.0:50051
-```
-
-### Using etcd as backing store
-
-_NOTE: This functionality is currently experimental_
-
-Ballista can optionally use [etcd](https://etcd.io/) as a backing store for the scheduler. Use the following commands
-to launch the scheduler with this option enabled.
-
-```bash
-docker run --network=host \
- -d ballista:0.7.0 \
- /scheduler --bind-port 50050 \
- --config-backend etcd \
- --etcd-urls etcd:2379
-```
-
-Please refer to the [etcd](https://etcd.io/) web site for installation instructions. Etcd version 3.4.9 or later is
-recommended.
diff --git a/ballista/_sources/user-guide/deployment/index.rst.txt b/ballista/_sources/user-guide/deployment/index.rst.txt
deleted file mode 100644
index ad9c07143f..0000000000
--- a/ballista/_sources/user-guide/deployment/index.rst.txt
+++ /dev/null
@@ -1,28 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-
-Start a Ballista Cluster
-========================
-
-.. toctree::
- :maxdepth: 2
-
- cargo-install
- docker
- docker-compose
- kubernetes
- configuration
diff --git a/ballista/_sources/user-guide/deployment/kubernetes.md.txt b/ballista/_sources/user-guide/deployment/kubernetes.md.txt
deleted file mode 100644
index 6886b879a6..0000000000
--- a/ballista/_sources/user-guide/deployment/kubernetes.md.txt
+++ /dev/null
@@ -1,308 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Deploying Ballista with Kubernetes
-
-Ballista can be deployed to any Kubernetes cluster using the following instructions. These instructions assume that
-you are already comfortable managing Kubernetes deployments.
-
-The Ballista deployment consists of:
-
-- k8s deployment for one or more scheduler processes
-- k8s deployment for one or more executor processes
-- k8s service to route traffic to the schedulers
-- k8s persistent volume and persistent volume claims to make local data accessible to Ballista
-- _(optional)_ a [keda](http://keda.sh) instance for autoscaling the number of executors
-
-## Testing locally
-
-[Microk8s](https://microk8s.io/) is recommended for installing a local k8s cluster. Once Microk8s is installed, DNS
-must be enabled using the following command.
-
-```bash
-microk8s enable dns
-```
-
-## Limitations
-
-Ballista is at an early stage of development and therefore has some significant limitations:
-
-- There is no support for shared object stores such as S3. All data must exist locally on each node in the
- cluster, including where any client process runs.
-- Only a single scheduler instance is currently supported unless the scheduler is configured to use `etcd` as a
- backing store.
-
-## Build Docker image
-
-There is no officially published Docker image so it is currently necessary to build the image from source instead.
-
-Run the following commands to clone the source repository and build the Docker image.
-
-```bash
-git clone git@github.com:apache/arrow-ballista.git
-cd arrow-ballista
-./dev/build-ballista-docker.sh
-```
-
-This will create an image with the tag `ballista:0.7.0`.
-
-## Publishing your images
-
-Once the images have been built, you can retag them and can push them to your favourite docker registry.
-
-```bash
-docker tag ballista:0.7.0 <your-repo>/ballista:0.7.0
-docker push <your-repo>/ballista:0.7.0
-```
-
-## Create Persistent Volume and Persistent Volume Claim
-
-Copy the following yaml to a `pv.yaml` file and apply to the cluster to create a persistent volume and a persistent
-volume claim so that the specified host directory is available to the containers. This is where any data should be
-located so that Ballista can execute queries against it.
-
-```yaml
-apiVersion: v1
-kind: PersistentVolume
-metadata:
- name: data-pv
- labels:
- type: local
-spec:
- storageClassName: manual
- capacity:
- storage: 10Gi
- accessModes:
- - ReadWriteOnce
- hostPath:
- path: "/mnt"
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
- name: data-pv-claim
-spec:
- storageClassName: manual
- accessModes:
- - ReadWriteOnce
- resources:
- requests:
- storage: 3Gi
-```
-
-To apply this yaml:
-
-```bash
-kubectl apply -f pv.yaml
-```
-
-You should see the following output:
-
-```bash
-persistentvolume/data-pv created
-persistentvolumeclaim/data-pv-claim created
-```
-
-## Deploying Ballista Scheduler and Executors
-
-Copy the following yaml to a `cluster.yaml` file and change `<your-image>` with the name of your Ballista Docker image.
-
-```yaml
-apiVersion: v1
-kind: Service
-metadata:
- name: ballista-scheduler
- labels:
- app: ballista-scheduler
-spec:
- ports:
- - port: 50050
- name: scheduler
- selector:
- app: ballista-scheduler
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: ballista-scheduler
-spec:
- replicas: 1
- selector:
- matchLabels:
- app: ballista-scheduler
- template:
- metadata:
- labels:
- app: ballista-scheduler
- ballista-cluster: ballista
- spec:
- containers:
- - name: ballista-scheduler
- image: <your-repo>/ballista:0.7.0
- command: ["/scheduler"]
- args: ["--bind-port=50050"]
- ports:
- - containerPort: 50050
- name: flight
- volumeMounts:
- - mountPath: /mnt
- name: data
- volumes:
- - name: data
- persistentVolumeClaim:
- claimName: data-pv-claim
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: ballista-executor
-spec:
- replicas: 2
- selector:
- matchLabels:
- app: ballista-executor
- template:
- metadata:
- labels:
- app: ballista-executor
- ballista-cluster: ballista
- spec:
- containers:
- - name: ballista-executor
- image: <your-repo>/ballista:0.7.0
- command: ["/executor"]
- args:
- - "--bind-port=50051"
- - "--scheduler-host=ballista-scheduler"
- - "--scheduler-port=50050"
- ports:
- - containerPort: 50051
- name: flight
- volumeMounts:
- - mountPath: /mnt
- name: data
- volumes:
- - name: data
- persistentVolumeClaim:
- claimName: data-pv-claim
-```
-
-```bash
-$ kubectl apply -f cluster.yaml
-```
-
-This should show the following output:
-
-```
-service/ballista-scheduler created
-deployment.apps/ballista-scheduler created
-deployment.apps/ballista-executor created
-```
-
-You can also check status by running `kubectl get pods`:
-
-```bash
-$ kubectl get pods
-NAME READY STATUS RESTARTS AGE
-ballista-executor-78cc5b6486-4rkn4 0/1 Pending 0 42s
-ballista-executor-78cc5b6486-7crdm 0/1 Pending 0 42s
-ballista-scheduler-879f874c5-rnbd6 0/1 Pending 0 42s
-```
-
-You can view the scheduler logs with `kubectl logs ballista-scheduler-0`:
-
-```
-$ kubectl logs ballista-scheduler-0
-[2021-02-19T00:24:01Z INFO scheduler] Ballista v0.7.0 Scheduler listening on 0.0.0.0:50050
-[2021-02-19T00:24:16Z INFO ballista::scheduler] Received register_executor request for ExecutorMetadata { id: "b5e81711-1c5c-46ec-8522-d8b359793188", host: "10.1.23.149", port: 50051 }
-[2021-02-19T00:24:17Z INFO ballista::scheduler] Received register_executor request for ExecutorMetadata { id: "816e4502-a876-4ed8-b33f-86d243dcf63f", host: "10.1.23.150", port: 50051 }
-```
-
-## Port Forwarding
-
-If you want to run applications outside of the cluster and have them connect to the scheduler then it is necessary to
-set up port forwarding.
-
-First, check that the `ballista-scheduler` service is running.
-
-```bash
-$ kubectl get services
-NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
-kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 26h
-ballista-scheduler ClusterIP 10.152.183.21 <none> 50050/TCP 24m
-```
-
-Use the following command to set up port-forwarding.
-
-```bash
-kubectl port-forward service/ballista-scheduler 50050:50050
-```
-
-## Deleting the Ballista cluster
-
-Run the following kubectl command to delete the cluster.
-
-```bash
-kubectl delete -f cluster.yaml
-```
-
-## Adding autoscaling for executors
-
-Ballista supports autoscaling for executors through [Keda](http://keda.sh). Keda allows scaling a deployment
-through custom metrics which are exposed through the Ballista scheduler, and it can even scale the number of
-executors down to 0 if there is no activity in the cluster.
-
-Keda can be installed in your kubernetes cluster through a single command line:
-
-```bash
-kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.7.1/keda-2.7.1.yaml
-```
-
-Once you have deployed Keda on your cluster, you can now deploy a new kubernetes object called `ScaledObject`
-which will let Keda know how to scale your executors. In order to do that, copy the following YAML into a
-`scale.yaml` file:
-
-```yaml
-apiVersion: keda.sh/v1alpha1
-kind: ScaledObject
-metadata:
- name: ballista-executor
-spec:
- scaleTargetRef:
- name: ballista-executor
- minReplicaCount: 0
- maxReplicaCount: 5
- triggers:
- - type: external
- metadata:
- # Change this DNS if the scheduler isn't deployed in the "default" namespace
- scalerAddress: ballista-scheduler.default.svc.cluster.local:50050
-```
-
-And then deploy it into the cluster:
-
-```bash
-kubectl apply -f scale.yaml
-```
-
-If the cluster is inactive, Keda will now scale the number of executors down to 0, and will scale them up when
-you launch a query. Please note that Keda will perform a scan once every 30 seconds, so it might take a bit to
-scale the executors.
-
-Please visit Keda's [documentation page](https://keda.sh/docs/2.7/concepts/scaling-deployments/) for more information.
diff --git a/ballista/_sources/user-guide/faq.md.txt b/ballista/_sources/user-guide/faq.md.txt
deleted file mode 100644
index f96306c1ce..0000000000
--- a/ballista/_sources/user-guide/faq.md.txt
+++ /dev/null
@@ -1,31 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Frequently Asked Questions
-
-## What is the relationship between Apache Arrow, DataFusion, and Ballista?
-
-Apache Arrow is a library which provides a standardized memory representation for columnar data. It also provides
-"kernels" for performing common operations on this data.
-
-DataFusion is a library for executing queries in-process using the Apache Arrow memory
-model and computational kernels. It is designed to run within a single process, using threads
-for parallel query execution.
-
-Ballista is a distributed compute platform built on DataFusion.
diff --git a/ballista/_sources/user-guide/introduction.md.txt b/ballista/_sources/user-guide/introduction.md.txt
deleted file mode 100644
index 77db6261c0..0000000000
--- a/ballista/_sources/user-guide/introduction.md.txt
+++ /dev/null
@@ -1,50 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Overview
-
-Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is
-built on an architecture that allows other programming languages to be supported as first-class citizens without paying
-a penalty for serialization costs.
-
-The foundational technologies in Ballista are:
-
-- [Apache Arrow](https://arrow.apache.org/) memory model and compute kernels for efficient processing of data.
-- [Apache Arrow Flight Protocol](https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/) for efficient data transfer between processes.
-- [Google Protocol Buffers](https://developers.google.com/protocol-buffers) for serializing query plans.
-- [DataFusion](https://github.com/apache/arrow-datafusion/) for query execution.
-
-## How does this compare to Apache Spark?
-
-Although Ballista is largely inspired by Apache Spark, there are some key differences.
-
-- The choice of Rust as the main execution language means that memory usage is deterministic and avoids the overhead
- of GC pauses.
-- Ballista is designed from the ground up to use columnar data, enabling a number of efficiencies such as vectorized
- processing (SIMD and GPU) and efficient compression. Although Spark does have some columnar support, it is still
- largely row-based today.
-- The combination of Rust and Arrow provides excellent memory efficiency and memory usage can be 5x - 10x lower than
- Apache Spark in some cases, which means that more processing can fit on a single node, reducing the overhead of
- distributed compute.
-- The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors
- in any programming language with minimal serialization overhead.
-
-## Status
-
-Ballista is still in the early stages of development but is capable of executing complex analytical queries at scale.
diff --git a/ballista/_sources/user-guide/python.md.txt b/ballista/_sources/user-guide/python.md.txt
deleted file mode 100644
index 3bd4fe50bb..0000000000
--- a/ballista/_sources/user-guide/python.md.txt
+++ /dev/null
@@ -1,90 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Ballista Python Bindings
-
-Ballista provides Python bindings, allowing SQL and DataFrame queries to be executed from the Python shell.
-
-## Connecting to a Cluster
-
-The following code demonstrates how to create a Ballista context and connect to a scheduler.
-
-```text
->>> import ballista
->>> ctx = ballista.BallistaContext("localhost", 50050)
-```
-
-## Registering Tables
-
-Tables can be registered against the context by calling one of the `register` methods, or by executing SQL.
-
-```text
->>> ctx.register_parquet("trips", "/mnt/bigdata/nyctaxi")
-```
-
-```text
->>> ctx.sql("CREATE EXTERNAL TABLE trips STORED AS PARQUET LOCATION '/mnt/bigdata/nyctaxi'")
-```
-
-## Executing Queries
-
-The `sql` method creates a `DataFrame`. The query is executed when an action such as `show` or `collect` is executed.
-
-### Showing Query Results
-
-```text
->>> df = ctx.sql("SELECT count(*) FROM trips")
->>> df.show()
-+-----------------+
-| COUNT(UInt8(1)) |
-+-----------------+
-| 9071244 |
-+-----------------+
-```
-
-### Collecting Query Results
-
-The `collect` method executres the query and returns the results in
-[PyArrow](https://arrow.apache.org/docs/python/index.html) record batches.
-
-```text
->>> df = ctx.sql("SELECT count(*) FROM trips")
->>> df.collect()
-[pyarrow.RecordBatch
-COUNT(UInt8(1)): int64]
-```
-
-### Viewing Query Plans
-
-The `explain` method can be used to show the logical and physical query plans for a query.
-
-```text
->>> df.explain()
-+---------------+-------------------------------------------------------------+
-| plan_type | plan |
-+---------------+-------------------------------------------------------------+
-| logical_plan | Projection: #COUNT(UInt8(1)) |
-| | Aggregate: groupBy=[[]], aggr=[[COUNT(UInt8(1))]] |
-| | TableScan: trips projection=[VendorID] |
-| physical_plan | ProjectionExec: expr=[COUNT(UInt8(1))@0 as COUNT(UInt8(1))] |
-| | ProjectionExec: expr=[9071244 as COUNT(UInt8(1))] |
-| | EmptyExec: produce_one_row=true |
-| | |
-+---------------+-------------------------------------------------------------+
-```
diff --git a/ballista/_sources/user-guide/rust.md.txt b/ballista/_sources/user-guide/rust.md.txt
deleted file mode 100644
index 8788d05831..0000000000
--- a/ballista/_sources/user-guide/rust.md.txt
+++ /dev/null
@@ -1,99 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Ballista Rust Client
-
-Ballista usage is very similar to DataFusion. Tha main difference is that the starting point is a `BallistaContext`
-instead of the DataFusion `SessionContext`. Ballista uses the same DataFrame API as DataFusion.
-
-The following code sample demonstrates how to create a `BallistaContext` to connect to a Ballista scheduler process.
-
-```rust
-let config = BallistaConfig::builder()
- .set("ballista.shuffle.partitions", "4")
- .build()?;
-
-// connect to Ballista scheduler
-let ctx = BallistaContext::remote("localhost", 50050, &config);
-```
-
-Here is a full example using the DataFrame API.
-
-```rust
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = BallistaConfig::builder()
- .set("ballista.shuffle.partitions", "4")
- .build()?;
-
- // connect to Ballista scheduler
- let ctx = BallistaContext::remote("localhost", 50050, &config);
-
- let testdata = datafusion::test_util::parquet_test_data();
-
- let filename = &format!("{}/alltypes_plain.parquet", testdata);
-
- // define the query using the DataFrame trait
- let df = ctx
- .read_parquet(filename)?
- .select_columns(&["id", "bool_col", "timestamp_col"])?
- .filter(col("id").gt(lit(1)))?;
-
- // print the results
- df.show().await?;
-
- Ok(())
-}
-```
-
-Here is a full example demonstrating SQL usage.
-
-```rust
-#[tokio::main]
-async fn main() -> Result<()> {
- let config = BallistaConfig::builder()
- .set("ballista.shuffle.partitions", "4")
- .build()?;
-
- // connect to Ballista scheduler
- let ctx = BallistaContext::remote("localhost", 50050, &config);
-
- let testdata = datafusion::test_util::arrow_test_data();
-
- // register csv file with the execution context
- ctx.register_csv(
- "aggregate_test_100",
- &format!("{}/csv/aggregate_test_100.csv", testdata),
- CsvReadOptions::new(),
- )?;
-
- // execute the query
- let df = ctx.sql(
- "SELECT c1, MIN(c12), MAX(c12) \
- FROM aggregate_test_100 \
- WHERE c11 > 0.1 AND c11 < 0.9 \
- GROUP BY c1",
- )?;
-
- // print the results
- df.show().await?;
-
- Ok(())
-}
-```
diff --git a/ballista/_sources/user-guide/tuning-guide.md.txt b/ballista/_sources/user-guide/tuning-guide.md.txt
deleted file mode 100644
index 3d2700512a..0000000000
--- a/ballista/_sources/user-guide/tuning-guide.md.txt
+++ /dev/null
@@ -1,71 +0,0 @@
-<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Tuning Guide
-
-## Partitions and Parallelism
-
-The goal of any distributed compute engine is to parallelize work as much as possible, allowing the work to scale
-by adding more compute resource.
-
-The basic unit of concurrency and parallelism in Ballista is the concept of a partition. The leaf nodes of a query
-are typically table scans that read from files on disk and Ballista currently treats each file within a table as a
-single partition (in the future, Ballista will support splitting files into partitions but this is not implemented yet).
-
-For example, if there is a table "customer" that consists of 200 Parquet files, that table scan will naturally have
-200 partitions and the table scan and certain subsequent operations will also have 200 partitions. Conversely, if the
-table only has a single Parquet file then there will be a single partition and the work will not be able to scale even
-if the cluster has resource available. Ballista supports repartitioning within a query to improve parallelism.
-The configuration setting `ballista.shuffle.partitions`can be set to the desired number of partitions. This is
-currently a global setting for the entire context. The default value for this setting is 16.
-
-Note that Ballista will never decrease the number of partitions based on this setting and will only repartition if
-the source operation has fewer partitions than this setting.
-
-Example: Setting the desired number of shuffle partitions when creating a context.
-
-```rust
-let config = BallistaConfig::builder()
- .set("ballista.shuffle.partitions", "200")
- .build()?;
-
-let ctx = BallistaContext::remote("localhost", 50050, &config).await?;
-```
-
-## Configuring Executor Concurrency Levels
-
-Each executor instance has a fixed number of tasks that it can process concurrently. This is specified by passing a
-`concurrent_tasks` command-line parameter. The default setting is to use all available CPU cores.
-
-Increasing this configuration setting will increase the number of tasks that each executor can run concurrently but
-this will also mean that the executor will use more memory. If executors are failing due to out-of-memory errors then
-decreasing the number of concurrent tasks may help.
-
-In the future, Ballista will have better support for tracking memory usage and allocating tasks based on available
-memory, as well as supporting spill-to-disk to reduce memory pressure.
-
-## Push-based vs Pull-based Task Scheduling
-
-Ballista supports both push-based and pull-based task scheduling. It is recommended that you try both to determine
-which is the best for your use case.
-
-Pull-based scheduling works in a similar way to Apache Spark and push-based scheduling can result in lower latency.
-
-The scheduling policy can be specified in the `--scheduler_policy` parameter when starting the scheduler and executor
-processes. The default is `pull-based`.
diff --git a/ballista/community/communication.html b/ballista/community/communication.html
index f582e795eb..582f094dac 100644
--- a/ballista/community/communication.html
+++ b/ballista/community/communication.html
@@ -78,12 +78,19 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="../user-guide/deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -93,46 +100,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="../user-guide/deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../user-guide/deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../user-guide/deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="../user-guide/deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="../user-guide/deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../user-guide/configs.html">
Configuration
diff --git a/ballista/genindex.html b/ballista/genindex.html
index b71494d00e..9c1239e64e 100644
--- a/ballista/genindex.html
+++ b/ballista/genindex.html
@@ -76,12 +76,19 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="user-guide/deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -91,46 +98,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="user-guide/deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/configs.html">
Configuration
diff --git a/ballista/index.html b/ballista/index.html
index ba62a5b510..9c3d691a8d 100644
--- a/ballista/index.html
+++ b/ballista/index.html
@@ -78,12 +78,19 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="user-guide/deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -93,46 +100,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="user-guide/deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/configs.html">
Configuration
@@ -248,11 +274,28 @@
<div class="toctree-wrapper compound" id="toc-guide">
<p aria-level="2" class="caption" role="heading"><span class="caption-text">User Guide</span></p>
<ul>
-<li class="toctree-l1"><a class="reference internal" href="user-guide/introduction.html">Overview</a></li>
-<li class="toctree-l1"><a class="reference internal" href="user-guide/deployment/index.html">Start a Ballista Cluster</a></li>
-<li class="toctree-l1"><a class="reference internal" href="user-guide/python.html">Ballista Python Bindings</a></li>
-<li class="toctree-l1"><a class="reference internal" href="user-guide/rust.html">Ballista Rust Client</a></li>
-<li class="toctree-l1"><a class="reference internal" href="user-guide/cli.html">DataFusion Command-line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/introduction.html">Introduction</a></li>
+</ul>
+</div>
+<div class="toctree-wrapper compound">
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Cluster Deployment</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/deployment/index.html">Deployment</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/scheduler.html">Scheduler</a></li>
+</ul>
+</div>
+<div class="toctree-wrapper compound">
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Clients</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/python.html">Python</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/rust.html">Rust</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/flightsql.html">Flight SQL JDBC</a></li>
+<li class="toctree-l1"><a class="reference internal" href="user-guide/cli.html">SQL CLI</a></li>
+</ul>
+</div>
+<div class="toctree-wrapper compound">
+<p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p>
+<ul>
<li class="toctree-l1"><a class="reference internal" href="user-guide/configs.html">Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-guide/tuning-guide.html">Tuning Guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="user-guide/faq.html">Frequently Asked Questions</a></li>
diff --git a/ballista/objects.inv b/ballista/objects.inv
index 64702f02c3..415ece7177 100644
Binary files a/ballista/objects.inv and b/ballista/objects.inv differ
diff --git a/ballista/search.html b/ballista/search.html
index ea7d331763..4e2d5e8ba7 100644
--- a/ballista/search.html
+++ b/ballista/search.html
@@ -82,12 +82,19 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="user-guide/deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -97,46 +104,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="user-guide/deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="user-guide/deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="user-guide/cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="user-guide/configs.html">
Configuration
diff --git a/ballista/searchindex.js b/ballista/searchindex.js
index 4a9ebd5ea3..c5e3af30ef 100644
--- a/ballista/searchindex.js
+++ b/ballista/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"docnames": ["community/communication", "index", "user-guide/cli", "user-guide/configs", "user-guide/deployment/cargo-install", "user-guide/deployment/configuration", "user-guide/deployment/docker", "user-guide/deployment/docker-compose", "user-guide/deployment/index", "user-guide/deployment/kubernetes", "user-guide/faq", "user-guide/introduction", "user-guide/python", "user-guide/rust", "user-guide/tuning-guide"], "filenames": ["community/communication.md", "index.rst", [...]
\ No newline at end of file
+Search.setIndex({"docnames": ["community/communication", "index", "user-guide/cli", "user-guide/configs", "user-guide/deployment/cargo-install", "user-guide/deployment/docker", "user-guide/deployment/docker-compose", "user-guide/deployment/index", "user-guide/deployment/kubernetes", "user-guide/faq", "user-guide/flightsql", "user-guide/introduction", "user-guide/python", "user-guide/rust", "user-guide/scheduler", "user-guide/tuning-guide"], "filenames": ["community/communication.md", "in [...]
\ No newline at end of file
diff --git a/ballista/user-guide/cli.html b/ballista/user-guide/cli.html
index a33e4d8cdd..8b4e989e2c 100644
--- a/ballista/user-guide/cli.html
+++ b/ballista/user-guide/cli.html
@@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />
- <title>DataFusion Command-line Interface — Arrow DataFusion documentation</title>
+ <title>Ballista Command-line Interface — Arrow DataFusion documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
@@ -37,7 +37,7 @@
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Configuration" href="configs.html" />
- <link rel="prev" title="Ballista Rust Client" href="rust.html" />
+ <link rel="prev" title="Using FlightSQL to Connect to Ballista" href="flightsql.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="configs.html">
Configuration
@@ -207,19 +233,24 @@
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#usage">
+ Usage
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#example">
Example
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#datafusion-cli">
- DataFusion-Cli
+ <a class="reference internal nav-link" href="#run-ballista-cli-in-distributed-mode">
+ Run Ballista CLI in Distributed Mode
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#ballista">
- Ballista
+ <a class="reference internal nav-link" href="#run-ballista-cli-in-standalone-mode">
+ Run Ballista CLI in Standalone Mode
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -273,36 +304,54 @@
specific language governing permissions and limitations
under the License.
-->
-<section id="datafusion-command-line-interface">
-<h1>DataFusion Command-line Interface<a class="headerlink" href="#datafusion-command-line-interface" title="Permalink to this heading">¶</a></h1>
-<p>The DataFusion CLI allows SQL queries to be executed by an in-process DataFusion context, or by a distributed
-Ballista context.</p>
+<section id="ballista-command-line-interface">
+<h1>Ballista Command-line Interface<a class="headerlink" href="#ballista-command-line-interface" title="Permalink to this heading">¶</a></h1>
+<p>The Ballista CLI allows SQL queries to be executed against a Ballista cluster, or in standalone mode in a single
+process.</p>
+<p>Use Cargo to install:</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>cargo install ballista-cli
+</pre></div>
+</div>
+<section id="usage">
+<h2>Usage<a class="headerlink" href="#usage" title="Permalink to this heading">¶</a></h2>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">USAGE</span><span class="p">:</span>
- <span class="n">datafusion</span><span class="o">-</span><span class="n">cli</span> <span class="p">[</span><span class="n">FLAGS</span><span class="p">]</span> <span class="p">[</span><span class="n">OPTIONS</span><span class="p">]</span>
-
-<span class="n">FLAGS</span><span class="p">:</span>
- <span class="o">-</span><span class="n">h</span><span class="p">,</span> <span class="o">--</span><span class="n">help</span> <span class="n">Prints</span> <span class="n">help</span> <span class="n">information</span>
- <span class="o">-</span><span class="n">q</span><span class="p">,</span> <span class="o">--</span><span class="n">quiet</span> <span class="n">Reduce</span> <span class="n">printing</span> <span class="n">other</span> <span class="n">than</span> <span class="n">the</span> <span class="n">results</span> <span class="ow">and</span> <span class="n">work</span> <span class="n">quietly</span>
- <span class="o">-</span><span class="n">V</span><span class="p">,</span> <span class="o">--</span><span class="n">version</span> <span class="n">Prints</span> <span class="n">version</span> <span class="n">information</span>
+ <span class="n">ballista</span><span class="o">-</span><span class="n">cli</span> <span class="p">[</span><span class="n">OPTIONS</span><span class="p">]</span>
<span class="n">OPTIONS</span><span class="p">:</span>
- <span class="o">-</span><span class="n">c</span><span class="p">,</span> <span class="o">--</span><span class="n">batch</span><span class="o">-</span><span class="n">size</span> <span class="o"><</span><span class="n">batch</span><span class="o">-</span><span class="n">size</span><span class="o">></span> <span class="n">The</span> <span class="n">batch</span> <span class="n">size</span> <span class="n">of</span> <span class="n">each</span> <span class="n">query</span><span c [...]
- <span class="o">-</span><span class="n">p</span><span class="p">,</span> <span class="o">--</span><span class="n">data</span><span class="o">-</span><span class="n">path</span> <span class="o"><</span><span class="n">data</span><span class="o">-</span><span class="n">path</span><span class="o">></span> <span class="n">Path</span> <span class="n">to</span> <span class="n">your</span> <span class="n">data</span><span class="p">,</span> <span class="n">default</span> <span cl [...]
- <span class="o">-</span><span class="n">f</span><span class="p">,</span> <span class="o">--</span><span class="n">file</span> <span class="o"><</span><span class="n">file</span><span class="o">>...</span> <span class="n">Execute</span> <span class="n">commands</span> <span class="kn">from</span> <span class="nn">file</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="n">then</span> <span class="n">exit</span>
- <span class="o">--</span><span class="nb">format</span> <span class="o"><</span><span class="nb">format</span><span class="o">></span> <span class="n">Output</span> <span class="nb">format</span> <span class="p">[</span><span class="n">default</span><span class="p">:</span> <span class="n">table</span><span class="p">]</span> <span class="p">[</span><span class="n">possible</span> <span class="n">values</span><span class="p">:</span> <span class="n">csv</span><s [...]
- <span class="o">--</span><span class="n">host</span> <span class="o"><</span><span class="n">host</span><span class="o">></span> <span class="n">Ballista</span> <span class="n">scheduler</span> <span class="n">host</span>
- <span class="o">--</span><span class="n">port</span> <span class="o"><</span><span class="n">port</span><span class="o">></span> <span class="n">Ballista</span> <span class="n">scheduler</span> <span class="n">port</span>
+ <span class="o">-</span><span class="n">c</span><span class="p">,</span> <span class="o">--</span><span class="n">batch</span><span class="o">-</span><span class="n">size</span> <span class="o"><</span><span class="n">BATCH_SIZE</span><span class="o">></span> <span class="n">The</span> <span class="n">batch</span> <span class="n">size</span> <span class="n">of</span> <span class="n">each</span> <span class="n">query</span><span class="p">,</span> <span class="ow">or</span> < [...]
+ <span class="o">-</span><span class="n">f</span><span class="p">,</span> <span class="o">--</span><span class="n">file</span> <span class="o"><</span><span class="n">FILE</span><span class="o">>...</span> <span class="n">Execute</span> <span class="n">commands</span> <span class="kn">from</span> <span class="nn">file</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="n">then</span> <span class="n">exit</span>
+ <span class="o">--</span><span class="nb">format</span> <span class="o"><</span><span class="n">FORMAT</span><span class="o">></span> <span class="p">[</span><span class="n">default</span><span class="p">:</span> <span class="n">table</span><span class="p">]</span> <span class="p">[</span><span class="n">possible</span> <span class="n">values</span><span class="p">:</span> <span class="n">csv</span><span class="p">,</span> <span class="n">tsv</span><span class="p [...]
+ <span class="n">nd</span><span class="o">-</span><span class="n">json</span><span class="p">]</span>
+ <span class="o">-</span><span class="n">h</span><span class="p">,</span> <span class="o">--</span><span class="n">help</span> <span class="n">Print</span> <span class="n">help</span> <span class="n">information</span>
+ <span class="o">--</span><span class="n">host</span> <span class="o"><</span><span class="n">HOST</span><span class="o">></span> <span class="n">Ballista</span> <span class="n">scheduler</span> <span class="n">host</span>
+ <span class="o">-</span><span class="n">p</span><span class="p">,</span> <span class="o">--</span><span class="n">data</span><span class="o">-</span><span class="n">path</span> <span class="o"><</span><span class="n">DATA_PATH</span><span class="o">></span> <span class="n">Path</span> <span class="n">to</span> <span class="n">your</span> <span class="n">data</span><span class="p">,</span> <span class="n">default</span> <span class="n">to</span> <span class="n">current</spa [...]
+ <span class="o">--</span><span class="n">port</span> <span class="o"><</span><span class="n">PORT</span><span class="o">></span> <span class="n">Ballista</span> <span class="n">scheduler</span> <span class="n">port</span>
+ <span class="o">-</span><span class="n">q</span><span class="p">,</span> <span class="o">--</span><span class="n">quiet</span> <span class="n">Reduce</span> <span class="n">printing</span> <span class="n">other</span> <span class="n">than</span> <span class="n">the</span> <span class="n">results</span> <span class="ow">and</span> <span class="n">work</span> <span class="n">quietly</span>
+ <span class="o">-</span><span class="n">r</span><span class="p">,</span> <span class="o">--</span><span class="n">rc</span> <span class="o"><</span><span class="n">RC</span><span class="o">>...</span> <span class="n">Run</span> <span class="n">the</span> <span class="n">provided</span> <span class="n">files</span> <span class="n">on</span> <span class="n">startup</span> <span class="n">instead</span> <span class="n">of</span> <span class="o">~/.</span><span clas [...]
+ <span class="o">-</span><span class="n">V</span><span class="p">,</span> <span class="o">--</span><span class="n">version</span> <span class="n">Print</span> <span class="n">version</span> <span class="n">information</span>
</pre></div>
</div>
+</section>
<section id="example">
<h2>Example<a class="headerlink" href="#example" title="Permalink to this heading">¶</a></h2>
<p>Create a CSV file to query.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ <span class="nb">echo</span> <span class="s2">"1,2"</span> > data.csv
</pre></div>
</div>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ datafusion-cli
+</section>
+<section id="run-ballista-cli-in-distributed-mode">
+<h2>Run Ballista CLI in Distributed Mode<a class="headerlink" href="#run-ballista-cli-in-distributed-mode" title="Permalink to this heading">¶</a></h2>
+<p>The CLI can connect to a Ballista scheduler for query execution.</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>datafusion-cli --host localhost --port <span class="m">50050</span>
+</pre></div>
+</div>
+</section>
+<section id="run-ballista-cli-in-standalone-mode">
+<h2>Run Ballista CLI in Standalone Mode<a class="headerlink" href="#run-ballista-cli-in-standalone-mode" title="Permalink to this heading">¶</a></h2>
+<p>It is also possible to run the CLI in standalone mode, where it will create a scheduler and executor in-process.</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ballista-cli
-DataFusion CLI v8.0.0
+Ballista CLI v8.0.0
> CREATE EXTERNAL TABLE foo <span class="o">(</span>a INT, b INT<span class="o">)</span> STORED AS CSV LOCATION <span class="s1">'data.csv'</span><span class="p">;</span>
<span class="m">0</span> rows <span class="k">in</span> set. Query took <span class="m">0</span>.001 seconds.
@@ -317,31 +366,9 @@ DataFusion CLI v8.0.0
</pre></div>
</div>
</section>
-<section id="datafusion-cli">
-<h2>DataFusion-Cli<a class="headerlink" href="#datafusion-cli" title="Permalink to this heading">¶</a></h2>
-<p>Build the <code class="docutils literal notranslate"><span class="pre">datafusion-cli</span></code> without the feature of ballista.</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span> arrow-datafusion/datafusion-cli
-cargo build
-</pre></div>
-</div>
-</section>
-<section id="ballista">
-<h2>Ballista<a class="headerlink" href="#ballista" title="Permalink to this heading">¶</a></h2>
-<p>The DataFusion CLI can also connect to a Ballista scheduler for query execution.</p>
-<p>Before you use the <code class="docutils literal notranslate"><span class="pre">datafusion-cli</span></code> to connect the Ballista scheduler, you should build/compile
-the <code class="docutils literal notranslate"><span class="pre">datafusion-cli</span></code> with feature of “ballista” first.</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span> arrow-datafusion/datafusion-cli
-cargo build --features ballista
-</pre></div>
-</div>
-<p>Then, you can connect the Ballista by below command.</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>datafusion-cli --host localhost --port <span class="m">50050</span>
-</pre></div>
-</div>
-</section>
<section id="cli-commands">
<h2>Cli commands<a class="headerlink" href="#cli-commands" title="Permalink to this heading">¶</a></h2>
-<p>Available commands inside DataFusion CLI are:</p>
+<p>Available commands inside Ballista CLI are:</p>
<ul class="simple">
<li><p>Quit</p></li>
</ul>
@@ -393,11 +420,11 @@ cargo build --features ballista
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="rust.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="flightsql.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Ballista Rust Client</p>
+ <p class="prev-next-title">Using FlightSQL to Connect to Ballista</p>
</div>
</a>
<a class='right-next' id="next-link" href="configs.html" title="next page">
diff --git a/ballista/user-guide/configs.html b/ballista/user-guide/configs.html
index 4086c7ede6..d90ca2966f 100644
--- a/ballista/user-guide/configs.html
+++ b/ballista/user-guide/configs.html
@@ -37,7 +37,7 @@
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Tuning Guide" href="tuning-guide.html" />
- <link rel="prev" title="DataFusion Command-line Interface" href="cli.html" />
+ <link rel="prev" title="Ballista Command-line Interface" href="cli.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Configuration
@@ -286,42 +312,47 @@
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>ballista.shuffle.partitions</p></td>
+<tr class="row-even"><td><p>ballista.job.name</p></td>
+<td><p>Utf8</p></td>
+<td><p>N/A</p></td>
+<td><p>Sets the job name that will appear in the web user interface for any submitted jobs.</p></td>
+</tr>
+<tr class="row-odd"><td><p>ballista.shuffle.partitions</p></td>
<td><p>UInt16</p></td>
<td><p>16</p></td>
<td><p>Sets the default number of partitions to create when repartitioning query stages.</p></td>
</tr>
-<tr class="row-odd"><td><p>ballista.batch.size</p></td>
+<tr class="row-even"><td><p>ballista.batch.size</p></td>
<td><p>UInt16</p></td>
<td><p>8192</p></td>
<td><p>Sets the default batch size.</p></td>
</tr>
-<tr class="row-even"><td><p>ballista.repartition.joins</p></td>
+<tr class="row-odd"><td><p>ballista.repartition.joins</p></td>
<td><p>Boolean</p></td>
<td><p>true</p></td>
<td><p>When set to true, Ballista will repartition data using the join keys to execute joins in parallel using the provided <code class="docutils literal notranslate"><span class="pre">ballista.shuffle.partitions</span></code> level.</p></td>
</tr>
-<tr class="row-odd"><td><p>ballista.repartition.aggregations</p></td>
+<tr class="row-even"><td><p>ballista.repartition.aggregations</p></td>
<td><p>Boolean</p></td>
<td><p>true</p></td>
<td><p>When set to true, Ballista will repartition data using the aggregate keys to execute aggregates in parallel using the provided <code class="docutils literal notranslate"><span class="pre">ballista.shuffle.partitions</span></code> level.</p></td>
</tr>
-<tr class="row-even"><td><p>ballista.repartition.windows</p></td>
+<tr class="row-odd"><td><p>ballista.repartition.windows</p></td>
<td><p>Boolean</p></td>
<td><p>true</p></td>
<td><p>When set to true, Ballista will repartition data using the partition keys to execute window functions in parallel using the provided <code class="docutils literal notranslate"><span class="pre">ballista.shuffle.partitions</span></code> level.</p></td>
</tr>
-<tr class="row-odd"><td><p>ballista.parquet.pruning</p></td>
+<tr class="row-even"><td><p>ballista.parquet.pruning</p></td>
<td><p>Boolean</p></td>
<td><p>true</p></td>
<td><p>Determines whether Parquet pruning should be enabled or not.</p></td>
</tr>
-<tr class="row-even"><td><p>ballista.with_information_schema</p></td>
+<tr class="row-odd"><td><p>ballista.with_information_schema</p></td>
<td><p>Boolean</p></td>
<td><p>true</p></td>
<td><p>Determines whether the <code class="docutils literal notranslate"><span class="pre">information_schema</span></code> should be created in the context. This is necessary for supporting DDL commands such as <code class="docutils literal notranslate"><span class="pre">SHOW</span> <span class="pre">TABLES</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p>ballista.plugin_dir</p></td>
+<tr class="row-even"><td><p>ballista.plugin_dir</p></td>
<td><p>Boolean</p></td>
<td><p>true</p></td>
<td><p>Specified a path for plugin files. Dynamic library files in this directory will be loaded when scheduler state initializes.</p></td>
@@ -386,7 +417,7 @@
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">DataFusion Command-line Interface</p>
+ <p class="prev-next-title">Ballista Command-line Interface</p>
</div>
</a>
<a class='right-next' id="next-link" href="tuning-guide.html" title="next page">
diff --git a/ballista/user-guide/deployment/cargo-install.html b/ballista/user-guide/deployment/cargo-install.html
index 3fd6863b55..94580bff3f 100644
--- a/ballista/user-guide/deployment/cargo-install.html
+++ b/ballista/user-guide/deployment/cargo-install.html
@@ -36,7 +36,7 @@
<script src="../../_static/doctools.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Starting a Ballista cluster using Docker" href="docker.html" />
+ <link rel="next" title="Starting a Ballista Cluster using Docker" href="docker.html" />
<link rel="prev" title="Start a Ballista Cluster" href="index.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul class="current">
<li class="toctree-l2 current active">
<a class="current reference internal" href="#">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../configs.html">
Configuration
@@ -261,14 +287,13 @@ cargo install --locked ballista-executor
</pre></div>
</div>
<p>The scheduler will bind to port 50050 by default.</p>
-<p>Next, start an executor processes in a new terminal session with the specified concurrency
-level.</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">RUST_LOG</span><span class="o">=</span>info ballista-executor -c <span class="m">4</span>
+<p>Next, start an executor processes in a new terminal session.</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">RUST_LOG</span><span class="o">=</span>info ballista-executor
</pre></div>
</div>
<p>The executor will bind to port 50051 by default. Additional executors can be started by
manually specifying a bind port. For example:</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">RUST_LOG</span><span class="o">=</span>info ballista-executor --bind-port <span class="m">50052</span> -c <span class="m">4</span>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">RUST_LOG</span><span class="o">=</span>info ballista-executor --bind-port <span class="m">50052</span>
</pre></div>
</div>
</section>
@@ -289,7 +314,7 @@ manually specifying a bind port. For example:</p>
<a class='right-next' id="next-link" href="docker.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Starting a Ballista cluster using Docker</p>
+ <p class="prev-next-title">Starting a Ballista Cluster using Docker</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/ballista/user-guide/deployment/docker-compose.html b/ballista/user-guide/deployment/docker-compose.html
index 6cae090e3b..445f9d74c0 100644
--- a/ballista/user-guide/deployment/docker-compose.html
+++ b/ballista/user-guide/deployment/docker-compose.html
@@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />
- <title>Starting a Ballista cluster using Docker Compose — Arrow DataFusion documentation</title>
+ <title>Starting a Ballista Cluster using Docker Compose — Arrow DataFusion documentation</title>
<link href="../../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
<link href="../../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
@@ -37,7 +37,7 @@
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
<link rel="next" title="Deploying Ballista with Kubernetes" href="kubernetes.html" />
- <link rel="prev" title="Starting a Ballista cluster using Docker" href="docker.html" />
+ <link rel="prev" title="Starting a Ballista Cluster using Docker" href="docker.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul class="current">
<li class="toctree-l2">
<a class="reference internal" href="cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2 current active">
<a class="current reference internal" href="#">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../configs.html">
Configuration
@@ -208,13 +234,13 @@
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#build-docker-image">
- Build Docker image
+ <a class="reference internal nav-link" href="#build-docker-images">
+ Build Docker Images
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#start-a-cluster">
- Start a cluster
+ Start a Cluster
</a>
</li>
</ul>
@@ -264,56 +290,28 @@
under the License.
-->
<section id="starting-a-ballista-cluster-using-docker-compose">
-<h1>Starting a Ballista cluster using Docker Compose<a class="headerlink" href="#starting-a-ballista-cluster-using-docker-compose" title="Permalink to this heading">¶</a></h1>
+<h1>Starting a Ballista Cluster using Docker Compose<a class="headerlink" href="#starting-a-ballista-cluster-using-docker-compose" title="Permalink to this heading">¶</a></h1>
<p>Docker Compose is a convenient way to launch a cluster when testing locally.</p>
-<section id="build-docker-image">
-<h2>Build Docker image<a class="headerlink" href="#build-docker-image" title="Permalink to this heading">¶</a></h2>
-<p>There is no officially published Docker image so it is currently necessary to build the image from source instead.</p>
-<p>Run the following commands to clone the source repository and build the Docker image.</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git clone git@github.com:apache/arrow-datafusion.git -b <span class="m">8</span>.0.0
-<span class="nb">cd</span> arrow-datafusion
+<section id="build-docker-images">
+<h2>Build Docker Images<a class="headerlink" href="#build-docker-images" title="Permalink to this heading">¶</a></h2>
+<p>There are no officially published Docker images, so it is currently necessary to build the images from source.
+Run the following commands to clone the source repository and build the Docker image.</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git clone git@github.com:apache/arrow-ballista.git -b <span class="m">8</span>.0.0
+<span class="nb">cd</span> arrow-ballista
./dev/build-ballista-docker.sh
</pre></div>
</div>
-<p>This will create an image with the tag <code class="docutils literal notranslate"><span class="pre">ballista:0.7.0</span></code>.</p>
+<p>This will create the following images:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">apache/arrow-ballista-scheduler:0.9.0</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">apache/arrow-ballista-executor:0.9.0</span></code></p></li>
+</ul>
</section>
<section id="start-a-cluster">
-<h2>Start a cluster<a class="headerlink" href="#start-a-cluster" title="Permalink to this heading">¶</a></h2>
-<p>The following Docker Compose example demonstrates how to start a cluster using one scheduler process and one
-executor process, with the scheduler using etcd as a backing store. A data volume is mounted into each container
-so that Ballista can access the host file system.</p>
-<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="s">"2.2"</span><span class="w"></span>
-<span class="nt">services</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="nt">etcd</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">quay.io/coreos/etcd:v3.4.9</span><span class="w"></span>
-<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="s">"etcd</span><span class="nv"> </span><span class="s">-advertise-client-urls</span><span class="nv"> </span><span class="s">http://etcd:2379</span><span class="nv"> </span><span class="s">-listen-client-urls</span><span class="nv"> </span><span class="s">http://0.0.0.0:2379"</span><span class="w"></span>
-<span class="w"> </span><span class="nt">ballista-scheduler</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ballista:0.8.0</span><span class="w"></span>
-<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="s">"/scheduler</span><span class="nv"> </span><span class="s">--config-backend</span><span class="nv"> </span><span class="s">etcd</span><span class="nv"> </span><span class="s">--etcd-urls</span><span class="nv"> </span><span class="s">etcd:2379</span><span class="nv"> </span><span class="s">--bind-host</span><span class="nv"> </span><span class="s">0.0.0.0</span>< [...]
-<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"50050:50050"</span><span class="w"></span>
-<span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">RUST_LOG=info</span><span class="w"></span>
-<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">./data:/data</span><span class="w"></span>
-<span class="w"> </span><span class="nt">depends_on</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">etcd</span><span class="w"></span>
-<span class="w"> </span><span class="nt">ballista-executor</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ballista:0.8.0</span><span class="w"></span>
-<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="s">"/executor</span><span class="nv"> </span><span class="s">--bind-host</span><span class="nv"> </span><span class="s">0.0.0.0</span><span class="nv"> </span><span class="s">--bind-port</span><span class="nv"> </span><span class="s">50051</span><span class="nv"> </span><span class="s">--scheduler-host</span><span class="nv"> </span><span class="s">ballista-schedule [...]
-<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"50051:50051"</span><span class="w"></span>
-<span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">RUST_LOG=info</span><span class="w"></span>
-<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">./data:/data</span><span class="w"></span>
-<span class="w"> </span><span class="nt">depends_on</span><span class="p">:</span><span class="w"></span>
-<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ballista-scheduler</span><span class="w"></span>
-</pre></div>
-</div>
-<p>With the above content saved to a <code class="docutils literal notranslate"><span class="pre">docker-compose.yaml</span></code> file, the following command can be used to start the single
-node cluster.</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker-compose up
+<h2>Start a Cluster<a class="headerlink" href="#start-a-cluster" title="Permalink to this heading">¶</a></h2>
+<p>Using the <a class="reference external" href="https://github.com/apache/arrow-ballista/blob/master/docker-compose.yml">docker-compose.yml</a> from the
+source repository, run the following command to start a cluster:</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker-compose up --build
</pre></div>
</div>
<p>This should show output similar to the following:</p>
@@ -326,11 +324,12 @@ Attaching to ballista-benchmarks_etcd_1, ballista-benchmarks_ballista-scheduler_
ballista-executor_1 <span class="p">|</span> <span class="o">[</span><span class="m">2021</span>-08-28T15:55:22Z INFO ballista_executor<span class="o">]</span> Running with config:
ballista-executor_1 <span class="p">|</span> <span class="o">[</span><span class="m">2021</span>-08-28T15:55:22Z INFO ballista_executor<span class="o">]</span> work_dir: /tmp/.tmpLVx39c
ballista-executor_1 <span class="p">|</span> <span class="o">[</span><span class="m">2021</span>-08-28T15:55:22Z INFO ballista_executor<span class="o">]</span> concurrent_tasks: <span class="m">4</span>
-ballista-scheduler_1 <span class="p">|</span> <span class="o">[</span><span class="m">2021</span>-08-28T15:55:22Z INFO ballista_scheduler<span class="o">]</span> Ballista v0.7.0 Scheduler listening on <span class="m">0</span>.0.0.0:50050
-ballista-executor_1 <span class="p">|</span> <span class="o">[</span><span class="m">2021</span>-08-28T15:55:22Z INFO ballista_executor<span class="o">]</span> Ballista v0.7.0 Rust Executor listening on <span class="m">0</span>.0.0.0:50051
+ballista-scheduler_1 <span class="p">|</span> <span class="o">[</span><span class="m">2021</span>-08-28T15:55:22Z INFO ballista_scheduler<span class="o">]</span> Ballista v0.9.0 Scheduler listening on <span class="m">0</span>.0.0.0:50050
+ballista-executor_1 <span class="p">|</span> <span class="o">[</span><span class="m">2021</span>-08-28T15:55:22Z INFO ballista_executor<span class="o">]</span> Ballista v0.9.0 Rust Executor listening on <span class="m">0</span>.0.0.0:50051
</pre></div>
</div>
<p>The scheduler listens on port 50050 and this is the port that clients will need to connect to.</p>
+<p>The scheduler web UI is available on port 80 in the scheduler.</p>
</section>
</section>
@@ -344,7 +343,7 @@ ballista-executor_1 <span class="p">|</span> <span class="o">[</span><span cla
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Starting a Ballista cluster using Docker</p>
+ <p class="prev-next-title">Starting a Ballista Cluster using Docker</p>
</div>
</a>
<a class='right-next' id="next-link" href="kubernetes.html" title="next page">
diff --git a/ballista/user-guide/deployment/docker.html b/ballista/user-guide/deployment/docker.html
index 113038f174..82989434dd 100644
--- a/ballista/user-guide/deployment/docker.html
+++ b/ballista/user-guide/deployment/docker.html
@@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />
- <title>Starting a Ballista cluster using Docker — Arrow DataFusion documentation</title>
+ <title>Starting a Ballista Cluster using Docker — Arrow DataFusion documentation</title>
<link href="../../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
<link href="../../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
@@ -36,7 +36,7 @@
<script src="../../_static/doctools.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Starting a Ballista cluster using Docker Compose" href="docker-compose.html" />
+ <link rel="next" title="Starting a Ballista Cluster using Docker Compose" href="docker-compose.html" />
<link rel="prev" title="Deploying a standalone Ballista cluster using cargo install" href="cargo-install.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul class="current">
<li class="toctree-l2">
<a class="reference internal" href="cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2 current active">
<a class="current reference internal" href="#">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../configs.html">
Configuration
@@ -208,8 +234,8 @@
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#build-docker-image">
- Build Docker image
+ <a class="reference internal nav-link" href="#build-docker-images">
+ Build Docker Images
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
@@ -219,12 +245,12 @@
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#start-executors">
- Start executors
+ Start Executors
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#using-etcd-as-backing-store">
- Using etcd as backing store
+ <a class="reference internal nav-link" href="#using-etcd-as-a-backing-store">
+ Using etcd as a Backing Store
</a>
</li>
</ul>
@@ -276,74 +302,83 @@
under the License.
-->
<section id="starting-a-ballista-cluster-using-docker">
-<h1>Starting a Ballista cluster using Docker<a class="headerlink" href="#starting-a-ballista-cluster-using-docker" title="Permalink to this heading">¶</a></h1>
-<section id="build-docker-image">
-<h2>Build Docker image<a class="headerlink" href="#build-docker-image" title="Permalink to this heading">¶</a></h2>
-<p>There is no officially published Docker image so it is currently necessary to build the image from source instead.</p>
+<h1>Starting a Ballista Cluster using Docker<a class="headerlink" href="#starting-a-ballista-cluster-using-docker" title="Permalink to this heading">¶</a></h1>
+<section id="build-docker-images">
+<h2>Build Docker Images<a class="headerlink" href="#build-docker-images" title="Permalink to this heading">¶</a></h2>
+<p>There are no officially published Docker images, so it is currently necessary to build the images from source.</p>
<p>Run the following commands to clone the source repository and build the Docker image.</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git clone git@github.com:apache/arrow-datafusion.git -b <span class="m">8</span>.0.0
-<span class="nb">cd</span> arrow-datafusion
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git clone git@github.com:apache/arrow-ballista.git -b <span class="m">0</span>.9.0
+<span class="nb">cd</span> arrow-ballista
./dev/build-ballista-docker.sh
</pre></div>
</div>
-<p>This will create an image with the tag <code class="docutils literal notranslate"><span class="pre">ballista:0.7.0</span></code>.</p>
+<p>This will create the following images:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">apache/arrow-ballista-scheduler:0.9.0</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">apache/arrow-ballista-executor:0.9.0</span></code></p></li>
+</ul>
<section id="start-a-scheduler">
<h3>Start a Scheduler<a class="headerlink" href="#start-a-scheduler" title="Permalink to this heading">¶</a></h3>
<p>Start a scheduler using the following syntax:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run --network<span class="o">=</span>host <span class="se">\</span>
- -d ballista:0.7.0 <span class="se">\</span>
- /scheduler --bind-port <span class="m">50050</span>
+ -d apache/arrow-ballista-scheduler:0.9.0 <span class="se">\</span>
+ --bind-port <span class="m">50050</span>
</pre></div>
</div>
<p>Run <code class="docutils literal notranslate"><span class="pre">docker</span> <span class="pre">ps</span></code> to check that the process is running:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ docker ps
-CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
-1f3f8b5ed93a ballista:0.7.0 "/scheduler --bind-p…" 2 seconds ago Up 1 second tender_archimedes
+CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
+8cdea4956c97 apache/arrow-ballista-scheduler:0.9.0 "/scheduler-entrypoi…" 2 seconds ago Up 1 second nervous_swirles
</pre></div>
</div>
<p>Run <code class="docutils literal notranslate"><span class="pre">docker</span> <span class="pre">logs</span> <span class="pre">CONTAINER_ID</span></code> to check the output from the process:</p>
-<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ docker logs 1f3f8b5ed93a
-[2021-08-28T15:45:11Z INFO ballista_scheduler] Ballista v0.7.0 Scheduler listening on 0.0.0.0:50050
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ docker logs 8cdea4956c97
+Starting nginx to serve Ballista Scheduler web UI on port 80
+2022-09-19T13:51:34.792363Z INFO main ThreadId(01) ballista_scheduler: Ballista v0.9.0 Scheduler listening on 0.0.0.0:50050
+2022-09-19T13:51:34.792395Z INFO main ThreadId(01) ballista_scheduler: Starting Scheduler grpc server with task scheduling policy of PullStaged
+2022-09-19T13:51:34.792494Z INFO main ThreadId(01) ballista_scheduler::scheduler_server::query_stage_scheduler: Starting QueryStageScheduler
+2022-09-19T13:51:34.792581Z INFO tokio-runtime-worker ThreadId(45) ballista_core::event_loop: Starting the event loop query_stage
</pre></div>
</div>
</section>
<section id="start-executors">
-<h3>Start executors<a class="headerlink" href="#start-executors" title="Permalink to this heading">¶</a></h3>
+<h3>Start Executors<a class="headerlink" href="#start-executors" title="Permalink to this heading">¶</a></h3>
<p>Start one or more executor processes. Each executor process will need to listen on a different port.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run --network<span class="o">=</span>host <span class="se">\</span>
- -d ballista:0.7.0 <span class="se">\</span>
- /executor --external-host localhost --bind-port <span class="m">50051</span>
+ -d apache/arrow-ballista-executor:0.9.0 <span class="se">\</span>
+ --external-host localhost --bind-port <span class="m">50051</span>
</pre></div>
</div>
-<p>Use <code class="docutils literal notranslate"><span class="pre">docker</span> <span class="pre">ps</span></code> to check that both the scheduer and executor(s) are now running:</p>
+<p>Use <code class="docutils literal notranslate"><span class="pre">docker</span> <span class="pre">ps</span></code> to check that both the scheduler and executor(s) are now running:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ docker ps
-CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
-7c6941bb8dc0 ballista:0.7.0 "/executor --externa…" 3 seconds ago Up 2 seconds tender_goldberg
-1f3f8b5ed93a ballista:0.7.0 "/scheduler --bind-p…" 50 seconds ago Up 49 seconds tender_archimedes
+CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
+f0b21f6b5050 apache/arrow-ballista-executor:0.9.0 "/executor-entrypoin…" 2 seconds ago Up 1 second relaxed_goldberg
+8cdea4956c97 apache/arrow-ballista-scheduler:0.9.0 "/scheduler-entrypoi…" 2 minutes ago Up 2 minutes nervous_swirles
</pre></div>
</div>
<p>Use <code class="docutils literal notranslate"><span class="pre">docker</span> <span class="pre">logs</span> <span class="pre">CONTAINER_ID</span></code> to check the output from the executor(s):</p>
-<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ docker logs 7c6941bb8dc0
-[2021-08-28T15:45:58Z INFO ballista_executor] Running with config:
-[2021-08-28T15:45:58Z INFO ballista_executor] work_dir: /tmp/.tmpeyEM76
-[2021-08-28T15:45:58Z INFO ballista_executor] concurrent_tasks: 4
-[2021-08-28T15:45:58Z INFO ballista_executor] Ballista v0.7.0 Rust Executor listening on 0.0.0.0:50051
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ docker logs f0b21f6b5050
+2022-09-19T13:54:10.806231Z INFO main ThreadId(01) ballista_executor: Running with config:
+2022-09-19T13:54:10.806261Z INFO main ThreadId(01) ballista_executor: work_dir: /tmp/.tmp5BdxT2
+2022-09-19T13:54:10.806265Z INFO main ThreadId(01) ballista_executor: concurrent_tasks: 48
+2022-09-19T13:54:10.807454Z INFO tokio-runtime-worker ThreadId(49) ballista_executor: Ballista v0.9.0 Rust Executor Flight Server listening on 0.0.0.0:50051
+2022-09-19T13:54:10.807467Z INFO tokio-runtime-worker ThreadId(46) ballista_executor::execution_loop: Starting poll work loop with scheduler
</pre></div>
</div>
</section>
-<section id="using-etcd-as-backing-store">
-<h3>Using etcd as backing store<a class="headerlink" href="#using-etcd-as-backing-store" title="Permalink to this heading">¶</a></h3>
+<section id="using-etcd-as-a-backing-store">
+<h3>Using etcd as a Backing Store<a class="headerlink" href="#using-etcd-as-a-backing-store" title="Permalink to this heading">¶</a></h3>
<p><em>NOTE: This functionality is currently experimental</em></p>
<p>Ballista can optionally use <a class="reference external" href="https://etcd.io/">etcd</a> as a backing store for the scheduler. Use the following commands
to launch the scheduler with this option enabled.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker run --network<span class="o">=</span>host <span class="se">\</span>
- -d ballista:0.7.0 <span class="se">\</span>
- /scheduler --bind-port <span class="m">50050</span> <span class="se">\</span>
+ -d apache/arrow-ballista-scheduler:0.9.0 <span class="se">\</span>
+ --bind-port <span class="m">50050</span> <span class="se">\</span>
--config-backend etcd <span class="se">\</span>
--etcd-urls etcd:2379
</pre></div>
</div>
-<p>Please refer to the <a class="reference external" href="https://etcd.io/">etcd</a> web site for installation instructions. Etcd version 3.4.9 or later is
+<p>Please refer to the <a class="reference external" href="https://etcd.io/">etcd</a> website for installation instructions. Etcd version 3.4.9 or later is
recommended.</p>
</section>
</section>
@@ -365,7 +400,7 @@ recommended.</p>
<a class='right-next' id="next-link" href="docker-compose.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Starting a Ballista cluster using Docker Compose</p>
+ <p class="prev-next-title">Starting a Ballista Cluster using Docker Compose</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/ballista/user-guide/deployment/index.html b/ballista/user-guide/deployment/index.html
index 248a54798f..90b317d12c 100644
--- a/ballista/user-guide/deployment/index.html
+++ b/ballista/user-guide/deployment/index.html
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="current reference internal" href="#">
- Start a Ballista Cluster
+ Deployment
</a>
<input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../configs.html">
Configuration
@@ -234,29 +260,27 @@
<h1>Start a Ballista Cluster<a class="headerlink" href="#start-a-ballista-cluster" title="Permalink to this heading">¶</a></h1>
<div class="toctree-wrapper compound">
<ul>
-<li class="toctree-l1"><a class="reference internal" href="cargo-install.html">Deploying a standalone Ballista cluster using cargo install</a></li>
-<li class="toctree-l1"><a class="reference internal" href="docker.html">Starting a Ballista cluster using Docker</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="docker.html#build-docker-image">Build Docker image</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cargo-install.html">Cargo Install</a></li>
+<li class="toctree-l1"><a class="reference internal" href="docker.html">Docker</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="docker.html#build-docker-images">Build Docker Images</a></li>
</ul>
</li>
-<li class="toctree-l1"><a class="reference internal" href="docker-compose.html">Starting a Ballista cluster using Docker Compose</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="docker-compose.html#build-docker-image">Build Docker image</a></li>
-<li class="toctree-l2"><a class="reference internal" href="docker-compose.html#start-a-cluster">Start a cluster</a></li>
+<li class="toctree-l1"><a class="reference internal" href="docker-compose.html">Docker Compose</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="docker-compose.html#build-docker-images">Build Docker Images</a></li>
+<li class="toctree-l2"><a class="reference internal" href="docker-compose.html#start-a-cluster">Start a Cluster</a></li>
</ul>
</li>
-<li class="toctree-l1"><a class="reference internal" href="kubernetes.html">Deploying Ballista with Kubernetes</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#testing-locally">Testing locally</a></li>
-<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#limitations">Limitations</a></li>
-<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#build-docker-image">Build Docker image</a></li>
-<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#publishing-your-images">Publishing your images</a></li>
+<li class="toctree-l1"><a class="reference internal" href="kubernetes.html">Kubernetes</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#testing-locally">Testing Locally</a></li>
+<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#build-docker-images">Build Docker Images</a></li>
+<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#publishing-docker-images">Publishing Docker Images</a></li>
<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#create-persistent-volume-and-persistent-volume-claim">Create Persistent Volume and Persistent Volume Claim</a></li>
-<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#deploying-ballista-scheduler-and-executors">Deploying Ballista Scheduler and Executors</a></li>
+<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#deploying-a-ballista-cluster">Deploying a Ballista Cluster</a></li>
<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#port-forwarding">Port Forwarding</a></li>
-<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#deleting-the-ballista-cluster">Deleting the Ballista cluster</a></li>
-<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#adding-autoscaling-for-executors">Adding autoscaling for executors</a></li>
+<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#deleting-the-ballista-cluster">Deleting the Ballista Cluster</a></li>
+<li class="toctree-l2"><a class="reference internal" href="kubernetes.html#autoscaling-executors">Autoscaling Executors</a></li>
</ul>
</li>
-<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
</ul>
</div>
</section>
diff --git a/ballista/user-guide/deployment/kubernetes.html b/ballista/user-guide/deployment/kubernetes.html
index 4db41153d2..b056e6b0d3 100644
--- a/ballista/user-guide/deployment/kubernetes.html
+++ b/ballista/user-guide/deployment/kubernetes.html
@@ -36,8 +36,8 @@
<script src="../../_static/doctools.js"></script>
<link rel="index" title="Index" href="../../genindex.html" />
<link rel="search" title="Search" href="../../search.html" />
- <link rel="next" title="Configuration" href="configuration.html" />
- <link rel="prev" title="Starting a Ballista cluster using Docker Compose" href="docker-compose.html" />
+ <link rel="next" title="Ballista Scheduler" href="../scheduler.html" />
+ <link rel="prev" title="Starting a Ballista Cluster using Docker Compose" href="docker-compose.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active has-children">
<a class="reference internal" href="index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input checked="" class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul class="current">
<li class="toctree-l2">
<a class="reference internal" href="cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2 current active">
<a class="current reference internal" href="#">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="../configs.html">
Configuration
@@ -209,22 +235,17 @@
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#testing-locally">
- Testing locally
+ Testing Locally
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#limitations">
- Limitations
+ <a class="reference internal nav-link" href="#build-docker-images">
+ Build Docker Images
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#build-docker-image">
- Build Docker image
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#publishing-your-images">
- Publishing your images
+ <a class="reference internal nav-link" href="#publishing-docker-images">
+ Publishing Docker Images
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -233,8 +254,8 @@
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#deploying-ballista-scheduler-and-executors">
- Deploying Ballista Scheduler and Executors
+ <a class="reference internal nav-link" href="#deploying-a-ballista-cluster">
+ Deploying a Ballista Cluster
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
@@ -244,12 +265,12 @@
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#deleting-the-ballista-cluster">
- Deleting the Ballista cluster
+ Deleting the Ballista Cluster
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#adding-autoscaling-for-executors">
- Adding autoscaling for executors
+ <a class="reference internal nav-link" href="#autoscaling-executors">
+ Autoscaling Executors
</a>
</li>
</ul>
@@ -311,39 +332,35 @@ you are already comfortable managing Kubernetes deployments.</p>
<li><p><em>(optional)</em> a <a class="reference external" href="http://keda.sh">keda</a> instance for autoscaling the number of executors</p></li>
</ul>
<section id="testing-locally">
-<h2>Testing locally<a class="headerlink" href="#testing-locally" title="Permalink to this heading">¶</a></h2>
+<h2>Testing Locally<a class="headerlink" href="#testing-locally" title="Permalink to this heading">¶</a></h2>
<p><a class="reference external" href="https://microk8s.io/">Microk8s</a> is recommended for installing a local k8s cluster. Once Microk8s is installed, DNS
must be enabled using the following command.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>microk8s <span class="nb">enable</span> dns
</pre></div>
</div>
</section>
-<section id="limitations">
-<h2>Limitations<a class="headerlink" href="#limitations" title="Permalink to this heading">¶</a></h2>
-<p>Ballista is at an early stage of development and therefore has some significant limitations:</p>
-<ul class="simple">
-<li><p>There is no support for shared object stores such as S3. All data must exist locally on each node in the
-cluster, including where any client process runs.</p></li>
-<li><p>Only a single scheduler instance is currently supported unless the scheduler is configured to use <code class="docutils literal notranslate"><span class="pre">etcd</span></code> as a
-backing store.</p></li>
-</ul>
-</section>
-<section id="build-docker-image">
-<h2>Build Docker image<a class="headerlink" href="#build-docker-image" title="Permalink to this heading">¶</a></h2>
-<p>There is no officially published Docker image so it is currently necessary to build the image from source instead.</p>
+<section id="build-docker-images">
+<h2>Build Docker Images<a class="headerlink" href="#build-docker-images" title="Permalink to this heading">¶</a></h2>
+<p>There are no officially published Docker images, so it is currently necessary to build the images from source.</p>
<p>Run the following commands to clone the source repository and build the Docker image.</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git clone git@github.com:apache/arrow-ballista.git
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git clone git@github.com:apache/arrow-ballista.git -b <span class="m">0</span>.9.0
<span class="nb">cd</span> arrow-ballista
./dev/build-ballista-docker.sh
</pre></div>
</div>
-<p>This will create an image with the tag <code class="docutils literal notranslate"><span class="pre">ballista:0.7.0</span></code>.</p>
+<p>This will create the following images:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">apache/arrow-ballista-scheduler:0.9.0</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">apache/arrow-ballista-executor:0.9.0</span></code></p></li>
+</ul>
</section>
-<section id="publishing-your-images">
-<h2>Publishing your images<a class="headerlink" href="#publishing-your-images" title="Permalink to this heading">¶</a></h2>
-<p>Once the images have been built, you can retag them and can push them to your favourite docker registry.</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker tag ballista:0.7.0 <your-repo>/ballista:0.7.0
-docker push <your-repo>/ballista:0.7.0
+<section id="publishing-docker-images">
+<h2>Publishing Docker Images<a class="headerlink" href="#publishing-docker-images" title="Permalink to this heading">¶</a></h2>
+<p>Once the images have been built, you can retag them and can push them to your favourite Docker registry.</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>docker tag apache/arrow-ballista-scheduler:0.9.0 <your-repo>/arrow-ballista-scheduler:0.9.0
+docker tag apache/arrow-ballista-executor:0.9.0 <your-repo>/arrow-ballista-executor:0.9.0
+docker push <your-repo>/arrow-ballista-scheduler:0.9.0
+docker push <your-repo>/arrow-ballista-executor:0.9.0
</pre></div>
</div>
</section>
@@ -390,8 +407,8 @@ persistentvolumeclaim/data-pv-claim created
</pre></div>
</div>
</section>
-<section id="deploying-ballista-scheduler-and-executors">
-<h2>Deploying Ballista Scheduler and Executors<a class="headerlink" href="#deploying-ballista-scheduler-and-executors" title="Permalink to this heading">¶</a></h2>
+<section id="deploying-a-ballista-cluster">
+<h2>Deploying a Ballista Cluster<a class="headerlink" href="#deploying-a-ballista-cluster" title="Permalink to this heading">¶</a></h2>
<p>Copy the following yaml to a <code class="docutils literal notranslate"><span class="pre">cluster.yaml</span></code> file and change <code class="docutils literal notranslate"><span class="pre"><your-image></span></code> with the name of your Ballista Docker image.</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">apiVersion</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">v1</span><span class="w"></span>
<span class="nt">kind</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Service</span><span class="w"></span>
@@ -403,6 +420,8 @@ persistentvolumeclaim/data-pv-claim created
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">50050</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">scheduler</span><span class="w"></span>
+<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span><span class="w"></span>
+<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">scheduler-ui</span><span class="w"></span>
<span class="w"> </span><span class="nt">selector</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">app</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ballista-scheduler</span><span class="w"></span>
<span class="nn">---</span><span class="w"></span>
@@ -423,12 +442,13 @@ persistentvolumeclaim/data-pv-claim created
<span class="w"> </span><span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">containers</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ballista-scheduler</span><span class="w"></span>
-<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain"><your-repo>/ballista:0.7.0</span><span class="w"></span>
-<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="s">"/scheduler"</span><span class="p p-Indicator">]</span><span class="w"></span>
+<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain"><your-repo>/arrow-ballista-scheduler:0.9.0</span><span class="w"></span>
<span class="w"> </span><span class="nt">args</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="s">"--bind-port=50050"</span><span class="p p-Indicator">]</span><span class="w"></span>
<span class="w"> </span><span class="nt">ports</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">50050</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">flight</span><span class="w"></span>
+<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">containerPort</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span><span class="w"></span>
+<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">webui</span><span class="w"></span>
<span class="w"> </span><span class="nt">volumeMounts</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">mountPath</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/mnt</span><span class="w"></span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">data</span><span class="w"></span>
@@ -454,8 +474,7 @@ persistentvolumeclaim/data-pv-claim created
<span class="w"> </span><span class="nt">spec</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">containers</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ballista-executor</span><span class="w"></span>
-<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain"><your-repo>/ballista:0.7.0</span><span class="w"></span>
-<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[</span><span class="s">"/executor"</span><span class="p p-Indicator">]</span><span class="w"></span>
+<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain"><your-repo>/arrow-ballista-executor:0.9.0</span><span class="w"></span>
<span class="w"> </span><span class="nt">args</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"--bind-port=50051"</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">"--scheduler-host=ballista-scheduler"</span><span class="w"></span>
@@ -514,14 +533,14 @@ ballista-scheduler ClusterIP <span class="m">10</span>.152.183.21 <none
</div>
</section>
<section id="deleting-the-ballista-cluster">
-<h2>Deleting the Ballista cluster<a class="headerlink" href="#deleting-the-ballista-cluster" title="Permalink to this heading">¶</a></h2>
+<h2>Deleting the Ballista Cluster<a class="headerlink" href="#deleting-the-ballista-cluster" title="Permalink to this heading">¶</a></h2>
<p>Run the following kubectl command to delete the cluster.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>kubectl delete -f cluster.yaml
</pre></div>
</div>
</section>
-<section id="adding-autoscaling-for-executors">
-<h2>Adding autoscaling for executors<a class="headerlink" href="#adding-autoscaling-for-executors" title="Permalink to this heading">¶</a></h2>
+<section id="autoscaling-executors">
+<h2>Autoscaling Executors<a class="headerlink" href="#autoscaling-executors" title="Permalink to this heading">¶</a></h2>
<p>Ballista supports autoscaling for executors through <a class="reference external" href="http://keda.sh">Keda</a>. Keda allows scaling a deployment
through custom metrics which are exposed through the Ballista scheduler, and it can even scale the number of
executors down to 0 if there is no activity in the cluster.</p>
@@ -569,13 +588,13 @@ scale the executors.</p>
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Starting a Ballista cluster using Docker Compose</p>
+ <p class="prev-next-title">Starting a Ballista Cluster using Docker Compose</p>
</div>
</a>
- <a class='right-next' id="next-link" href="configuration.html" title="next page">
+ <a class='right-next' id="next-link" href="../scheduler.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Configuration</p>
+ <p class="prev-next-title">Ballista Scheduler</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/ballista/user-guide/faq.html b/ballista/user-guide/faq.html
index 0bf3dd228a..26845a5714 100644
--- a/ballista/user-guide/faq.html
+++ b/ballista/user-guide/faq.html
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="configs.html">
Configuration
diff --git a/ballista/user-guide/flightsql.html b/ballista/user-guide/flightsql.html
new file mode 100644
index 0000000000..c453323bba
--- /dev/null
+++ b/ballista/user-guide/flightsql.html
@@ -0,0 +1,518 @@
+
+<!DOCTYPE html>
+
+<html lang="en">
+ <head>
+ <meta charset="utf-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />
+
+ <title>Using FlightSQL to Connect to Ballista — Arrow DataFusion documentation</title>
+
+ <link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
+
+
+ <link rel="stylesheet"
+ href="../_static/vendor/fontawesome/5.13.0/css/all.min.css">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
+ <link rel="preload" as="font" type="font/woff2" crossorigin
+ href="../_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
+
+
+
+
+
+ <link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
+ <link rel="stylesheet" type="text/css" href="../_static/styles/pydata-sphinx-theme.css" />
+ <link rel="stylesheet" type="text/css" href="../_static/theme_overrides.css" />
+
+ <link rel="preload" as="script" href="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf">
+
+ <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
+ <script src="../_static/jquery.js"></script>
+ <script src="../_static/underscore.js"></script>
+ <script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
+ <script src="../_static/doctools.js"></script>
+ <link rel="index" title="Index" href="../genindex.html" />
+ <link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="Ballista Command-line Interface" href="cli.html" />
+ <link rel="prev" title="Ballista Rust Client" href="rust.html" />
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
+ <meta name="docsearch:language" content="en">
+
+
+ <!-- Google Analytics -->
+
+ </head>
+ <body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
+
+ <div class="container-fluid" id="banner"></div>
+
+
+
+
+ <div class="container-xl">
+ <div class="row">
+
+
+ <!-- Only show if we have sidebars configured, else just a small margin -->
+ <div class="col-12 col-md-3 bd-sidebar">
+ <div class="sidebar-start-items">
+<a class="navbar-brand" href="../index.html">
+ <img src="../_static/images/ballista-logo.png" class="logo" alt="logo">
+</a>
+
+<form class="bd-search d-flex align-items-center" action="../search.html" method="get">
+ <i class="icon fas fa-search"></i>
+ <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
+</form>
+
+<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
+ <div class="bd-toc-item active">
+
+ <p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ User Guide
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="introduction.html">
+ Introduction
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1 has-children">
+ <a class="reference internal" href="deployment/index.html">
+ Deployment
+ </a>
+ <input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
+ <label for="toctree-checkbox-1">
+ <i class="fas fa-chevron-down">
+ </i>
+ </label>
+ <ul>
+ <li class="toctree-l2">
+ <a class="reference internal" href="deployment/cargo-install.html">
+ Cargo Install
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="deployment/docker.html">
+ Docker
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="deployment/docker-compose.html">
+ Docker Compose
+ </a>
+ </li>
+ <li class="toctree-l2">
+ <a class="reference internal" href="deployment/kubernetes.html">
+ Kubernetes
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="python.html">
+ Python
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="rust.html">
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Flight SQL JDBC
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="cli.html">
+ SQL CLI
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="configs.html">
+ Configuration
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="tuning-guide.html">
+ Tuning Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="faq.html">
+ Frequently Asked Questions
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Source Code
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-ballista/">
+ Ballista
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Community
+ </span>
+</p>
+<ul class="nav bd-sidenav">
+ <li class="toctree-l1">
+ <a class="reference internal" href="../community/communication.html">
+ Communication
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-ballista/issues">
+ Issue tracker
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external" href="https://github.com/apache/arrow-ballista/blob/master/CODE_OF_CONDUCT.md">
+ Code of conduct
+ </a>
+ </li>
+</ul>
+
+
+ </div>
+</nav>
+ </div>
+ <div class="sidebar-end-items">
+ </div>
+ </div>
+
+
+
+
+ <div class="d-none d-xl-block col-xl-2 bd-toc">
+
+
+ <div class="toc-item">
+
+<div class="tocsection onthispage pt-5 pb-3">
+ <i class="fas fa-list"></i> On this page
+</div>
+
+<nav id="bd-toc-nav">
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#a-name-prereq-prerequisites">
+ <a name="prereq">
+ </a>
+ Prerequisites
+ </a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#ubuntu">
+ Ubuntu
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#macos">
+ MacOS
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#windows">
+ Windows
+ </a>
+ </li>
+ </ul>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#a-name-rust-building-ballista">
+ <a name="rust">
+ </a>
+ Building Ballista
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#a-name-docker-run-docker-containers">
+ <a name="docker">
+ </a>
+ Run Docker Containers
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#a-name-jdbc-build-the-flightsql-jdbc-driver">
+ <a name="jdbc">
+ </a>
+ Build the FlightSQL JDBC Driver
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#a-name-tool-use-the-driver-in-your-favorite-data-tool">
+ <a name="tool">
+ </a>
+ Use the Driver in your Favorite Data Tool
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#a-name-hello-run-a-hello-world-query">
+ <a name="hello">
+ </a>
+ Run a “Hello, World!” Query
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#a-name-complex-run-a-complex-query">
+ <a name="complex">
+ </a>
+ Run a Complex Query
+ </a>
+ </li>
+</ul>
+
+</nav>
+ </div>
+
+ <div class="toc-item">
+
+
+<div class="tocsection editthispage">
+ <a href="https://github.com/apache/arrow-datafusion/edit/master/docs/source/user-guide/flightsql.md">
+ <i class="fas fa-pencil-alt"></i> Edit this page
+ </a>
+</div>
+
+ </div>
+
+
+ </div>
+
+
+
+
+
+
+ <main class="col-12 col-md-9 col-xl-7 py-md-5 pl-md-5 pr-md-4 bd-content" role="main">
+
+ <div>
+
+ <!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<section id="using-flightsql-to-connect-to-ballista">
+<h1>Using FlightSQL to Connect to Ballista<a class="headerlink" href="#using-flightsql-to-connect-to-ballista" title="Permalink to this heading">¶</a></h1>
+<p>One of the easiest ways to start with Ballista is to plug it into your existing data infrastructure using support for Arrow Flight SQL JDBC.</p>
+<p>Getting started involves these main steps:</p>
+<ol class="arabic simple">
+<li><p><span class="xref myst">Installing prerequisites</span></p></li>
+<li><p>Build the <span class="xref myst">Ballista rust code</span></p></li>
+<li><p>Build and run the <span class="xref myst">Ballista docker containers</span></p></li>
+<li><p>Build the <span class="xref myst">Arrow Flight SQL JDBC Driver</span></p></li>
+<li><p><span class="xref myst">Install the driver</span> into your favorite JDBC tool</p></li>
+<li><p>Run a <span class="xref myst">“hello, world!”</span> query</p></li>
+<li><p>Register a table and run more complicated queries</p></li>
+</ol>
+<section id="a-name-prereq-prerequisites">
+<h2><a name="prereq"/>Prerequisites<a class="headerlink" href="#a-name-prereq-prerequisites" title="Permalink to this heading">¶</a></h2>
+<section id="ubuntu">
+<h3>Ubuntu<a class="headerlink" href="#ubuntu" title="Permalink to this heading">¶</a></h3>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>sudo apt-get update
+sudo apt-get install -y docker.io docker-compose
+</pre></div>
+</div>
+</section>
+<section id="macos">
+<h3>MacOS<a class="headerlink" href="#macos" title="Permalink to this heading">¶</a></h3>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>brew install docker docker-compose
+</pre></div>
+</div>
+</section>
+<section id="windows">
+<h3>Windows<a class="headerlink" href="#windows" title="Permalink to this heading">¶</a></h3>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>choco install docker-desktop docker-compose
+</pre></div>
+</div>
+</section>
+</section>
+<section id="a-name-rust-building-ballista">
+<h2><a name="rust"/>Building Ballista<a class="headerlink" href="#a-name-rust-building-ballista" title="Permalink to this heading">¶</a></h2>
+<p>To build in docker (non-linux systems):</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>git clone https://github.com/apache/arrow-ballista.git
+dev/build-ballista-rust.sh
+</pre></div>
+</div>
+<p>Or in linux-based systems with the correct dependencies installed, one can simply:</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>cargo build --release --all --features flight-sql
+</pre></div>
+</div>
+</section>
+<section id="a-name-docker-run-docker-containers">
+<h2><a name="docker"/> Run Docker Containers<a class="headerlink" href="#a-name-docker-run-docker-containers" title="Permalink to this heading">¶</a></h2>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>docker-compose up --build
+</pre></div>
+</div>
+</section>
+<section id="a-name-jdbc-build-the-flightsql-jdbc-driver">
+<h2><a name="jdbc"/>Build the FlightSQL JDBC Driver<a class="headerlink" href="#a-name-jdbc-build-the-flightsql-jdbc-driver" title="Permalink to this heading">¶</a></h2>
+<p><strong>Note:</strong> this will no longer be necessary when Arrow v10 is released approximately 2022-10-31</p>
+<p><strong>Note:</strong> A full explaination of the Arrow Java build is out-of-scope for this document. Please refer to that project for more detailed instructions.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>git clone https://github.com/apache/arrow.git
+<span class="nb">cd</span> java
+mvn install -DskipTests -Dcheckstyle.skip -Drat.skip<span class="o">=</span><span class="nb">true</span> -pl :flight-sql-jdbc-driver -am
+find . -name <span class="s2">"*.jar"</span>
+...
+./flight/flight-sql-jdbc-driver/target/flight-sql-jdbc-driver-10.0.0-SNAPSHOT.jar
+...
+</pre></div>
+</div>
+</section>
+<section id="a-name-tool-use-the-driver-in-your-favorite-data-tool">
+<h2><a name="tool"/>Use the Driver in your Favorite Data Tool<a class="headerlink" href="#a-name-tool-use-the-driver-in-your-favorite-data-tool" title="Permalink to this heading">¶</a></h2>
+<p>The important pieces of information:</p>
+<table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>Key</p></th>
+<th class="head"><p>Value</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>Driver file</p></td>
+<td><p>flight-sql-jdbc-driver-10.0.0-SNAPSHOT.jar</p></td>
+</tr>
+<tr class="row-odd"><td><p>Class Name</p></td>
+<td><p>org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver</p></td>
+</tr>
+<tr class="row-even"><td><p>Authentication</p></td>
+<td><p>User & Password</p></td>
+</tr>
+<tr class="row-odd"><td><p>Username</p></td>
+<td><p>admin</p></td>
+</tr>
+<tr class="row-even"><td><p>Password</p></td>
+<td><p>password</p></td>
+</tr>
+<tr class="row-odd"><td><p>Advanced Options</p></td>
+<td><p>useEncryption=false</p></td>
+</tr>
+<tr class="row-even"><td><p>URL</p></td>
+<td><p>jdbc:arrow-flight://127.0.0.1:50050</p></td>
+</tr>
+</tbody>
+</table>
+</section>
+<section id="a-name-hello-run-a-hello-world-query">
+<h2><a name="hello"/>Run a “Hello, World!” Query<a class="headerlink" href="#a-name-hello-run-a-hello-world-query" title="Permalink to this heading">¶</a></h2>
+<div class="highlight-sql notranslate"><div class="highlight"><pre><span></span><span class="k">select</span><span class="w"> </span><span class="s1">'Hello from Arrow Ballista!'</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">greeting</span><span class="p">;</span><span class="w"></span>
+</pre></div>
+</div>
+</section>
+<section id="a-name-complex-run-a-complex-query">
+<h2><a name="complex"/>Run a Complex Query<a class="headerlink" href="#a-name-complex-run-a-complex-query" title="Permalink to this heading">¶</a></h2>
+<p>In order to run queries against data, tables need to be “registered” with the current session (and re-registered upon each new connection).</p>
+<p>To register a table, find a <code class="docutils literal notranslate"><span class="pre">.csv</span></code>, <code class="docutils literal notranslate"><span class="pre">.json</span></code>, or <code class="docutils literal notranslate"><span class="pre">.parquet</span></code> file for testing, and use the syntax below:</p>
+<div class="highlight-sql notranslate"><div class="highlight"><pre><span></span><span class="k">create</span><span class="w"> </span><span class="k">external</span><span class="w"> </span><span class="k">table</span><span class="w"> </span><span class="n">customer</span><span class="w"> </span><span class="n">stored</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">CSV</span><span class="w"> </span><span class="k">with</span><span class="w"> < [...]
+<span class="w"> </span><span class="k">location</span><span class="w"> </span><span class="s1">'/path/to/customer.csv'</span><span class="p">;</span><span class="w"></span>
+</pre></div>
+</div>
+<p>Once the table has been registered, all the normal SQL queries can be performed:</p>
+<div class="highlight-sql notranslate"><div class="highlight"><pre><span></span><span class="k">select</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">customer</span><span class="p">;</span><span class="w"></span>
+</pre></div>
+</div>
+<p>🎉 Happy querying! 🎉</p>
+</section>
+</section>
+
+
+ </div>
+
+
+ <!-- Previous / next buttons -->
+<div class='prev-next-area'>
+ <a class='left-prev' id="prev-link" href="rust.html" title="previous page">
+ <i class="fas fa-angle-left"></i>
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">previous</p>
+ <p class="prev-next-title">Ballista Rust Client</p>
+ </div>
+ </a>
+ <a class='right-next' id="next-link" href="cli.html" title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Ballista Command-line Interface</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
+</div>
+
+ </main>
+
+
+ </div>
+ </div>
+
+ <script src="../_static/scripts/pydata-sphinx-theme.js?digest=1999514e3f237ded88cf"></script>
+ <footer class="footer mt-5 mt-md-0">
+ <div class="container">
+
+ <div class="footer-item">
+ <p class="copyright">
+ © Copyright 2022, Apache Software Foundation.<br>
+</p>
+ </div>
+
+ <div class="footer-item">
+ <p class="sphinx-version">
+Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.1.1.<br>
+</p>
+ </div>
+
+ </div>
+</footer>
+ </body>
+</html>
\ No newline at end of file
diff --git a/ballista/user-guide/introduction.html b/ballista/user-guide/introduction.html
index d21eb0f663..40c9b2cb40 100644
--- a/ballista/user-guide/introduction.html
+++ b/ballista/user-guide/introduction.html
@@ -79,12 +79,19 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="configs.html">
Configuration
@@ -212,11 +238,6 @@
How does this compare to Apache Spark?
</a>
</li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#status">
- Status
- </a>
- </li>
</ul>
</nav>
@@ -265,16 +286,15 @@
-->
<section id="overview">
<h1>Overview<a class="headerlink" href="#overview" title="Permalink to this heading">¶</a></h1>
-<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow. It is
-built on an architecture that allows other programming languages to be supported as first-class citizens without paying
-a penalty for serialization costs.</p>
-<p>The foundational technologies in Ballista are:</p>
-<ul class="simple">
-<li><p><a class="reference external" href="https://arrow.apache.org/">Apache Arrow</a> memory model and compute kernels for efficient processing of data.</p></li>
-<li><p><a class="reference external" href="https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/">Apache Arrow Flight Protocol</a> for efficient data transfer between processes.</p></li>
-<li><p><a class="reference external" href="https://developers.google.com/protocol-buffers">Google Protocol Buffers</a> for serializing query plans.</p></li>
-<li><p><a class="reference external" href="https://github.com/apache/arrow-datafusion/">DataFusion</a> for query execution.</p></li>
-</ul>
+<p>Ballista is a distributed compute platform primarily implemented in Rust, and powered by Apache Arrow.</p>
+<p>Ballista has a scheduler and an executor process that are standard Rust executables and can be executed directly, but
+Dockerfiles are provided to build images for use in containerized environments, such as Docker, Docker Compose, and
+Kubernetes. See the <span class="xref myst">deployment guide</span> for more information</p>
+<p>SQL and DataFrame queries can be submitted from Python and Rust, and SQL queries can be submitted via the Arrow
+Flight SQL JDBC driver, supporting your favorite JDBC compliant tools such as <span class="xref myst">DataGrip</span>
+or <span class="xref myst">tableau</span>. For setup instructions, please see the <a class="reference internal" href="flightsql.html"><span class="doc std std-doc">FlightSQL guide</span></a>.</p>
+<p>The scheduler has a web user interface for monitoring query status as well as a REST API.</p>
+<p><img alt="Ballista Scheduler Web UI" src="../_images/ballista-web-ui.png" /></p>
<section id="how-does-this-compare-to-apache-spark">
<h2>How does this compare to Apache Spark?<a class="headerlink" href="#how-does-this-compare-to-apache-spark" title="Permalink to this heading">¶</a></h2>
<p>Although Ballista is largely inspired by Apache Spark, there are some key differences.</p>
@@ -290,10 +310,9 @@ distributed compute.</p></li>
<li><p>The use of Apache Arrow as the memory model and network protocol means that data can be exchanged between executors
in any programming language with minimal serialization overhead.</p></li>
</ul>
-</section>
-<section id="status">
-<h2>Status<a class="headerlink" href="#status" title="Permalink to this heading">¶</a></h2>
-<p>Ballista is still in the early stages of development but is capable of executing complex analytical queries at scale.</p>
+<p><span class="xref myst">deployment</span>
+<a class="reference external" href="https://www.jetbrains.com/datagrip/">datagrip</a>
+<a class="reference external" href="https://help.tableau.com/current/pro/desktop/en-us/examples_otherdatabases_jdbc.htm">tableau</a></p>
</section>
</section>
diff --git a/ballista/user-guide/python.html b/ballista/user-guide/python.html
index 8b14ca9013..3d47864941 100644
--- a/ballista/user-guide/python.html
+++ b/ballista/user-guide/python.html
@@ -37,7 +37,7 @@
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Ballista Rust Client" href="rust.html" />
- <link rel="prev" title="Configuration" href="deployment/configuration.html" />
+ <link rel="prev" title="Ballista Scheduler" href="scheduler.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="configs.html">
Configuration
@@ -213,15 +239,20 @@
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#registering-tables">
- Registering Tables
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#executing-queries">
- Executing Queries
+ <a class="reference internal nav-link" href="#sql">
+ SQL
</a>
<ul class="nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#registering-tables">
+ Registering Tables
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#executing-queries">
+ Executing Queries
+ </a>
+ </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#showing-query-results">
Showing Query Results
@@ -239,6 +270,16 @@
</li>
</ul>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#dataframe">
+ DataFrame
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#user-defined-functions">
+ User Defined Functions
+ </a>
+ </li>
</ul>
</nav>
@@ -288,6 +329,8 @@
<section id="ballista-python-bindings">
<h1>Ballista Python Bindings<a class="headerlink" href="#ballista-python-bindings" title="Permalink to this heading">¶</a></h1>
<p>Ballista provides Python bindings, allowing SQL and DataFrame queries to be executed from the Python shell.</p>
+<p>Like PySpark, it allows you to build a plan through SQL or a DataFrame API against Parquet, CSV, JSON, and other
+popular file formats files, run it in a distributed environment, and obtain the result back in Python.</p>
<section id="connecting-to-a-cluster">
<h2>Connecting to a Cluster<a class="headerlink" href="#connecting-to-a-cluster" title="Permalink to this heading">¶</a></h2>
<p>The following code demonstrates how to create a Ballista context and connect to a scheduler.</p>
@@ -296,8 +339,12 @@
</pre></div>
</div>
</section>
+<section id="sql">
+<h2>SQL<a class="headerlink" href="#sql" title="Permalink to this heading">¶</a></h2>
+<p>The Python bindings support executing SQL queries as well.</p>
<section id="registering-tables">
-<h2>Registering Tables<a class="headerlink" href="#registering-tables" title="Permalink to this heading">¶</a></h2>
+<h3>Registering Tables<a class="headerlink" href="#registering-tables" title="Permalink to this heading">¶</a></h3>
+<p>Before SQL queries can be executed, tables need to be registered with the context.</p>
<p>Tables can be registered against the context by calling one of the <code class="docutils literal notranslate"><span class="pre">register</span></code> methods, or by executing SQL.</p>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>>>> ctx.register_parquet("trips", "/mnt/bigdata/nyctaxi")
</pre></div>
@@ -307,8 +354,9 @@
</div>
</section>
<section id="executing-queries">
-<h2>Executing Queries<a class="headerlink" href="#executing-queries" title="Permalink to this heading">¶</a></h2>
+<h3>Executing Queries<a class="headerlink" href="#executing-queries" title="Permalink to this heading">¶</a></h3>
<p>The <code class="docutils literal notranslate"><span class="pre">sql</span></code> method creates a <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code>. The query is executed when an action such as <code class="docutils literal notranslate"><span class="pre">show</span></code> or <code class="docutils literal notranslate"><span class="pre">collect</span></code> is executed.</p>
+</section>
<section id="showing-query-results">
<h3>Showing Query Results<a class="headerlink" href="#showing-query-results" title="Permalink to this heading">¶</a></h3>
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>>>> df = ctx.sql("SELECT count(*) FROM trips")
@@ -351,6 +399,44 @@ COUNT(UInt8(1)): int64]
</div>
</section>
</section>
+<section id="dataframe">
+<h2>DataFrame<a class="headerlink" href="#dataframe" title="Permalink to this heading">¶</a></h2>
+<p>The following example demonstrates creating arrays with PyArrow and then creating a Ballista DataFrame.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">ballista</span>
+<span class="kn">import</span> <span class="nn">pyarrow</span>
+
+<span class="c1"># an alias</span>
+<span class="n">f</span> <span class="o">=</span> <span class="n">ballista</span><span class="o">.</span><span class="n">functions</span>
+
+<span class="c1"># create a context</span>
+<span class="n">ctx</span> <span class="o">=</span> <span class="n">ballista</span><span class="o">.</span><span class="n">BallistaContext</span><span class="p">(</span><span class="s2">"localhost"</span><span class="p">,</span> <span class="mi">50050</span><span class="p">)</span>
+
+<span class="c1"># create a RecordBatch and a new DataFrame from it</span>
+<span class="n">batch</span> <span class="o">=</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">RecordBatch</span><span class="o">.</span><span class="n">from_arrays</span><span class="p">(</span>
+ <span class="p">[</span><span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]),</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span [...]
+ <span class="n">names</span><span class="o">=</span><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">],</span>
+<span class="p">)</span>
+<span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">create_dataframe</span><span class="p">([[</span><span class="n">batch</span><span class="p">]])</span>
+
+<span class="c1"># create a new statement</span>
+<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">+</span> <span class="n">f</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">-</span> <span class="n">f</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span>
+<span class="p">)</span>
+
+<span class="c1"># execute and collect the first (and only) batch</span>
+<span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">collect</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
+
+<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">5</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">9</span><span class="p">])</span>
+<span class="k">assert</span> <span class="n">result</span><span class="o">.</span><span class="n">column</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="n">pyarrow</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="o">-</span>< [...]
+</pre></div>
+</div>
+</section>
+<section id="user-defined-functions">
+<h2>User Defined Functions<a class="headerlink" href="#user-defined-functions" title="Permalink to this heading">¶</a></h2>
+<p>The underlying DataFusion query engine supports Python UDFs but this functionality has not yet been implemented in
+Ballista. It is planned for a future release. The tracking issue is <a class="reference external" href="https://github.com/apache/arrow-ballista/issues/173">#173</a>.</p>
+</section>
</section>
@@ -359,11 +445,11 @@ COUNT(UInt8(1)): int64]
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="deployment/configuration.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="scheduler.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Configuration</p>
+ <p class="prev-next-title">Ballista Scheduler</p>
</div>
</a>
<a class='right-next' id="next-link" href="rust.html" title="next page">
diff --git a/ballista/user-guide/rust.html b/ballista/user-guide/rust.html
index efecee3df5..f1c859ad27 100644
--- a/ballista/user-guide/rust.html
+++ b/ballista/user-guide/rust.html
@@ -36,7 +36,7 @@
<script src="../_static/doctools.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="DataFusion Command-line Interface" href="cli.html" />
+ <link rel="next" title="Using FlightSQL to Connect to Ballista" href="flightsql.html" />
<link rel="prev" title="Ballista Python Bindings" href="python.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="configs.html">
Configuration
@@ -250,9 +276,7 @@
-->
<section id="ballista-rust-client">
<h1>Ballista Rust Client<a class="headerlink" href="#ballista-rust-client" title="Permalink to this heading">¶</a></h1>
-<p>Ballista usage is very similar to DataFusion. Tha main difference is that the starting point is a <code class="docutils literal notranslate"><span class="pre">BallistaContext</span></code>
-instead of the DataFusion <code class="docutils literal notranslate"><span class="pre">SessionContext</span></code>. Ballista uses the same DataFrame API as DataFusion.</p>
-<p>The following code sample demonstrates how to create a <code class="docutils literal notranslate"><span class="pre">BallistaContext</span></code> to connect to a Ballista scheduler process.</p>
+<p>To connect to a Ballista cluster from Rust, first start by creating a <code class="docutils literal notranslate"><span class="pre">BallistaContext</span></code>.</p>
<div class="highlight-rust notranslate"><div class="highlight"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BallistaConfig</span>::<span class="n">builder</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">"ballista.shuffle.partitions"</span><span class="p">,</span><span class="w"> </span><span class="s">"4"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">build</span><span class="p">()</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
@@ -337,10 +361,10 @@ instead of the DataFusion <code class="docutils literal notranslate"><span class
<p class="prev-next-title">Ballista Python Bindings</p>
</div>
</a>
- <a class='right-next' id="next-link" href="cli.html" title="next page">
+ <a class='right-next' id="next-link" href="flightsql.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">DataFusion Command-line Interface</p>
+ <p class="prev-next-title">Using FlightSQL to Connect to Ballista</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/ballista/user-guide/faq.html b/ballista/user-guide/scheduler.html
similarity index 72%
copy from ballista/user-guide/faq.html
copy to ballista/user-guide/scheduler.html
index 0bf3dd228a..55db1067a8 100644
--- a/ballista/user-guide/faq.html
+++ b/ballista/user-guide/scheduler.html
@@ -6,7 +6,7 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />
- <title>Frequently Asked Questions — Arrow DataFusion documentation</title>
+ <title>Ballista Scheduler — Arrow DataFusion documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
<link href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf" rel="stylesheet">
@@ -36,8 +36,8 @@
<script src="../_static/doctools.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Communication" href="../community/communication.html" />
- <link rel="prev" title="Tuning Guide" href="tuning-guide.html" />
+ <link rel="next" title="Ballista Python Bindings" href="python.html" />
+ <link rel="prev" title="Deploying Ballista with Kubernetes" href="deployment/kubernetes.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="configs.html">
Configuration
@@ -144,8 +170,8 @@
Tuning Guide
</a>
</li>
- <li class="toctree-l1 current active">
- <a class="current reference internal" href="#">
+ <li class="toctree-l1">
+ <a class="reference internal" href="faq.html">
Frequently Asked Questions
</a>
</li>
@@ -208,8 +234,13 @@
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#what-is-the-relationship-between-apache-arrow-datafusion-and-ballista">
- What is the relationship between Apache Arrow, DataFusion, and Ballista?
+ <a class="reference internal nav-link" href="#web-user-interface">
+ Web User Interface
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#rest-api">
+ REST API
</a>
</li>
</ul>
@@ -221,7 +252,7 @@
<div class="tocsection editthispage">
- <a href="https://github.com/apache/arrow-datafusion/edit/master/docs/source/user-guide/faq.md">
+ <a href="https://github.com/apache/arrow-datafusion/edit/master/docs/source/user-guide/scheduler.md">
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -258,16 +289,42 @@
specific language governing permissions and limitations
under the License.
-->
-<section id="frequently-asked-questions">
-<h1>Frequently Asked Questions<a class="headerlink" href="#frequently-asked-questions" title="Permalink to this heading">¶</a></h1>
-<section id="what-is-the-relationship-between-apache-arrow-datafusion-and-ballista">
-<h2>What is the relationship between Apache Arrow, DataFusion, and Ballista?<a class="headerlink" href="#what-is-the-relationship-between-apache-arrow-datafusion-and-ballista" title="Permalink to this heading">¶</a></h2>
-<p>Apache Arrow is a library which provides a standardized memory representation for columnar data. It also provides
-“kernels” for performing common operations on this data.</p>
-<p>DataFusion is a library for executing queries in-process using the Apache Arrow memory
-model and computational kernels. It is designed to run within a single process, using threads
-for parallel query execution.</p>
-<p>Ballista is a distributed compute platform built on DataFusion.</p>
+<section id="ballista-scheduler">
+<h1>Ballista Scheduler<a class="headerlink" href="#ballista-scheduler" title="Permalink to this heading">¶</a></h1>
+<section id="web-user-interface">
+<h2>Web User Interface<a class="headerlink" href="#web-user-interface" title="Permalink to this heading">¶</a></h2>
+<p>The scheduler provides a web user interface that allows queries to be monitored.</p>
+<p><img alt="Ballista Scheduler Web UI" src="../_images/ballista-web-ui.png" /></p>
+</section>
+<section id="rest-api">
+<h2>REST API<a class="headerlink" href="#rest-api" title="Permalink to this heading">¶</a></h2>
+<p>The scheduler also provides a REST API that allows jobs to be monitored.</p>
+<table class="table">
+<thead>
+<tr class="row-odd"><th class="head"><p>API</p></th>
+<th class="head"><p>Method</p></th>
+<th class="head"><p>Description</p></th>
+</tr>
+</thead>
+<tbody>
+<tr class="row-even"><td><p>/api/jobs</p></td>
+<td><p>GET</p></td>
+<td><p>Get a list of jobs that have been submitted to the cluster.</p></td>
+</tr>
+<tr class="row-odd"><td><p>/api/job/{job_id}</p></td>
+<td><p>GET</p></td>
+<td><p>Get a summary of a submitted job.</p></td>
+</tr>
+<tr class="row-even"><td><p>/api/job/{job_id}/dot</p></td>
+<td><p>GET</p></td>
+<td><p>Produce a query plan in DOT (graphviz) format.</p></td>
+</tr>
+<tr class="row-odd"><td><p>/api/job/{job_id}</p></td>
+<td><p>PATCH</p></td>
+<td><p>Cancel a currently running job</p></td>
+</tr>
+</tbody>
+</table>
</section>
</section>
@@ -277,17 +334,17 @@ for parallel query execution.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="tuning-guide.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="deployment/kubernetes.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Tuning Guide</p>
+ <p class="prev-next-title">Deploying Ballista with Kubernetes</p>
</div>
</a>
- <a class='right-next' id="next-link" href="../community/communication.html" title="next page">
+ <a class='right-next' id="next-link" href="python.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Communication</p>
+ <p class="prev-next-title">Ballista Python Bindings</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/ballista/user-guide/tuning-guide.html b/ballista/user-guide/tuning-guide.html
index 27f4ede44f..4d23b45e57 100644
--- a/ballista/user-guide/tuning-guide.html
+++ b/ballista/user-guide/tuning-guide.html
@@ -76,15 +76,22 @@
User Guide
</span>
</p>
-<ul class="current nav bd-sidenav">
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="introduction.html">
- Overview
+ Introduction
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Cluster Deployment
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1 has-children">
<a class="reference internal" href="deployment/index.html">
- Start a Ballista Cluster
+ Deployment
</a>
<input class="toctree-checkbox" id="toctree-checkbox-1" name="toctree-checkbox-1" type="checkbox"/>
<label for="toctree-checkbox-1">
@@ -94,46 +101,65 @@
<ul>
<li class="toctree-l2">
<a class="reference internal" href="deployment/cargo-install.html">
- Deploying a standalone Ballista cluster using cargo install
+ Cargo Install
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker.html">
- Starting a Ballista cluster using Docker
+ Docker
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/docker-compose.html">
- Starting a Ballista cluster using Docker Compose
+ Docker Compose
</a>
</li>
<li class="toctree-l2">
<a class="reference internal" href="deployment/kubernetes.html">
- Deploying Ballista with Kubernetes
- </a>
- </li>
- <li class="toctree-l2">
- <a class="reference internal" href="deployment/configuration.html">
- Configuration
+ Kubernetes
</a>
</li>
</ul>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="scheduler.html">
+ Scheduler
+ </a>
+ </li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Clients
+ </span>
+</p>
+<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="python.html">
- Ballista Python Bindings
+ Python
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="rust.html">
- Ballista Rust Client
+ Rust
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="flightsql.html">
+ Flight SQL JDBC
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="cli.html">
- DataFusion Command-line Interface
+ SQL CLI
</a>
</li>
+</ul>
+<p aria-level="2" class="caption" role="heading">
+ <span class="caption-text">
+ Reference
+ </span>
+</p>
+<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="configs.html">
Configuration
@@ -222,6 +248,11 @@
Push-based vs Pull-based Task Scheduling
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#viewing-query-plans-and-metrics">
+ Viewing Query Plans and Metrics
+ </a>
+ </li>
</ul>
</nav>
@@ -312,6 +343,21 @@ which is the best for your use case.</p>
<p>The scheduling policy can be specified in the <code class="docutils literal notranslate"><span class="pre">--scheduler_policy</span></code> parameter when starting the scheduler and executor
processes. The default is <code class="docutils literal notranslate"><span class="pre">pull-based</span></code>.</p>
</section>
+<section id="viewing-query-plans-and-metrics">
+<h2>Viewing Query Plans and Metrics<a class="headerlink" href="#viewing-query-plans-and-metrics" title="Permalink to this heading">¶</a></h2>
+<p>The scheduler provides a web user interface as well as a REST API for monitoring jobs. See the
+<a class="reference internal" href="scheduler.html"><span class="doc std std-doc">scheduler documentation</span></a> for more information.</p>
+<p>To download a query plan in dot format from the scheduler, submit a request to the following API endpoint</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">localhost</span><span class="p">:</span><span class="mi">50050</span><span class="o">/</span><span class="n">api</span><span class="o">/</span><span class="n">job</span><span class="o">/</span><span class="p">{</span><span class="n">job_id</span><span class="p">}</span><span class="o">/</span><span class="n">dot</span>
+</pre></div>
+</div>
+<p>The resulting file can be converted into an image using <code class="docutils literal notranslate"><span class="pre">graphviz</span></code>:</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>dot -Tpng query.dot > query.png
+</pre></div>
+</div>
+<p>Here is an example query plan:</p>
+<p><img alt="query plan" src="../_images/example-query-plan.png" /></p>
+</section>
</section>