You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ag...@apache.org on 2022/05/12 14:48:03 UTC
[arrow-datafusion] branch master updated: Add new `ballista-cli` crate (#2495)
This is an automated email from the ASF dual-hosted git repository.
agrove pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/master by this push:
new 80f7bbf4a Add new `ballista-cli` crate (#2495)
80f7bbf4a is described below
commit 80f7bbf4ab183c8ec551fc9c26a4365fca65b68c
Author: Andy Grove <ag...@apache.org>
AuthorDate: Thu May 12 08:47:57 2022 -0600
Add new `ballista-cli` crate (#2495)
* Add new ballista-cli crate
* update dependency diagram
* re-use PrintFormat from datafusion-cli
* re-use PrintOptions from datafusion-cli
* re-use Helper
* re-use functions
* update dev scripts
* update diagram and docs
* stop building DataFusion CLI with ballista in CI
* update user guide
* docs for building ballista-cli with docker
* make version numbers consistent with repo
* update Cargo.lock files for CLIs and add ballista-cli to GitHub workflow
* disable ballista tests
* fix ci
* fix merge conflict
* fix
---
.github/workflows/rust.yml | 48 +-
.gitignore | 1 +
Cargo.toml | 2 +-
{datafusion-cli => ballista-cli}/Cargo.lock | 108 ++--
{datafusion-cli => ballista-cli}/Cargo.toml | 11 +-
Cargo.toml => ballista-cli/Dockerfile | 44 +-
{datafusion-cli => ballista-cli}/README.md | 28 +-
ballista-cli/src/command.rs | 222 +++++++
{datafusion-cli => ballista-cli}/src/context.rs | 0
ballista-cli/src/exec.rs | 170 +++++
ballista-cli/src/lib.rs | 25 +
{datafusion-cli => ballista-cli}/src/main.rs | 10 +-
datafusion-cli/Cargo.lock | 709 +--------------------
datafusion-cli/Cargo.toml | 3 +-
datafusion-cli/README.md | 2 +-
datafusion-cli/src/context.rs | 5 -
datafusion-cli/src/helper.rs | 2 +-
datafusion-cli/src/main.rs | 11 +-
dev/docker/ballista.dockerfile | 3 +
dev/release/README.md | 5 +-
dev/release/crate-deps.dot | 5 +-
dev/release/crate-deps.svg | 186 +++---
dev/update_ballista_versions.py | 4 +-
docs/source/cli/index.rst | 49 +-
docs/source/user-guide/cli.md | 2 +-
.../distributed/clients/cli.rst} | 74 +--
.../user-guide/distributed/clients/index.rst | 1 +
27 files changed, 747 insertions(+), 983 deletions(-)
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index 056e83df0..4a4e32f90 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -75,14 +75,6 @@ jobs:
env:
CARGO_HOME: "/github/home/.cargo"
CARGO_TARGET_DIR: "/github/home/target"
- - name: Check DataFusion CLI builds with ballista
- run: |
- (cd datafusion-cli && cargo check)
- (cd datafusion-cli && cargo check --no-default-features)
- (cd datafusion-cli && cargo check --features=ballista)
- env:
- CARGO_HOME: "/github/home/.cargo"
- CARGO_TARGET_DIR: "/github/home/target"
# test the crate
linux-test:
@@ -123,9 +115,9 @@ jobs:
run: |
export ARROW_TEST_DATA=$(pwd)/testing/data
export PARQUET_TEST_DATA=$(pwd)/parquet-testing/data
+ cargo test
# test datafusion examples
cd datafusion-examples
- cargo test --no-default-features
cargo run --example csv_sql
cargo run --example parquet_sql
cargo run --example avro_sql --features=datafusion/avro
@@ -134,6 +126,42 @@ jobs:
env:
CARGO_HOME: "/github/home/.cargo"
CARGO_TARGET_DIR: "/github/home/target"
+
+ # run ballista tests
+ ballista-test:
+ name: Test Ballista on AMD64 Rust ${{ matrix.rust }}
+ needs: [linux-build-lib]
+ runs-on: ubuntu-latest
+ strategy:
+ matrix:
+ arch: [amd64]
+ rust: [stable]
+ container:
+ image: ${{ matrix.arch }}/rust
+ env:
+ # Disable full debug symbol generation to speed up CI build and keep memory down
+ # "1" means line tables only, which is useful for panic tracebacks.
+ RUSTFLAGS: "-C debuginfo=1"
+ steps:
+ - uses: actions/checkout@v2
+ with:
+ submodules: true
+ - name: Cache Cargo
+ uses: actions/cache@v2
+ with:
+ path: /github/home/.cargo
+ # this key equals the ones on `linux-build-lib` for re-use
+ key: cargo-cache-
+ - name: Cache Rust dependencies
+ uses: actions/cache@v2
+ with:
+ path: /github/home/target
+ # this key equals the ones on `linux-build-lib` for re-use
+ key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}
+ - name: Setup Rust toolchain
+ uses: ./.github/actions/setup-builder
+ with:
+ rust-version: ${{ matrix.rust }}
# Ballista is currently not part of the main workspace so requires a separate test step
- name: Run Ballista tests
run: |
@@ -209,6 +237,8 @@ jobs:
POSTGRES_PASSWORD: postgres
- name: Build datafusion-cli
run: (cd datafusion-cli && cargo build)
+ - name: Build ballista-cli
+ run: (cd ballista-cli && cargo build)
- name: Test Psql Parity
run: python -m pytest -v integration-tests/test_psql_parity.py
env:
diff --git a/.gitignore b/.gitignore
index 23d609710..1cfeb1f0b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -85,6 +85,7 @@ cpp/Brewfile.lock.json
target
Cargo.lock
!datafusion-cli/Cargo.lock
+!ballista-cli/Cargo.lock
rusty-tags.vi
.history
diff --git a/Cargo.toml b/Cargo.toml
index 3567a4cea..feb3ebe5c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -33,7 +33,7 @@ members = [
"ballista/rust/scheduler",
"ballista-examples",
]
-exclude = ["datafusion-cli"]
+exclude = ["ballista-cli", "datafusion-cli"]
[profile.release]
codegen-units = 1
diff --git a/datafusion-cli/Cargo.lock b/ballista-cli/Cargo.lock
similarity index 97%
copy from datafusion-cli/Cargo.lock
copy to ballista-cli/Cargo.lock
index 9a84e35c2..528148c9b 100644
--- a/datafusion-cli/Cargo.lock
+++ b/ballista-cli/Cargo.lock
@@ -155,9 +155,9 @@ checksum = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa"
[[package]]
name = "axum"
-version = "0.5.5"
+version = "0.5.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "00f1e8a972137fad81e2a1a60b86ff17ce0338f8017264e45a9723d0083c39a1"
+checksum = "f4af7447fc1214c1f3a1ace861d0216a6c8bb13965b64bbad9650f375b67689a"
dependencies = [
"async-trait",
"axum-core",
@@ -205,11 +205,27 @@ dependencies = [
"futures",
"log",
"parking_lot",
- "sqlparser 0.16.0",
+ "sqlparser",
"tempfile",
"tokio",
]
+[[package]]
+name = "ballista-cli"
+version = "0.6.0"
+dependencies = [
+ "arrow",
+ "ballista",
+ "clap",
+ "datafusion",
+ "datafusion-cli",
+ "dirs",
+ "env_logger",
+ "mimalloc",
+ "rustyline",
+ "tokio",
+]
+
[[package]]
name = "ballista-core"
version = "0.6.0"
@@ -232,7 +248,7 @@ dependencies = [
"prost-types",
"rustc_version",
"serde",
- "sqlparser 0.16.0",
+ "sqlparser",
"tokio",
"tonic",
"tonic-build",
@@ -358,9 +374,9 @@ dependencies = [
[[package]]
name = "clap"
-version = "3.1.17"
+version = "3.1.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "47582c09be7c8b32c0ab3a6181825ababb713fde6fff20fc573a3870dd45c6a0"
+checksum = "85a35a599b11c089a7f49105658d089b8f2cf0882993c17daf6de15285c2c35d"
dependencies = [
"atty",
"bitflags",
@@ -507,7 +523,7 @@ dependencies = [
"pin-project-lite",
"rand",
"smallvec",
- "sqlparser 0.17.0",
+ "sqlparser",
"tempfile",
"tokio",
"tokio-stream",
@@ -519,7 +535,6 @@ name = "datafusion-cli"
version = "7.0.0"
dependencies = [
"arrow",
- "ballista",
"clap",
"datafusion",
"dirs",
@@ -536,12 +551,12 @@ dependencies = [
"arrow",
"ordered-float 3.0.0",
"parquet",
- "sqlparser 0.17.0",
+ "sqlparser",
]
[[package]]
name = "datafusion-data-access"
-version = "1.0.0"
+version = "7.0.0"
dependencies = [
"async-trait",
"chrono",
@@ -559,7 +574,7 @@ dependencies = [
"ahash",
"arrow",
"datafusion-common",
- "sqlparser 0.17.0",
+ "sqlparser",
]
[[package]]
@@ -1292,14 +1307,25 @@ dependencies = [
[[package]]
name = "mio"
-version = "0.8.3"
+version = "0.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "713d550d9b44d89174e066b7a6217ae06234c10cb47819a88290d2b353c31799"
+checksum = "52da4364ffb0e4fe33a9841a98a3f3014fb964045ce4f7a45a398243c8d6b0c9"
dependencies = [
"libc",
"log",
+ "miow",
+ "ntapi",
"wasi 0.11.0+wasi-snapshot-preview1",
- "windows-sys 0.36.1",
+ "winapi",
+]
+
+[[package]]
+name = "miow"
+version = "0.3.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b9f1c5b025cda876f66ef43a113f91ebc9f4ccef34843000e0adf6ebbab84e21"
+dependencies = [
+ "winapi",
]
[[package]]
@@ -1350,6 +1376,15 @@ dependencies = [
"memoffset",
]
+[[package]]
+name = "ntapi"
+version = "0.3.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c28774a7fd2fbb4f0babd8237ce554b73af68021b5f695a3cebd6c59bac0980f"
+dependencies = [
+ "winapi",
+]
+
[[package]]
name = "num"
version = "0.4.0"
@@ -1622,18 +1657,18 @@ dependencies = [
[[package]]
name = "proc-macro2"
-version = "1.0.38"
+version = "1.0.37"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9027b48e9d4c9175fa2218adf3557f91c1137021739951d4932f5f8268ac48aa"
+checksum = "ec757218438d5fda206afc041538b2f6d889286160d649a86a24d37e1235afd1"
dependencies = [
"unicode-xid",
]
[[package]]
name = "prost"
-version = "0.10.3"
+version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bc03e116981ff7d8da8e5c220e374587b98d294af7ba7dd7fda761158f00086f"
+checksum = "a07b0857a71a8cb765763950499cae2413c3f9cede1133478c43600d9e146890"
dependencies = [
"bytes",
"prost-derive",
@@ -1641,9 +1676,9 @@ dependencies = [
[[package]]
name = "prost-build"
-version = "0.10.3"
+version = "0.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "65a1118354442de7feb8a2a76f3d80ef01426bd45542c8c1fdffca41a758f846"
+checksum = "120fbe7988713f39d780a58cf1a7ef0d7ef66c6d87e5aa3438940c05357929f4"
dependencies = [
"bytes",
"cfg-if",
@@ -1887,9 +1922,9 @@ dependencies = [
[[package]]
name = "serde_json"
-version = "1.0.81"
+version = "1.0.80"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9b7ce2b32a1aed03c558dc61a5cd328f15aff2dbc17daad8fb8af04d2100e15c"
+checksum = "f972498cf015f7c0746cac89ebe1d6ef10c293b94175a243a2d9442c163d9944"
dependencies = [
"indexmap",
"itoa 1.0.1",
@@ -1936,15 +1971,6 @@ dependencies = [
"winapi",
]
-[[package]]
-name = "sqlparser"
-version = "0.16.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7e9a527b68048eb95495a1508f6c8395c8defcff5ecdbe8ad4106d08a2ef2a3c"
-dependencies = [
- "log",
-]
-
[[package]]
name = "sqlparser"
version = "0.17.0"
@@ -1999,9 +2025,9 @@ checksum = "6bdef32e8150c2a081110b42772ffe7d7c9032b606bc226c8260fd97e0976601"
[[package]]
name = "syn"
-version = "1.0.93"
+version = "1.0.92"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "04066589568b72ec65f42d65a1a52436e954b168773148893c020269563decf2"
+checksum = "7ff7c592601f11445996a06f8ad0c27f094a58857c2f89e97974ab9235b92c52"
dependencies = [
"proc-macro2",
"quote",
@@ -2087,9 +2113,9 @@ dependencies = [
[[package]]
name = "tokio"
-version = "1.18.2"
+version = "1.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4903bf0427cf68dddd5aa6a93220756f8be0c34fcfa9f5e6191e103e15a31395"
+checksum = "dce653fb475565de9f6fb0614b28bca8df2c430c0cf84bcd9c843f15de5414cc"
dependencies = [
"bytes",
"libc",
@@ -2152,9 +2178,9 @@ dependencies = [
[[package]]
name = "tonic"
-version = "0.7.2"
+version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5be9d60db39854b30b835107500cf0aca0b0d14d6e1c3de124217c23a29c2ddb"
+checksum = "30fb54bf1e446f44d870d260d99957e7d11fb9d0a0f5bd1a662ad1411cc103f9"
dependencies = [
"async-stream",
"async-trait",
@@ -2184,9 +2210,9 @@ dependencies = [
[[package]]
name = "tonic-build"
-version = "0.7.2"
+version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d9263bf4c9bfaae7317c1c2faf7f18491d2fe476f70c414b73bf5d445b00ffa1"
+checksum = "c03447cdc9eaf8feffb6412dcb27baf2db11669a6c4789f29da799aabfb99547"
dependencies = [
"prettyplease",
"proc-macro2",
@@ -2217,9 +2243,9 @@ dependencies = [
[[package]]
name = "tower-http"
-version = "0.3.3"
+version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7d342c6d58709c0a6d48d48dabbb62d4ef955cf5f0f3bbfd845838e7ae88dbae"
+checksum = "e980386f06883cf4d0578d6c9178c81f68b45d77d00f2c2c1bc034b3439c2c56"
dependencies = [
"bitflags",
"bytes",
diff --git a/datafusion-cli/Cargo.toml b/ballista-cli/Cargo.toml
similarity index 86%
copy from datafusion-cli/Cargo.toml
copy to ballista-cli/Cargo.toml
index 0f0deb3f0..d921f9b0e 100644
--- a/datafusion-cli/Cargo.toml
+++ b/ballista-cli/Cargo.toml
@@ -16,12 +16,12 @@
# under the License.
[package]
-name = "datafusion-cli"
-description = "Command Line Client for DataFusion query engine and Ballista distributed computation engine."
-version = "7.0.0"
+name = "ballista-cli"
+description = "Command Line Client for Ballista distributed query engine."
+version = "0.6.0"
authors = ["Apache Arrow <de...@arrow.apache.org>"]
edition = "2021"
-keywords = [ "arrow", "datafusion", "ballista", "query", "sql" ]
+keywords = [ "ballista", "cli", ]
license = "Apache-2.0"
homepage = "https://github.com/apache/arrow-datafusion"
repository = "https://github.com/apache/arrow-datafusion"
@@ -30,9 +30,10 @@ readme = "README.md"
[dependencies]
arrow = { version = "13" }
-ballista = { path = "../ballista/rust/client", version = "0.6.0", optional = true }
+ballista = { path = "../ballista/rust/client", version = "0.6.0" }
clap = { version = "3", features = ["derive", "cargo"] }
datafusion = { path = "../datafusion/core", version = "7.0.0" }
+datafusion-cli = { path = "../datafusion-cli", version = "7.0.0" }
dirs = "4.0.0"
env_logger = "0.9"
mimalloc = { version = "*", default-features = false }
diff --git a/Cargo.toml b/ballista-cli/Dockerfile
similarity index 61%
copy from Cargo.toml
copy to ballista-cli/Dockerfile
index 3567a4cea..7a35d185b 100644
--- a/Cargo.toml
+++ b/ballista-cli/Dockerfile
@@ -15,26 +15,24 @@
# specific language governing permissions and limitations
# under the License.
-[workspace]
-members = [
- "datafusion/common",
- "datafusion/core",
- "datafusion/data-access",
- "datafusion/expr",
- "datafusion/jit",
- "datafusion/physical-expr",
- "datafusion/proto",
- "datafusion/row",
- "datafusion-examples",
- "benchmarks",
- "ballista/rust/client",
- "ballista/rust/core",
- "ballista/rust/executor",
- "ballista/rust/scheduler",
- "ballista-examples",
-]
-exclude = ["datafusion-cli"]
-
-[profile.release]
-codegen-units = 1
-lto = true
+FROM rust:1.59 as builder
+
+COPY ./datafusion /usr/src/datafusion
+
+COPY ./ballista-cli /usr/src/ballista-cli
+
+COPY ./ballista /usr/src/ballista
+
+WORKDIR /usr/src/ballista-cli
+
+RUN rustup component add rustfmt
+
+RUN cargo build --release
+
+FROM debian:bullseye-slim
+
+COPY --from=builder /usr/src/ballista-cli/target/release/ballista-cli /usr/local/bin
+
+ENTRYPOINT ["ballista-cli"]
+
+CMD ["--data-path", "/data"]
diff --git a/datafusion-cli/README.md b/ballista-cli/README.md
similarity index 71%
copy from datafusion-cli/README.md
copy to ballista-cli/README.md
index 5c72f16a4..733329e0c 100644
--- a/datafusion-cli/README.md
+++ b/ballista-cli/README.md
@@ -17,16 +17,16 @@
under the License.
-->
-# DataFusion Command-line Interface
+# Ballista Command-line Interface
-[DataFusion](df) is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
+[Ballista](df) is a distributed query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.
-The DataFusion CLI allows SQL queries to be executed by an in-process DataFusion context, or by a distributed
+The Ballista CLI allows SQL queries to be executed by an in-process DataFusion context, or by a distributed
Ballista context.
```ignore
USAGE:
- datafusion-cli [FLAGS] [OPTIONS]
+ ballista-cli [FLAGS] [OPTIONS]
FLAGS:
-h, --help Prints help information
@@ -51,9 +51,9 @@ $ echo "1,2" > data.csv
```
```sql,ignore
-$ datafusion-cli
+$ ballista-cli
-DataFusion CLI v4.0.0-SNAPSHOT
+Ballista CLI v0.6.0
> CREATE EXTERNAL TABLE foo (a INT, b INT) STORED AS CSV LOCATION 'data.csv';
0 rows in set. Query took 0.001 seconds.
@@ -67,28 +67,28 @@ DataFusion CLI v4.0.0-SNAPSHOT
1 row in set. Query took 0.017 seconds.
```
-## DataFusion-Cli
+## Ballista-Cli
-Build the `datafusion-cli` without the feature of ballista.
+Build the `ballista-cli` without the feature of ballista.
```bash
-cd arrow-datafusion/datafusion-cli
+cd arrow-datafusion/ballista-cli
cargo build
```
## Ballista
-If you want to execute the SQL in ballista by `datafusion-cli`, you must build/compile the `datafusion-cli` with features of "ballista" first.
+If you want to execute the SQL in ballista by `ballista-cli`, you must build/compile the `ballista-cli` with features of "ballista" first.
```bash
-cd arrow-datafusion/datafusion-cli
-cargo build --features ballista
+cd arrow-datafusion/ballista-cli
+cargo build
```
-The DataFusion CLI can connect to a Ballista scheduler for query execution.
+The Ballista CLI can connect to a Ballista scheduler for query execution.
```bash
-datafusion-cli --host localhost --port 50050
+ballista-cli --host localhost --port 50050
```
[df]: https://crates.io/crates/datafusion
diff --git a/ballista-cli/src/command.rs b/ballista-cli/src/command.rs
new file mode 100644
index 000000000..0fd43a307
--- /dev/null
+++ b/ballista-cli/src/command.rs
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Command within CLI
+
+use crate::context::Context;
+use crate::functions::{display_all_functions, Function};
+use crate::print_format::PrintFormat;
+use crate::print_options::PrintOptions;
+use clap::ArgEnum;
+use datafusion::arrow::array::{ArrayRef, StringArray};
+use datafusion::arrow::datatypes::{DataType, Field, Schema};
+use datafusion::arrow::record_batch::RecordBatch;
+use datafusion::error::{DataFusionError, Result};
+use std::str::FromStr;
+use std::sync::Arc;
+use std::time::Instant;
+
+/// Command
+#[derive(Debug)]
+pub enum Command {
+ Quit,
+ Help,
+ ListTables,
+ DescribeTable(String),
+ ListFunctions,
+ SearchFunctions(String),
+ QuietMode(Option<bool>),
+ OutputFormat(Option<String>),
+}
+
+pub enum OutputFormat {
+ ChangeFormat(String),
+}
+
+impl Command {
+ pub async fn execute(
+ &self,
+ ctx: &mut Context,
+ print_options: &mut PrintOptions,
+ ) -> Result<()> {
+ let now = Instant::now();
+ match self {
+ Self::Help => print_options
+ .print_batches(&[all_commands_info()], now)
+ .map_err(|e| DataFusionError::Execution(e.to_string())),
+ Self::ListTables => {
+ let df = ctx.sql("SHOW TABLES").await?;
+ let batches = df.collect().await?;
+ print_options
+ .print_batches(&batches, now)
+ .map_err(|e| DataFusionError::Execution(e.to_string()))
+ }
+ Self::DescribeTable(name) => {
+ let df = ctx.sql(&format!("SHOW COLUMNS FROM {}", name)).await?;
+ let batches = df.collect().await?;
+ print_options
+ .print_batches(&batches, now)
+ .map_err(|e| DataFusionError::Execution(e.to_string()))
+ }
+ Self::QuietMode(quiet) => {
+ if let Some(quiet) = quiet {
+ print_options.quiet = *quiet;
+ println!(
+ "Quiet mode set to {}",
+ if print_options.quiet { "true" } else { "false" }
+ );
+ } else {
+ println!(
+ "Quiet mode is {}",
+ if print_options.quiet { "true" } else { "false" }
+ );
+ }
+ Ok(())
+ }
+ Self::Quit => Err(DataFusionError::Execution(
+ "Unexpected quit, this should be handled outside".into(),
+ )),
+ Self::ListFunctions => display_all_functions(),
+ Self::SearchFunctions(function) => {
+ if let Ok(func) = function.parse::<Function>() {
+ let details = func.function_details()?;
+ println!("{}", details);
+ Ok(())
+ } else {
+ let msg = format!("{} is not a supported function", function);
+ Err(DataFusionError::Execution(msg))
+ }
+ }
+ Self::OutputFormat(_) => Err(DataFusionError::Execution(
+ "Unexpected change output format, this should be handled outside".into(),
+ )),
+ }
+ }
+
+ fn get_name_and_description(&self) -> (&'static str, &'static str) {
+ match self {
+ Self::Quit => ("\\q", "quit datafusion-cli"),
+ Self::ListTables => ("\\d", "list tables"),
+ Self::DescribeTable(_) => ("\\d name", "describe table"),
+ Self::Help => ("\\?", "help"),
+ Self::ListFunctions => ("\\h", "function list"),
+ Self::SearchFunctions(_) => ("\\h function", "search function"),
+ Self::QuietMode(_) => ("\\quiet (true|false)?", "print or set quiet mode"),
+ Self::OutputFormat(_) => {
+ ("\\pset [NAME [VALUE]]", "set table output option\n(format)")
+ }
+ }
+ }
+}
+
+const ALL_COMMANDS: [Command; 8] = [
+ Command::ListTables,
+ Command::DescribeTable(String::new()),
+ Command::Quit,
+ Command::Help,
+ Command::ListFunctions,
+ Command::SearchFunctions(String::new()),
+ Command::QuietMode(None),
+ Command::OutputFormat(None),
+];
+
+fn all_commands_info() -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("Command", DataType::Utf8, false),
+ Field::new("Description", DataType::Utf8, false),
+ ]));
+ let (names, description): (Vec<&str>, Vec<&str>) = ALL_COMMANDS
+ .into_iter()
+ .map(|c| c.get_name_and_description())
+ .unzip();
+ RecordBatch::try_new(
+ schema,
+ [names, description]
+ .into_iter()
+ .map(|i| Arc::new(StringArray::from(i)) as ArrayRef)
+ .collect::<Vec<_>>(),
+ )
+ .expect("This should not fail")
+}
+
+impl FromStr for Command {
+ type Err = ();
+
+ fn from_str(s: &str) -> std::result::Result<Self, Self::Err> {
+ let (c, arg) = if let Some((a, b)) = s.split_once(' ') {
+ (a, Some(b))
+ } else {
+ (s, None)
+ };
+ Ok(match (c, arg) {
+ ("q", None) => Self::Quit,
+ ("d", None) => Self::ListTables,
+ ("d", Some(name)) => Self::DescribeTable(name.into()),
+ ("?", None) => Self::Help,
+ ("h", None) => Self::ListFunctions,
+ ("h", Some(function)) => Self::SearchFunctions(function.into()),
+ ("quiet", Some("true" | "t" | "yes" | "y" | "on")) => {
+ Self::QuietMode(Some(true))
+ }
+ ("quiet", Some("false" | "f" | "no" | "n" | "off")) => {
+ Self::QuietMode(Some(false))
+ }
+ ("quiet", None) => Self::QuietMode(None),
+ ("pset", Some(subcommand)) => {
+ Self::OutputFormat(Some(subcommand.to_string()))
+ }
+ ("pset", None) => Self::OutputFormat(None),
+ _ => return Err(()),
+ })
+ }
+}
+
+impl FromStr for OutputFormat {
+ type Err = ();
+
+ fn from_str(s: &str) -> std::result::Result<Self, Self::Err> {
+ let (c, arg) = if let Some((a, b)) = s.split_once(' ') {
+ (a, Some(b))
+ } else {
+ (s, None)
+ };
+ Ok(match (c, arg) {
+ ("format", Some(format)) => Self::ChangeFormat(format.to_string()),
+ _ => return Err(()),
+ })
+ }
+}
+
+impl OutputFormat {
+ pub async fn execute(&self, print_options: &mut PrintOptions) -> Result<()> {
+ match self {
+ Self::ChangeFormat(format) => {
+ if let Ok(format) = format.parse::<PrintFormat>() {
+ print_options.format = format;
+ println!("Output format is {:?}.", print_options.format);
+ Ok(())
+ } else {
+ Err(DataFusionError::Execution(format!(
+ "{:?} is not a valid format type [possible values: {:?}]",
+ format,
+ PrintFormat::value_variants()
+ )))
+ }
+ }
+ }
+ }
+}
diff --git a/datafusion-cli/src/context.rs b/ballista-cli/src/context.rs
similarity index 100%
copy from datafusion-cli/src/context.rs
copy to ballista-cli/src/context.rs
diff --git a/ballista-cli/src/exec.rs b/ballista-cli/src/exec.rs
new file mode 100644
index 000000000..dc3c73e5a
--- /dev/null
+++ b/ballista-cli/src/exec.rs
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Execution functions
+
+use crate::{
+ command::{Command, OutputFormat},
+ context::Context,
+ helper::CliHelper,
+ print_options::PrintOptions,
+};
+use datafusion::error::Result;
+use rustyline::error::ReadlineError;
+use rustyline::Editor;
+use std::fs::File;
+use std::io::prelude::*;
+use std::io::BufReader;
+use std::time::Instant;
+
+/// run and execute SQL statements and commands from a file, against a context with the given print options
+pub async fn exec_from_lines(
+ ctx: &mut Context,
+ reader: &mut BufReader<File>,
+ print_options: &PrintOptions,
+) {
+ let mut query = "".to_owned();
+
+ for line in reader.lines() {
+ match line {
+ Ok(line) if line.starts_with("--") => {
+ continue;
+ }
+ Ok(line) => {
+ let line = line.trim_end();
+ query.push_str(line);
+ if line.ends_with(';') {
+ match exec_and_print(ctx, print_options, query).await {
+ Ok(_) => {}
+ Err(err) => println!("{:?}", err),
+ }
+ query = "".to_owned();
+ } else {
+ query.push('\n');
+ }
+ }
+ _ => {
+ break;
+ }
+ }
+ }
+
+ // run the left over query if the last statement doesn't contain ‘;’
+ if !query.is_empty() {
+ match exec_and_print(ctx, print_options, query).await {
+ Ok(_) => {}
+ Err(err) => println!("{:?}", err),
+ }
+ }
+}
+
+pub async fn exec_from_files(
+ files: Vec<String>,
+ ctx: &mut Context,
+ print_options: &PrintOptions,
+) {
+ let files = files
+ .into_iter()
+ .map(|file_path| File::open(file_path).unwrap())
+ .collect::<Vec<_>>();
+ for file in files {
+ let mut reader = BufReader::new(file);
+ exec_from_lines(ctx, &mut reader, print_options).await;
+ }
+}
+
+/// run and execute SQL statements and commands against a context with the given print options
+pub async fn exec_from_repl(ctx: &mut Context, print_options: &mut PrintOptions) {
+ let mut rl = Editor::<CliHelper>::new();
+ rl.set_helper(Some(CliHelper::default()));
+ rl.load_history(".history").ok();
+
+ let mut print_options = print_options.clone();
+
+ loop {
+ match rl.readline("❯ ") {
+ Ok(line) if line.starts_with('\\') => {
+ rl.add_history_entry(line.trim_end());
+ let command = line.split_whitespace().collect::<Vec<_>>().join(" ");
+ if let Ok(cmd) = &command[1..].parse::<Command>() {
+ match cmd {
+ Command::Quit => break,
+ Command::OutputFormat(subcommand) => {
+ if let Some(subcommand) = subcommand {
+ if let Ok(command) = subcommand.parse::<OutputFormat>() {
+ if let Err(e) =
+ command.execute(&mut print_options).await
+ {
+ eprintln!("{}", e)
+ }
+ } else {
+ eprintln!(
+ "'\\{}' is not a valid command",
+ &line[1..]
+ );
+ }
+ } else {
+ println!("Output format is {:?}.", print_options.format);
+ }
+ }
+ _ => {
+ if let Err(e) = cmd.execute(ctx, &mut print_options).await {
+ eprintln!("{}", e)
+ }
+ }
+ }
+ } else {
+ eprintln!("'\\{}' is not a valid command", &line[1..]);
+ }
+ }
+ Ok(line) => {
+ rl.add_history_entry(line.trim_end());
+ match exec_and_print(ctx, &print_options, line).await {
+ Ok(_) => {}
+ Err(err) => eprintln!("{:?}", err),
+ }
+ }
+ Err(ReadlineError::Interrupted) => {
+ println!("^C");
+ continue;
+ }
+ Err(ReadlineError::Eof) => {
+ println!("\\q");
+ break;
+ }
+ Err(err) => {
+ eprintln!("Unknown error happened {:?}", err);
+ break;
+ }
+ }
+ }
+
+ rl.save_history(".history").ok();
+}
+
+async fn exec_and_print(
+ ctx: &mut Context,
+ print_options: &PrintOptions,
+ sql: String,
+) -> Result<()> {
+ let now = Instant::now();
+ let df = ctx.sql(&sql).await?;
+ let results = df.collect().await?;
+ print_options.print_batches(&results, now)?;
+
+ Ok(())
+}
diff --git a/ballista-cli/src/lib.rs b/ballista-cli/src/lib.rs
new file mode 100644
index 000000000..a51aad013
--- /dev/null
+++ b/ballista-cli/src/lib.rs
@@ -0,0 +1,25 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#![doc = include_str!("../README.md")]
+pub const BALLISTA_CLI_VERSION: &str = env!("CARGO_PKG_VERSION");
+
+pub mod command;
+pub mod context;
+pub mod exec;
+
+pub use datafusion_cli::{helper, print_format, print_options, functions};
diff --git a/datafusion-cli/src/main.rs b/ballista-cli/src/main.rs
similarity index 97%
copy from datafusion-cli/src/main.rs
copy to ballista-cli/src/main.rs
index d76fe38e5..69868091d 100644
--- a/datafusion-cli/src/main.rs
+++ b/ballista-cli/src/main.rs
@@ -15,13 +15,13 @@
// specific language governing permissions and limitations
// under the License.
+use ballista_cli::{
+ context::Context, exec, print_format::PrintFormat, print_options::PrintOptions,
+ BALLISTA_CLI_VERSION,
+};
use clap::Parser;
use datafusion::error::Result;
use datafusion::execution::context::SessionConfig;
-use datafusion_cli::{
- context::Context, exec, print_format::PrintFormat, print_options::PrintOptions,
- DATAFUSION_CLI_VERSION,
-};
use mimalloc::MiMalloc;
use std::env;
use std::path::Path;
@@ -90,7 +90,7 @@ pub async fn main() -> Result<()> {
let args = Args::parse();
if !args.quiet {
- println!("DataFusion CLI v{}", DATAFUSION_CLI_VERSION);
+ println!("Ballista CLI v{}", BALLISTA_CLI_VERSION);
}
if let Some(ref path) = args.data_path {
diff --git a/datafusion-cli/Cargo.lock b/datafusion-cli/Cargo.lock
index 9a84e35c2..4309a4216 100644
--- a/datafusion-cli/Cargo.lock
+++ b/datafusion-cli/Cargo.lock
@@ -43,12 +43,6 @@ dependencies = [
"alloc-no-stdlib",
]
-[[package]]
-name = "anyhow"
-version = "1.0.57"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "08f9b8508dccb7687a1d6c4ce66b2b0ecef467c94667de27d8d7fe1f8d2a9cdc"
-
[[package]]
name = "arrayref"
version = "0.3.6"
@@ -86,45 +80,6 @@ dependencies = [
"serde_json",
]
-[[package]]
-name = "arrow-flight"
-version = "13.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0a3666d2dbc637fa979d1f0bf3031d39a80e709f3b9ec88e3d573c1d666bf553"
-dependencies = [
- "arrow",
- "base64",
- "bytes",
- "futures",
- "proc-macro2",
- "prost",
- "prost-derive",
- "tokio",
- "tonic",
- "tonic-build",
-]
-
-[[package]]
-name = "async-stream"
-version = "0.3.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dad5c83079eae9969be7fadefe640a1c566901f05ff91ab221de4b6f68d9507e"
-dependencies = [
- "async-stream-impl",
- "futures-core",
-]
-
-[[package]]
-name = "async-stream-impl"
-version = "0.3.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "10f203db73a71dfa2fb6dd22763990fa26f3d2625a6da2da900d23b87d26be27"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn",
-]
-
[[package]]
name = "async-trait"
version = "0.1.53"
@@ -153,93 +108,6 @@ version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa"
-[[package]]
-name = "axum"
-version = "0.5.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "00f1e8a972137fad81e2a1a60b86ff17ce0338f8017264e45a9723d0083c39a1"
-dependencies = [
- "async-trait",
- "axum-core",
- "bitflags",
- "bytes",
- "futures-util",
- "http",
- "http-body",
- "hyper",
- "itoa 1.0.1",
- "matchit",
- "memchr",
- "mime",
- "percent-encoding",
- "pin-project-lite",
- "serde",
- "sync_wrapper",
- "tokio",
- "tower",
- "tower-http",
- "tower-layer",
- "tower-service",
-]
-
-[[package]]
-name = "axum-core"
-version = "0.2.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "da31c0ed7b4690e2c78fe4b880d21cd7db04a346ebc658b4270251b695437f17"
-dependencies = [
- "async-trait",
- "bytes",
- "futures-util",
- "http",
- "http-body",
- "mime",
-]
-
-[[package]]
-name = "ballista"
-version = "0.6.0"
-dependencies = [
- "ballista-core",
- "datafusion",
- "futures",
- "log",
- "parking_lot",
- "sqlparser 0.16.0",
- "tempfile",
- "tokio",
-]
-
-[[package]]
-name = "ballista-core"
-version = "0.6.0"
-dependencies = [
- "ahash",
- "arrow-flight",
- "async-trait",
- "chrono",
- "clap",
- "datafusion",
- "datafusion-proto",
- "futures",
- "hashbrown 0.12.1",
- "libloading",
- "log",
- "once_cell",
- "parking_lot",
- "parse_arg",
- "prost",
- "prost-types",
- "rustc_version",
- "serde",
- "sqlparser 0.16.0",
- "tokio",
- "tonic",
- "tonic-build",
- "uuid",
- "walkdir",
-]
-
[[package]]
name = "base64"
version = "0.13.0"
@@ -323,12 +191,6 @@ version = "1.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610"
-[[package]]
-name = "bytes"
-version = "1.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c4872d67bab6358e59559027aa3b9157c53d9358c51423c17554809a8858e0f8"
-
[[package]]
name = "cc"
version = "1.0.73"
@@ -358,9 +220,9 @@ dependencies = [
[[package]]
name = "clap"
-version = "3.1.17"
+version = "3.1.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "47582c09be7c8b32c0ab3a6181825ababb713fde6fff20fc573a3870dd45c6a0"
+checksum = "d2dbdf4bdacb33466e854ce889eee8dfd5729abf7ccd7664d0a2d60cd384440b"
dependencies = [
"atty",
"bitflags",
@@ -375,9 +237,9 @@ dependencies = [
[[package]]
name = "clap_derive"
-version = "3.1.7"
+version = "3.1.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a3aab4734e083b809aaf5794e14e756d1c798d2c69c7f7de7a09a2f5214993c1"
+checksum = "25320346e922cffe59c0bbc5410c8d8784509efb321488971081313cb1e1a33c"
dependencies = [
"heck 0.4.0",
"proc-macro-error",
@@ -406,15 +268,6 @@ dependencies = [
"winapi",
]
-[[package]]
-name = "cmake"
-version = "0.1.48"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e8ad8cef104ac57b68b89df3208164d228503abbdce70f6880ffa3d970e7443a"
-dependencies = [
- "cc",
-]
-
[[package]]
name = "comfy-table"
version = "5.0.1"
@@ -507,7 +360,7 @@ dependencies = [
"pin-project-lite",
"rand",
"smallvec",
- "sqlparser 0.17.0",
+ "sqlparser",
"tempfile",
"tokio",
"tokio-stream",
@@ -519,7 +372,6 @@ name = "datafusion-cli"
version = "7.0.0"
dependencies = [
"arrow",
- "ballista",
"clap",
"datafusion",
"dirs",
@@ -536,12 +388,12 @@ dependencies = [
"arrow",
"ordered-float 3.0.0",
"parquet",
- "sqlparser 0.17.0",
+ "sqlparser",
]
[[package]]
name = "datafusion-data-access"
-version = "1.0.0"
+version = "7.0.0"
dependencies = [
"async-trait",
"chrono",
@@ -559,7 +411,7 @@ dependencies = [
"ahash",
"arrow",
"datafusion-common",
- "sqlparser 0.17.0",
+ "sqlparser",
]
[[package]]
@@ -585,15 +437,6 @@ dependencies = [
"unicode-segmentation",
]
-[[package]]
-name = "datafusion-proto"
-version = "7.0.0"
-dependencies = [
- "datafusion",
- "prost",
- "tonic-build",
-]
-
[[package]]
name = "datafusion-row"
version = "7.0.0"
@@ -656,12 +499,6 @@ dependencies = [
"winapi",
]
-[[package]]
-name = "either"
-version = "1.6.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e78d4f1cc4ae33bbfc157ed5d5a5ef3bc29227303d595861deb238fcec4e9457"
-
[[package]]
name = "endian-type"
version = "0.1.2"
@@ -732,12 +569,6 @@ dependencies = [
"windows-sys 0.30.0",
]
-[[package]]
-name = "fixedbitset"
-version = "0.4.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "279fb028e20b3c4c320317955b77c5e0c9701f05a1d309905d6fc702cdc5053e"
-
[[package]]
name = "flatbuffers"
version = "2.1.2"
@@ -761,12 +592,6 @@ dependencies = [
"miniz_oxide",
]
-[[package]]
-name = "fnv"
-version = "1.0.7"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
-
[[package]]
name = "futures"
version = "0.3.21"
@@ -874,7 +699,7 @@ checksum = "9be70c98951c83b8d2f8f60d7065fa6d5146873094452a1008da8c2f1e4205ad"
dependencies = [
"cfg-if",
"libc",
- "wasi 0.10.2+wasi-snapshot-preview1",
+ "wasi",
]
[[package]]
@@ -883,25 +708,6 @@ version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b919933a397b79c37e33b77bb2aa3dc8eb6e165ad809e58ff75bc7db2e34574"
-[[package]]
-name = "h2"
-version = "0.3.13"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "37a82c6d637fc9515a4694bbf1cb2457b79d81ce52b3108bdeea58b07dd34a57"
-dependencies = [
- "bytes",
- "fnv",
- "futures-core",
- "futures-sink",
- "futures-util",
- "http",
- "indexmap",
- "slab",
- "tokio",
- "tokio-util",
- "tracing",
-]
-
[[package]]
name = "half"
version = "1.8.2"
@@ -953,88 +759,12 @@ version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
-[[package]]
-name = "http"
-version = "0.2.7"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ff8670570af52249509a86f5e3e18a08c60b177071826898fde8997cf5f6bfbb"
-dependencies = [
- "bytes",
- "fnv",
- "itoa 1.0.1",
-]
-
-[[package]]
-name = "http-body"
-version = "0.4.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1ff4f84919677303da5f147645dbea6b1881f368d03ac84e1dc09031ebd7b2c6"
-dependencies = [
- "bytes",
- "http",
- "pin-project-lite",
-]
-
-[[package]]
-name = "http-range-header"
-version = "0.3.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0bfe8eed0a9285ef776bb792479ea3834e8b94e13d615c2f66d03dd50a435a29"
-
-[[package]]
-name = "httparse"
-version = "1.7.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "496ce29bb5a52785b44e0f7ca2847ae0bb839c9bd28f69acac9b99d461c0c04c"
-
-[[package]]
-name = "httpdate"
-version = "1.0.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c4a1e36c821dbe04574f602848a19f742f4fb3c98d40449f11bcad18d6b17421"
-
[[package]]
name = "humantime"
version = "2.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9a3a5bfb195931eeb336b2a7b4d761daec841b97f947d34394601737a7bba5e4"
-[[package]]
-name = "hyper"
-version = "0.14.18"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b26ae0a80afebe130861d90abf98e3814a4f28a4c6ffeb5ab8ebb2be311e0ef2"
-dependencies = [
- "bytes",
- "futures-channel",
- "futures-core",
- "futures-util",
- "h2",
- "http",
- "http-body",
- "httparse",
- "httpdate",
- "itoa 1.0.1",
- "pin-project-lite",
- "socket2",
- "tokio",
- "tower-service",
- "tracing",
- "want",
-]
-
-[[package]]
-name = "hyper-timeout"
-version = "0.4.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bbb958482e8c7be4bc3cf272a766a2b0bf1a6755e7a6ae777f017a31d11b13b1"
-dependencies = [
- "hyper",
- "pin-project-lite",
- "tokio",
- "tokio-io-timeout",
-]
-
[[package]]
name = "indexmap"
version = "1.8.1"
@@ -1066,15 +796,6 @@ version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9448015e586b611e5d322f6703812bbca2f1e709d5773ecd38ddb4e3bb649504"
-[[package]]
-name = "itertools"
-version = "0.10.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a9a9d19fa1e79b6215ff29b9d6880b706147f16e9b1dbb1e4e5947b5b02bc5e3"
-dependencies = [
- "either",
-]
-
[[package]]
name = "itoa"
version = "0.4.8"
@@ -1172,16 +893,6 @@ version = "0.2.125"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5916d2ae698f6de9bfb891ad7a8d65c09d232dc58cc4ac433c7da3b2fd84bc2b"
-[[package]]
-name = "libloading"
-version = "0.7.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "efbc0f03f9a775e9f6aed295c6a1ba2253c5757a9e03d55c6caa46a681abcddd"
-dependencies = [
- "cfg-if",
- "winapi",
-]
-
[[package]]
name = "libmimalloc-sys"
version = "0.1.25"
@@ -1236,12 +947,6 @@ dependencies = [
"libc",
]
-[[package]]
-name = "matchit"
-version = "0.5.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "73cbba799671b762df5a175adf59ce145165747bb891505c43d09aefbbf38beb"
-
[[package]]
name = "md-5"
version = "0.10.1"
@@ -1275,12 +980,6 @@ dependencies = [
"libmimalloc-sys",
]
-[[package]]
-name = "mime"
-version = "0.3.16"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2a60c7ce501c71e03a9c9c0d35b861413ae925bd979cc7a4e30d060069aaac8d"
-
[[package]]
name = "miniz_oxide"
version = "0.5.1"
@@ -1290,24 +989,6 @@ dependencies = [
"adler",
]
-[[package]]
-name = "mio"
-version = "0.8.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "713d550d9b44d89174e066b7a6217ae06234c10cb47819a88290d2b353c31799"
-dependencies = [
- "libc",
- "log",
- "wasi 0.11.0+wasi-snapshot-preview1",
- "windows-sys 0.36.1",
-]
-
-[[package]]
-name = "multimap"
-version = "0.8.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e5ce46fe64a9d73be07dcbe690a38ce1b293be448fd8ce1e6c1b8062c9f72c6a"
-
[[package]]
name = "multiversion"
version = "0.6.1"
@@ -1520,54 +1201,12 @@ dependencies = [
"thrift",
]
-[[package]]
-name = "parse_arg"
-version = "0.1.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "14248cc8eced350e20122a291613de29e4fa129ba2731818c4cdbb44fccd3e55"
-
[[package]]
name = "paste"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c520e05135d6e763148b6426a837e239041653ba7becd2e538c076c738025fc"
-[[package]]
-name = "percent-encoding"
-version = "2.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d4fd5641d01c8f18a23da7b6fe29298ff4b55afcccdf78973b24cf3175fee32e"
-
-[[package]]
-name = "petgraph"
-version = "0.6.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4a13a2fa9d0b63e5f22328828741e523766fff0ee9e779316902290dff3f824f"
-dependencies = [
- "fixedbitset",
- "indexmap",
-]
-
-[[package]]
-name = "pin-project"
-version = "1.0.10"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "58ad3879ad3baf4e44784bc6a718a8698867bb991f8ce24d1bcbe2cfb4c3a75e"
-dependencies = [
- "pin-project-internal",
-]
-
-[[package]]
-name = "pin-project-internal"
-version = "1.0.10"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "744b6f092ba29c3650faf274db506afd39944f48420f6c86b17cfe0ee1cb36bb"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn",
-]
-
[[package]]
name = "pin-project-lite"
version = "0.2.9"
@@ -1586,16 +1225,6 @@ version = "0.2.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eb9f9e6e233e5c4a35559a617bf40a4ec447db2e84c20b55a6f83167b7e57872"
-[[package]]
-name = "prettyplease"
-version = "0.1.10"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d9e07e3a46d0771a8a06b5f4441527802830b43e679ba12f44960f48dd4c6803"
-dependencies = [
- "proc-macro2",
- "syn",
-]
-
[[package]]
name = "proc-macro-error"
version = "1.0.4"
@@ -1629,61 +1258,6 @@ dependencies = [
"unicode-xid",
]
-[[package]]
-name = "prost"
-version = "0.10.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bc03e116981ff7d8da8e5c220e374587b98d294af7ba7dd7fda761158f00086f"
-dependencies = [
- "bytes",
- "prost-derive",
-]
-
-[[package]]
-name = "prost-build"
-version = "0.10.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "65a1118354442de7feb8a2a76f3d80ef01426bd45542c8c1fdffca41a758f846"
-dependencies = [
- "bytes",
- "cfg-if",
- "cmake",
- "heck 0.4.0",
- "itertools",
- "lazy_static",
- "log",
- "multimap",
- "petgraph",
- "prost",
- "prost-types",
- "regex",
- "tempfile",
- "which",
-]
-
-[[package]]
-name = "prost-derive"
-version = "0.10.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7b670f45da57fb8542ebdbb6105a925fe571b67f9e7ed9f47a06a84e72b4e7cc"
-dependencies = [
- "anyhow",
- "itertools",
- "proc-macro2",
- "quote",
- "syn",
-]
-
-[[package]]
-name = "prost-types"
-version = "0.10.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2d0a014229361011dc8e69c8a1ec6c2e8d0f2af7c91e3ea3f5b2170298461e68"
-dependencies = [
- "bytes",
- "prost",
-]
-
[[package]]
name = "quote"
version = "1.0.18"
@@ -1785,15 +1359,6 @@ dependencies = [
"winapi",
]
-[[package]]
-name = "rustc_version"
-version = "0.4.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bfa0f585226d2e68097d4f95d113b15b83a82e819ab25717ec0590d9584ef366"
-dependencies = [
- "semver",
-]
-
[[package]]
name = "rustix"
version = "0.34.6"
@@ -1844,35 +1409,17 @@ version = "1.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "73b4b750c782965c211b42f022f59af1fbceabdd026623714f104152f1ec149f"
-[[package]]
-name = "same-file"
-version = "1.0.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
-dependencies = [
- "winapi-util",
-]
-
[[package]]
name = "scopeguard"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"
-[[package]]
-name = "semver"
-version = "1.0.9"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8cb243bdfdb5936c8dc3c45762a19d12ab4550cdc753bc247637d4ec35a040fd"
-
[[package]]
name = "serde"
version = "1.0.137"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61ea8d54c77f8315140a05f4c7237403bf38b72704d031543aa1d16abbf517d1"
-dependencies = [
- "serde_derive",
-]
[[package]]
name = "serde_derive"
@@ -1926,25 +1473,6 @@ version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "45456094d1983e2ee2a18fdfebce3189fa451699d0502cb8e3b49dba5ba41451"
-[[package]]
-name = "socket2"
-version = "0.4.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "66d72b759436ae32898a2af0a14218dbf55efde3feeb170eb623637db85ee1e0"
-dependencies = [
- "libc",
- "winapi",
-]
-
-[[package]]
-name = "sqlparser"
-version = "0.16.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7e9a527b68048eb95495a1508f6c8395c8defcff5ecdbe8ad4106d08a2ef2a3c"
-dependencies = [
- "log",
-]
-
[[package]]
name = "sqlparser"
version = "0.17.0"
@@ -2008,12 +1536,6 @@ dependencies = [
"unicode-xid",
]
-[[package]]
-name = "sync_wrapper"
-version = "0.1.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "20518fe4a4c9acf048008599e464deb21beeae3d3578418951a189c235a7a9a8"
-
[[package]]
name = "tempfile"
version = "3.3.0"
@@ -2091,27 +1613,11 @@ version = "1.18.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4903bf0427cf68dddd5aa6a93220756f8be0c34fcfa9f5e6191e103e15a31395"
dependencies = [
- "bytes",
- "libc",
- "memchr",
- "mio",
"num_cpus",
"once_cell",
"parking_lot",
"pin-project-lite",
- "socket2",
"tokio-macros",
- "winapi",
-]
-
-[[package]]
-name = "tokio-io-timeout"
-version = "1.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "30b74022ada614a1b4834de765f9bb43877f910cc8ce4be40e89042c9223a8bf"
-dependencies = [
- "pin-project-lite",
- "tokio",
]
[[package]]
@@ -2136,165 +1642,6 @@ dependencies = [
"tokio",
]
-[[package]]
-name = "tokio-util"
-version = "0.7.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0edfdeb067411dba2044da6d1cb2df793dd35add7888d73c16e3381ded401764"
-dependencies = [
- "bytes",
- "futures-core",
- "futures-sink",
- "pin-project-lite",
- "tokio",
- "tracing",
-]
-
-[[package]]
-name = "tonic"
-version = "0.7.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5be9d60db39854b30b835107500cf0aca0b0d14d6e1c3de124217c23a29c2ddb"
-dependencies = [
- "async-stream",
- "async-trait",
- "axum",
- "base64",
- "bytes",
- "futures-core",
- "futures-util",
- "h2",
- "http",
- "http-body",
- "hyper",
- "hyper-timeout",
- "percent-encoding",
- "pin-project",
- "prost",
- "prost-derive",
- "tokio",
- "tokio-stream",
- "tokio-util",
- "tower",
- "tower-layer",
- "tower-service",
- "tracing",
- "tracing-futures",
-]
-
-[[package]]
-name = "tonic-build"
-version = "0.7.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d9263bf4c9bfaae7317c1c2faf7f18491d2fe476f70c414b73bf5d445b00ffa1"
-dependencies = [
- "prettyplease",
- "proc-macro2",
- "prost-build",
- "quote",
- "syn",
-]
-
-[[package]]
-name = "tower"
-version = "0.4.12"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9a89fd63ad6adf737582df5db40d286574513c69a11dac5214dc3b5603d6713e"
-dependencies = [
- "futures-core",
- "futures-util",
- "indexmap",
- "pin-project",
- "pin-project-lite",
- "rand",
- "slab",
- "tokio",
- "tokio-util",
- "tower-layer",
- "tower-service",
- "tracing",
-]
-
-[[package]]
-name = "tower-http"
-version = "0.3.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7d342c6d58709c0a6d48d48dabbb62d4ef955cf5f0f3bbfd845838e7ae88dbae"
-dependencies = [
- "bitflags",
- "bytes",
- "futures-core",
- "futures-util",
- "http",
- "http-body",
- "http-range-header",
- "pin-project-lite",
- "tower",
- "tower-layer",
- "tower-service",
-]
-
-[[package]]
-name = "tower-layer"
-version = "0.3.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "343bc9466d3fe6b0f960ef45960509f84480bf4fd96f92901afe7ff3df9d3a62"
-
-[[package]]
-name = "tower-service"
-version = "0.3.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "360dfd1d6d30e05fda32ace2c8c70e9c0a9da713275777f5a4dbb8a1893930c6"
-
-[[package]]
-name = "tracing"
-version = "0.1.34"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5d0ecdcb44a79f0fe9844f0c4f33a342cbcbb5117de8001e6ba0dc2351327d09"
-dependencies = [
- "cfg-if",
- "log",
- "pin-project-lite",
- "tracing-attributes",
- "tracing-core",
-]
-
-[[package]]
-name = "tracing-attributes"
-version = "0.1.21"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cc6b8ad3567499f98a1db7a752b07a7c8c7c7c34c332ec00effb2b0027974b7c"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn",
-]
-
-[[package]]
-name = "tracing-core"
-version = "0.1.26"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f54c8ca710e81886d498c2fd3331b56c93aa248d49de2222ad2742247c60072f"
-dependencies = [
- "lazy_static",
-]
-
-[[package]]
-name = "tracing-futures"
-version = "0.2.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "97d095ae15e245a057c8e8451bab9b3ee1e1f68e9ba2b4fbc18d0ac5237835f2"
-dependencies = [
- "pin-project",
- "tracing",
-]
-
-[[package]]
-name = "try-lock"
-version = "0.2.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "59547bce71d9c38b83d9c0e92b6066c4253371f15005def0c30d9657f50c7642"
-
[[package]]
name = "typenum"
version = "1.15.0"
@@ -2340,50 +1687,12 @@ version = "0.9.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "49874b5167b65d7193b8aba1567f5c7d93d001cafc34600cee003eda787e483f"
-[[package]]
-name = "walkdir"
-version = "2.3.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "808cf2735cd4b6866113f648b791c6adc5714537bc222d9347bb203386ffda56"
-dependencies = [
- "same-file",
- "winapi",
- "winapi-util",
-]
-
-[[package]]
-name = "want"
-version = "0.3.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1ce8a968cb1cd110d136ff8b819a556d6fb6d919363c61534f6860c7eb172ba0"
-dependencies = [
- "log",
- "try-lock",
-]
-
[[package]]
name = "wasi"
version = "0.10.2+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd6fbd9a79829dd1ad0cc20627bf1ed606756a7f77edff7b66b7064f9cb327c6"
-[[package]]
-name = "wasi"
-version = "0.11.0+wasi-snapshot-preview1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423"
-
-[[package]]
-name = "which"
-version = "4.2.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5c4fb54e6113b6a8772ee41c3404fb0301ac79604489467e0a9ce1f3e97c24ae"
-dependencies = [
- "either",
- "lazy_static",
- "libc",
-]
-
[[package]]
name = "winapi"
version = "0.3.9"
diff --git a/datafusion-cli/Cargo.toml b/datafusion-cli/Cargo.toml
index 0f0deb3f0..08dc44fe7 100644
--- a/datafusion-cli/Cargo.toml
+++ b/datafusion-cli/Cargo.toml
@@ -17,7 +17,7 @@
[package]
name = "datafusion-cli"
-description = "Command Line Client for DataFusion query engine and Ballista distributed computation engine."
+description = "Command Line Client for DataFusion query engine."
version = "7.0.0"
authors = ["Apache Arrow <de...@arrow.apache.org>"]
edition = "2021"
@@ -30,7 +30,6 @@ readme = "README.md"
[dependencies]
arrow = { version = "13" }
-ballista = { path = "../ballista/rust/client", version = "0.6.0", optional = true }
clap = { version = "3", features = ["derive", "cargo"] }
datafusion = { path = "../datafusion/core", version = "7.0.0" }
dirs = "4.0.0"
diff --git a/datafusion-cli/README.md b/datafusion-cli/README.md
index 5c72f16a4..2b21aa2b8 100644
--- a/datafusion-cli/README.md
+++ b/datafusion-cli/README.md
@@ -53,7 +53,7 @@ $ echo "1,2" > data.csv
```sql,ignore
$ datafusion-cli
-DataFusion CLI v4.0.0-SNAPSHOT
+DataFusion CLI v7.0.0
> CREATE EXTERNAL TABLE foo (a INT, b INT) STORED AS CSV LOCATION 'data.csv';
0 rows in set. Query took 0.001 seconds.
diff --git a/datafusion-cli/src/context.rs b/datafusion-cli/src/context.rs
index c96b0e766..8645a74d5 100644
--- a/datafusion-cli/src/context.rs
+++ b/datafusion-cli/src/context.rs
@@ -31,11 +31,6 @@ pub enum Context {
}
impl Context {
- /// create a new remote context with given host and port
- pub async fn new_remote(host: &str, port: u16) -> Result<Context> {
- Ok(Context::Remote(BallistaContext::try_new(host, port).await?))
- }
-
/// create a local context using the given config
pub fn new_local(config: &SessionConfig) -> Context {
Context::Local(SessionContext::with_config(config.clone()))
diff --git a/datafusion-cli/src/helper.rs b/datafusion-cli/src/helper.rs
index e229255f5..42eeb83f8 100644
--- a/datafusion-cli/src/helper.rs
+++ b/datafusion-cli/src/helper.rs
@@ -33,7 +33,7 @@ use rustyline::Helper;
use rustyline::Result;
#[derive(Default)]
-pub(crate) struct CliHelper {
+pub struct CliHelper {
completer: FilenameCompleter,
}
diff --git a/datafusion-cli/src/main.rs b/datafusion-cli/src/main.rs
index d76fe38e5..0702ff447 100644
--- a/datafusion-cli/src/main.rs
+++ b/datafusion-cli/src/main.rs
@@ -70,12 +70,6 @@ struct Args {
#[clap(long, arg_enum, default_value_t = PrintFormat::Table)]
format: PrintFormat,
- #[clap(long, help = "Ballista scheduler host")]
- host: Option<String>,
-
- #[clap(long, help = "Ballista scheduler port")]
- port: Option<u16>,
-
#[clap(
short,
long,
@@ -104,10 +98,7 @@ pub async fn main() -> Result<()> {
session_config = session_config.with_batch_size(batch_size);
};
- let mut ctx: Context = match (args.host, args.port) {
- (Some(ref h), Some(p)) => Context::new_remote(h, p).await?,
- _ => Context::new_local(&session_config),
- };
+ let mut ctx: Context = Context::new_local(&session_config);
let mut print_options = PrintOptions {
format: args.format,
diff --git a/dev/docker/ballista.dockerfile b/dev/docker/ballista.dockerfile
index 7fdb0d4bc..dac756633 100644
--- a/dev/docker/ballista.dockerfile
+++ b/dev/docker/ballista.dockerfile
@@ -30,6 +30,7 @@ RUN cargo install cargo-chef --version 0.1.34
FROM base as planner
ADD Cargo.toml .
COPY ballista ./ballista/
+COPY ballista-cli ./ballista-cli/
COPY ballista-examples ./ballista-examples/
COPY benchmarks ./benchmarks/
COPY datafusion ./datafusion/
@@ -43,6 +44,7 @@ RUN cargo chef cook $RELEASE_FLAG --recipe-path recipe.json
FROM base as builder
RUN mkdir /tmp/ballista/ballista
+RUN mkdir /tmp/ballista/ballista-cli
RUN mkdir /tmp/ballista/ballista-examples
RUN mkdir /tmp/ballista/benchmarks
RUN mkdir /tmp/ballista/datafusion
@@ -50,6 +52,7 @@ RUN mkdir /tmp/ballista/datafusion-cli
RUN mkdir /tmp/ballista/datafusion-examples
ADD Cargo.toml .
COPY ballista ./ballista/
+COPY ballista-cli ./ballista-cli/
COPY ballista-examples ./ballista-examples/
COPY benchmarks ./benchmarks/
COPY datafusion ./datafusion/
diff --git a/dev/release/README.md b/dev/release/README.md
index d6f481ebf..6437353dc 100644
--- a/dev/release/README.md
+++ b/dev/release/README.md
@@ -285,6 +285,7 @@ of the following crates:
- [datafusion-proto](https://crates.io/crates/datafusion-proto)
- [datafusion-row](https://crates.io/crates/datafusion-row)
- [ballista](https://crates.io/crates/ballista)
+- [ballista-cli](https://crates.io/crates/ballista-cli)
- [ballista-core](https://crates.io/crates/ballista-core)
- [ballista-executor](https://crates.io/crates/ballista-executor)
- [ballista-scheduler](https://crates.io/crates/ballista-scheduler)
@@ -315,9 +316,6 @@ dot -Tsvg dev/release/crate-deps.dot > dev/release/crate-deps.svg
(cd datafusion-cli && cargo publish)
```
-Note that `datafusion-cli` will depend on the previously published version of Ballista (which depends on the
-previously published version of DataFusion) until https://github.com/apache/arrow-datafusion/issues/2433 is resolved.
-
If there is a ballista release, run
```shell
@@ -325,6 +323,7 @@ If there is a ballista release, run
(cd ballista/rust/executor && cargo publish)
(cd ballista/rust/scheduler && cargo publish)
(cd ballista/rust/client && cargo publish)
+(cd ballista-cli && cargo publish)
```
### Publish datafusion-cli on Homebrew and crates.io
diff --git a/dev/release/crate-deps.dot b/dev/release/crate-deps.dot
index fd292fa2d..5176814e3 100644
--- a/dev/release/crate-deps.dot
+++ b/dev/release/crate-deps.dot
@@ -56,6 +56,9 @@ digraph G {
ballista -> ballista_executor
datafusion_cli -> datafusion
- datafusion_cli -> ballista
+
+ ballista_cli -> datafusion
+ ballista_cli -> datafusion_cli
+ ballista_cli -> ballista
}
diff --git a/dev/release/crate-deps.svg b/dev/release/crate-deps.svg
index c41cd0aa7..eb98a00ce 100644
--- a/dev/release/crate-deps.svg
+++ b/dev/release/crate-deps.svg
@@ -1,247 +1,265 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<!-- Generated by graphviz version 3.0.0 (20220226.1711)
+<!-- Generated by graphviz version 2.43.0 (0)
-->
<!-- Title: G Pages: 1 -->
-<svg width="678pt" height="692pt"
- viewBox="0.00 0.00 677.68 692.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+<svg width="812pt" height="692pt"
+ viewBox="0.00 0.00 812.33 692.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 688)">
<title>G</title>
-<polygon fill="white" stroke="transparent" points="-4,4 -4,-688 673.68,-688 673.68,4 -4,4"/>
+<polygon fill="white" stroke="transparent" points="-4,4 -4,-688 808.33,-688 808.33,4 -4,4"/>
<!-- datafusion_common -->
<g id="node1" class="node">
<title>datafusion_common</title>
-<ellipse fill="none" stroke="black" cx="325.74" cy="-18" rx="83.39" ry="18"/>
-<text text-anchor="middle" x="325.74" y="-14.3" font-family="Times,serif" font-size="14.00">datafusion_common</text>
+<ellipse fill="none" stroke="black" cx="395.39" cy="-18" rx="102.88" ry="18"/>
+<text text-anchor="middle" x="395.39" y="-14.3" font-family="Times,serif" font-size="14.00">datafusion_common</text>
</g>
<!-- datafusion_data_access -->
<g id="node2" class="node">
<title>datafusion_data_access</title>
-<ellipse fill="none" stroke="black" cx="576.74" cy="-234" rx="92.88" ry="18"/>
-<text text-anchor="middle" x="576.74" y="-230.3" font-family="Times,serif" font-size="14.00">datafusion_data_access</text>
+<ellipse fill="none" stroke="black" cx="685.39" cy="-234" rx="118.88" ry="18"/>
+<text text-anchor="middle" x="685.39" y="-230.3" font-family="Times,serif" font-size="14.00">datafusion_data_access</text>
</g>
<!-- datafusion_expr -->
<g id="node3" class="node">
<title>datafusion_expr</title>
-<ellipse fill="none" stroke="black" cx="229.74" cy="-90" rx="67.69" ry="18"/>
-<text text-anchor="middle" x="229.74" y="-86.3" font-family="Times,serif" font-size="14.00">datafusion_expr</text>
+<ellipse fill="none" stroke="black" cx="282.39" cy="-90" rx="85.29" ry="18"/>
+<text text-anchor="middle" x="282.39" y="-86.3" font-family="Times,serif" font-size="14.00">datafusion_expr</text>
</g>
<!-- datafusion_expr->datafusion_common -->
<g id="edge1" class="edge">
<title>datafusion_expr->datafusion_common</title>
-<path fill="none" stroke="black" d="M252.01,-72.76C264.76,-63.47 280.95,-51.66 294.91,-41.48"/>
-<polygon fill="black" stroke="black" points="297.25,-44.11 303.26,-35.39 293.12,-38.45 297.25,-44.11"/>
+<path fill="none" stroke="black" d="M308.6,-72.76C323.78,-63.36 343.11,-51.39 359.67,-41.13"/>
+<polygon fill="black" stroke="black" points="361.88,-43.87 368.54,-35.63 358.19,-37.92 361.88,-43.87"/>
</g>
<!-- datafusion_physical_expr -->
<g id="node4" class="node">
<title>datafusion_physical_expr</title>
-<ellipse fill="none" stroke="black" cx="100.74" cy="-162" rx="100.98" ry="18"/>
-<text text-anchor="middle" x="100.74" y="-158.3" font-family="Times,serif" font-size="14.00">datafusion_physical_expr</text>
+<ellipse fill="none" stroke="black" cx="127.39" cy="-162" rx="127.28" ry="18"/>
+<text text-anchor="middle" x="127.39" y="-158.3" font-family="Times,serif" font-size="14.00">datafusion_physical_expr</text>
</g>
<!-- datafusion_physical_expr->datafusion_common -->
<g id="edge2" class="edge">
<title>datafusion_physical_expr->datafusion_common</title>
-<path fill="none" stroke="black" d="M106.33,-144.01C113.78,-123.93 128.89,-90.88 152.74,-72 179.52,-50.8 214.76,-38.05 246.44,-30.39"/>
-<polygon fill="black" stroke="black" points="247.62,-33.71 256.59,-28.08 246.06,-26.89 247.62,-33.71"/>
+<path fill="none" stroke="black" d="M134.58,-143.8C143.96,-123.51 162.29,-90.25 188.39,-72 220.21,-49.75 260.61,-36.83 297.25,-29.34"/>
+<polygon fill="black" stroke="black" points="298.32,-32.7 307.48,-27.37 297,-25.82 298.32,-32.7"/>
</g>
<!-- datafusion_physical_expr->datafusion_expr -->
<g id="edge3" class="edge">
<title>datafusion_physical_expr->datafusion_expr</title>
-<path fill="none" stroke="black" d="M130.66,-144.76C149.04,-134.79 172.73,-121.94 192.34,-111.3"/>
-<polygon fill="black" stroke="black" points="194.19,-114.27 201.31,-106.43 190.85,-108.12 194.19,-114.27"/>
+<path fill="none" stroke="black" d="M163.73,-144.59C186.16,-134.46 215.08,-121.4 238.75,-110.71"/>
+<polygon fill="black" stroke="black" points="240.2,-113.89 247.87,-106.59 237.32,-107.51 240.2,-113.89"/>
</g>
<!-- datafusion_jit -->
<g id="node5" class="node">
<title>datafusion_jit</title>
-<ellipse fill="none" stroke="black" cx="316.74" cy="-162" rx="59.29" ry="18"/>
-<text text-anchor="middle" x="316.74" y="-158.3" font-family="Times,serif" font-size="14.00">datafusion_jit</text>
+<ellipse fill="none" stroke="black" cx="383.39" cy="-162" rx="73.39" ry="18"/>
+<text text-anchor="middle" x="383.39" y="-158.3" font-family="Times,serif" font-size="14.00">datafusion_jit</text>
</g>
<!-- datafusion_jit->datafusion_common -->
<g id="edge4" class="edge">
<title>datafusion_jit->datafusion_common</title>
-<path fill="none" stroke="black" d="M317.83,-143.87C319.36,-119.67 322.18,-75.21 324,-46.39"/>
-<polygon fill="black" stroke="black" points="327.51,-46.39 324.65,-36.19 320.53,-45.95 327.51,-46.39"/>
+<path fill="none" stroke="black" d="M384.84,-143.87C386.88,-119.67 390.64,-75.21 393.07,-46.39"/>
+<polygon fill="black" stroke="black" points="396.58,-46.45 393.94,-36.19 389.61,-45.86 396.58,-46.45"/>
</g>
<!-- datafusion_jit->datafusion_expr -->
<g id="edge5" class="edge">
<title>datafusion_jit->datafusion_expr</title>
-<path fill="none" stroke="black" d="M296.56,-144.76C285.12,-135.55 270.61,-123.88 258.04,-113.77"/>
-<polygon fill="black" stroke="black" points="260.1,-110.93 250.11,-107.39 255.71,-116.39 260.1,-110.93"/>
+<path fill="none" stroke="black" d="M359.96,-144.76C346.42,-135.38 329.19,-123.44 314.41,-113.19"/>
+<polygon fill="black" stroke="black" points="316.25,-110.21 306.04,-107.39 312.26,-115.96 316.25,-110.21"/>
</g>
<!-- datafusion_row -->
<g id="node6" class="node">
<title>datafusion_row</title>
-<ellipse fill="none" stroke="black" cx="361.74" cy="-234" rx="65.79" ry="18"/>
-<text text-anchor="middle" x="361.74" y="-230.3" font-family="Times,serif" font-size="14.00">datafusion_row</text>
+<ellipse fill="none" stroke="black" cx="429.39" cy="-234" rx="81.49" ry="18"/>
+<text text-anchor="middle" x="429.39" y="-230.3" font-family="Times,serif" font-size="14.00">datafusion_row</text>
</g>
<!-- datafusion_row->datafusion_common -->
<g id="edge6" class="edge">
<title>datafusion_row->datafusion_common</title>
-<path fill="none" stroke="black" d="M371.28,-215.93C376.38,-205.82 382.13,-192.57 384.74,-180 387.99,-164.33 388.24,-159.61 384.74,-144 376.55,-107.46 356.42,-69.01 341.95,-44.57"/>
-<polygon fill="black" stroke="black" points="344.89,-42.66 336.72,-35.92 338.9,-46.29 344.89,-42.66"/>
+<path fill="none" stroke="black" d="M444.62,-216.24C452.48,-206.44 461.27,-193.4 465.39,-180 470.09,-164.71 469.43,-159.48 465.39,-144 455.6,-106.49 431.78,-68.32 414.65,-44.2"/>
+<polygon fill="black" stroke="black" points="417.39,-42.01 408.68,-35.97 411.72,-46.12 417.39,-42.01"/>
</g>
<!-- datafusion_row->datafusion_jit -->
<g id="edge7" class="edge">
<title>datafusion_row->datafusion_jit</title>
-<path fill="none" stroke="black" d="M350.85,-216.05C345.49,-207.71 338.91,-197.49 332.97,-188.25"/>
-<polygon fill="black" stroke="black" points="335.89,-186.31 327.53,-179.79 330,-190.09 335.89,-186.31"/>
+<path fill="none" stroke="black" d="M418.25,-216.05C412.85,-207.84 406.25,-197.79 400.25,-188.66"/>
+<polygon fill="black" stroke="black" points="403.16,-186.71 394.74,-180.28 397.31,-190.55 403.16,-186.71"/>
</g>
<!-- datafusion -->
<g id="node7" class="node">
<title>datafusion</title>
-<ellipse fill="none" stroke="black" cx="314.74" cy="-306" rx="47.39" ry="18"/>
-<text text-anchor="middle" x="314.74" y="-302.3" font-family="Times,serif" font-size="14.00">datafusion</text>
+<ellipse fill="none" stroke="black" cx="374.39" cy="-306" rx="59.59" ry="18"/>
+<text text-anchor="middle" x="374.39" y="-302.3" font-family="Times,serif" font-size="14.00">datafusion</text>
</g>
<!-- datafusion->datafusion_common -->
<g id="edge8" class="edge">
<title>datafusion->datafusion_common</title>
-<path fill="none" stroke="black" d="M358.07,-298.44C385.67,-291.77 419.39,-278.45 436.74,-252 483.66,-180.45 398.83,-86 352.69,-42.63"/>
-<polygon fill="black" stroke="black" points="354.82,-39.84 345.1,-35.63 350.08,-44.98 354.82,-39.84"/>
+<path fill="none" stroke="black" d="M430.68,-299.87C462.34,-293.78 499.42,-280.56 519.39,-252 554.53,-201.74 521.44,-158.92 457.39,-72 449.24,-60.94 438.35,-50.64 427.98,-42.13"/>
+<polygon fill="black" stroke="black" points="429.86,-39.15 419.84,-35.7 425.52,-44.64 429.86,-39.15"/>
</g>
<!-- datafusion->datafusion_data_access -->
<g id="edge9" class="edge">
<title>datafusion->datafusion_data_access</title>
-<path fill="none" stroke="black" d="M352.28,-294.97C394.6,-283.66 464.06,-265.11 514.52,-251.62"/>
-<polygon fill="black" stroke="black" points="515.59,-254.96 524.34,-249 513.78,-248.2 515.59,-254.96"/>
+<path fill="none" stroke="black" d="M420.95,-294.52C471.49,-283.15 552.88,-264.83 612.03,-251.51"/>
+<polygon fill="black" stroke="black" points="613.07,-254.86 622.06,-249.25 611.53,-248.04 613.07,-254.86"/>
</g>
<!-- datafusion->datafusion_expr -->
<g id="edge10" class="edge">
<title>datafusion->datafusion_expr</title>
-<path fill="none" stroke="black" d="M303.04,-288.21C296.15,-277.98 287.48,-264.51 280.74,-252 264.13,-221.17 259.57,-213.3 248.74,-180 242.12,-159.62 237.18,-135.86 234,-118.01"/>
-<polygon fill="black" stroke="black" points="237.42,-117.27 232.29,-108.01 230.52,-118.45 237.42,-117.27"/>
+<path fill="none" stroke="black" d="M359.17,-288.18C350.59,-278.15 340.09,-264.9 332.39,-252 314.53,-222.09 311.97,-213.2 301.39,-180 294.92,-159.71 290,-136.11 286.79,-118.32"/>
+<polygon fill="black" stroke="black" points="290.21,-117.58 285.05,-108.32 283.31,-118.77 290.21,-117.58"/>
</g>
<!-- datafusion->datafusion_physical_expr -->
<g id="edge12" class="edge">
<title>datafusion->datafusion_physical_expr</title>
-<path fill="none" stroke="black" d="M286.29,-291.36C266.11,-281.3 238.64,-266.85 215.74,-252 184.42,-231.7 150.92,-205.16 128.21,-186.36"/>
-<polygon fill="black" stroke="black" points="130.27,-183.53 120.35,-179.81 125.79,-188.91 130.27,-183.53"/>
+<path fill="none" stroke="black" d="M343.35,-290.49C321.99,-280.26 293.12,-265.94 268.39,-252 230.73,-230.77 189.03,-204.02 160.81,-185.42"/>
+<polygon fill="black" stroke="black" points="162.42,-182.28 152.15,-179.68 158.55,-188.12 162.42,-182.28"/>
</g>
<!-- datafusion->datafusion_jit -->
<g id="edge11" class="edge">
<title>datafusion->datafusion_jit</title>
-<path fill="none" stroke="black" d="M303.15,-288.2C296.95,-278.18 289.95,-264.93 286.74,-252 282.89,-236.47 282.67,-231.47 286.74,-216 289.27,-206.39 293.99,-196.69 298.95,-188.26"/>
-<polygon fill="black" stroke="black" points="301.98,-190.02 304.33,-179.68 296.05,-186.3 301.98,-190.02"/>
+<path fill="none" stroke="black" d="M359.57,-288.17C351.93,-278.35 343.39,-265.31 339.39,-252 334.79,-236.68 333.96,-231.05 339.39,-216 343.2,-205.43 350.15,-195.48 357.41,-187.12"/>
+<polygon fill="black" stroke="black" points="360.13,-189.33 364.37,-179.62 355,-184.56 360.13,-189.33"/>
</g>
<!-- datafusion->datafusion_row -->
<g id="edge13" class="edge">
<title>datafusion->datafusion_row</title>
-<path fill="none" stroke="black" d="M325.88,-288.41C331.47,-280.08 338.38,-269.8 344.63,-260.49"/>
-<polygon fill="black" stroke="black" points="347.69,-262.21 350.36,-251.96 341.88,-258.31 347.69,-262.21"/>
+<path fill="none" stroke="black" d="M387.42,-288.41C394.11,-279.91 402.38,-269.37 409.81,-259.91"/>
+<polygon fill="black" stroke="black" points="412.64,-261.98 416.07,-251.96 407.14,-257.66 412.64,-261.98"/>
</g>
<!-- datafusion_proto -->
<g id="node8" class="node">
<title>datafusion_proto</title>
-<ellipse fill="none" stroke="black" cx="265.74" cy="-378" rx="70.39" ry="18"/>
-<text text-anchor="middle" x="265.74" y="-374.3" font-family="Times,serif" font-size="14.00">datafusion_proto</text>
+<ellipse fill="none" stroke="black" cx="374.39" cy="-378" rx="89.08" ry="18"/>
+<text text-anchor="middle" x="374.39" y="-374.3" font-family="Times,serif" font-size="14.00">datafusion_proto</text>
</g>
<!-- datafusion_proto->datafusion -->
<g id="edge14" class="edge">
<title>datafusion_proto->datafusion</title>
-<path fill="none" stroke="black" d="M277.6,-360.05C283.44,-351.71 290.6,-341.49 297.06,-332.25"/>
-<polygon fill="black" stroke="black" points="300.12,-333.99 302.99,-323.79 294.39,-329.97 300.12,-333.99"/>
+<path fill="none" stroke="black" d="M374.39,-359.7C374.39,-351.98 374.39,-342.71 374.39,-334.11"/>
+<polygon fill="black" stroke="black" points="377.89,-334.1 374.39,-324.1 370.89,-334.1 377.89,-334.1"/>
</g>
<!-- ballista_core -->
<g id="node9" class="node">
<title>ballista_core</title>
-<ellipse fill="none" stroke="black" cx="209.74" cy="-450" rx="55.79" ry="18"/>
-<text text-anchor="middle" x="209.74" y="-446.3" font-family="Times,serif" font-size="14.00">ballista_core</text>
+<ellipse fill="none" stroke="black" cx="368.39" cy="-450" rx="69.59" ry="18"/>
+<text text-anchor="middle" x="368.39" y="-446.3" font-family="Times,serif" font-size="14.00">ballista_core</text>
</g>
<!-- ballista_core->datafusion -->
<g id="edge15" class="edge">
<title>ballista_core->datafusion</title>
-<path fill="none" stroke="black" d="M197.96,-432.24C186.23,-413.28 171.93,-382.16 186.74,-360 203.29,-335.24 233.76,-322.04 260.83,-315"/>
-<polygon fill="black" stroke="black" points="261.69,-318.39 270.61,-312.67 260.07,-311.58 261.69,-318.39"/>
+<path fill="none" stroke="black" d="M326.02,-435.61C307.33,-427.35 287.18,-414.67 276.39,-396 268.38,-382.15 268.2,-373.74 276.39,-360 286.68,-342.74 304.76,-330.76 322.61,-322.62"/>
+<polygon fill="black" stroke="black" points="324.03,-325.82 331.88,-318.7 321.3,-319.37 324.03,-325.82"/>
</g>
<!-- ballista_core->datafusion_proto -->
<g id="edge16" class="edge">
<title>ballista_core->datafusion_proto</title>
-<path fill="none" stroke="black" d="M223.01,-432.41C229.82,-423.91 238.24,-413.37 245.81,-403.91"/>
-<polygon fill="black" stroke="black" points="248.66,-405.95 252.18,-395.96 243.2,-401.58 248.66,-405.95"/>
+<path fill="none" stroke="black" d="M369.87,-431.7C370.53,-423.98 371.33,-414.71 372.06,-406.11"/>
+<polygon fill="black" stroke="black" points="375.56,-406.37 372.92,-396.1 368.58,-405.77 375.56,-406.37"/>
</g>
<!-- ballista_scheduler -->
<g id="node10" class="node">
<title>ballista_scheduler</title>
-<ellipse fill="none" stroke="black" cx="311.74" cy="-522" rx="74.19" ry="18"/>
-<text text-anchor="middle" x="311.74" y="-518.3" font-family="Times,serif" font-size="14.00">ballista_scheduler</text>
+<ellipse fill="none" stroke="black" cx="491.39" cy="-522" rx="94.78" ry="18"/>
+<text text-anchor="middle" x="491.39" y="-518.3" font-family="Times,serif" font-size="14.00">ballista_scheduler</text>
</g>
<!-- ballista_scheduler->datafusion -->
<g id="edge18" class="edge">
<title>ballista_scheduler->datafusion</title>
-<path fill="none" stroke="black" d="M320.38,-503.94C334.2,-474.39 358.4,-411.87 344.74,-360 342.21,-350.39 337.49,-340.69 332.54,-332.26"/>
-<polygon fill="black" stroke="black" points="335.43,-330.3 327.16,-323.68 329.5,-334.02 335.43,-330.3"/>
+<path fill="none" stroke="black" d="M494.17,-503.74C498.29,-472.21 502.58,-404.48 472.39,-360 461.01,-343.24 442.54,-331.32 424.68,-323.09"/>
+<polygon fill="black" stroke="black" points="426.01,-319.85 415.44,-319.11 423.24,-326.28 426.01,-319.85"/>
</g>
<!-- ballista_scheduler->ballista_core -->
<g id="edge17" class="edge">
<title>ballista_scheduler->ballista_core</title>
-<path fill="none" stroke="black" d="M288.08,-504.76C274.11,-495.18 256.25,-482.92 241.12,-472.53"/>
-<polygon fill="black" stroke="black" points="242.79,-469.44 232.57,-466.67 238.83,-475.21 242.79,-469.44"/>
+<path fill="none" stroke="black" d="M462.86,-504.76C445.54,-494.91 423.27,-482.23 404.7,-471.66"/>
+<polygon fill="black" stroke="black" points="406.34,-468.57 395.92,-466.67 402.88,-474.65 406.34,-468.57"/>
</g>
<!-- ballista_executor -->
<g id="node11" class="node">
<title>ballista_executor</title>
-<ellipse fill="none" stroke="black" cx="110.74" cy="-522" rx="70.69" ry="18"/>
-<text text-anchor="middle" x="110.74" y="-518.3" font-family="Times,serif" font-size="14.00">ballista_executor</text>
+<ellipse fill="none" stroke="black" cx="250.39" cy="-522" rx="89.88" ry="18"/>
+<text text-anchor="middle" x="250.39" y="-518.3" font-family="Times,serif" font-size="14.00">ballista_executor</text>
</g>
<!-- ballista_executor->datafusion -->
<g id="edge20" class="edge">
<title>ballista_executor->datafusion</title>
-<path fill="none" stroke="black" d="M113.08,-503.84C118.42,-471.27 134.47,-400.15 176.74,-360 200.3,-337.63 234.29,-324.42 262.65,-316.79"/>
-<polygon fill="black" stroke="black" points="263.58,-320.16 272.42,-314.31 261.87,-313.38 263.58,-320.16"/>
+<path fill="none" stroke="black" d="M247.08,-503.9C241.98,-472.36 235.71,-404.19 266.39,-360 279.09,-341.71 299.94,-329.46 319.95,-321.39"/>
+<polygon fill="black" stroke="black" points="321.23,-324.65 329.36,-317.87 318.77,-318.1 321.23,-324.65"/>
</g>
<!-- ballista_executor->ballista_core -->
<g id="edge19" class="edge">
<title>ballista_executor->ballista_core</title>
-<path fill="none" stroke="black" d="M133.7,-504.76C147.26,-495.18 164.6,-482.92 179.29,-472.53"/>
-<polygon fill="black" stroke="black" points="181.44,-475.3 187.59,-466.67 177.4,-469.58 181.44,-475.3"/>
+<path fill="none" stroke="black" d="M277.76,-504.76C294.22,-495 315.35,-482.46 333.07,-471.95"/>
+<polygon fill="black" stroke="black" points="335.17,-474.78 341.98,-466.67 331.59,-468.76 335.17,-474.78"/>
</g>
<!-- ballista -->
<g id="node12" class="node">
<title>ballista</title>
-<ellipse fill="none" stroke="black" cx="311.74" cy="-594" rx="36.29" ry="18"/>
-<text text-anchor="middle" x="311.74" y="-590.3" font-family="Times,serif" font-size="14.00">ballista</text>
+<ellipse fill="none" stroke="black" cx="368.39" cy="-594" rx="44.69" ry="18"/>
+<text text-anchor="middle" x="368.39" y="-590.3" font-family="Times,serif" font-size="14.00">ballista</text>
</g>
<!-- ballista->datafusion -->
<g id="edge21" class="edge">
<title>ballista->datafusion</title>
-<path fill="none" stroke="black" d="M340.75,-582.97C359.83,-574.68 383.32,-560.84 394.74,-540 417.46,-498.54 407.39,-448.69 361.74,-360 356.23,-349.28 348.04,-339.01 340.05,-330.4"/>
-<polygon fill="black" stroke="black" points="342.32,-327.72 332.84,-323 337.31,-332.61 342.32,-327.72"/>
+<path fill="none" stroke="black" d="M325.2,-588.69C268.21,-582 172.58,-566.99 151.39,-540 101.01,-475.84 142.07,-446.76 247.39,-360 268.62,-342.51 296.44,-330.03 320.63,-321.59"/>
+<polygon fill="black" stroke="black" points="322.02,-324.82 330.4,-318.34 319.81,-318.18 322.02,-324.82"/>
</g>
<!-- ballista->ballista_core -->
<g id="edge22" class="edge">
<title>ballista->ballista_core</title>
-<path fill="none" stroke="black" d="M284.06,-582.14C265.59,-573.47 242.29,-559.53 228.74,-540 216.19,-521.9 211.67,-497.12 210.16,-478.3"/>
-<polygon fill="black" stroke="black" points="213.65,-478.06 209.59,-468.27 206.67,-478.46 213.65,-478.06"/>
+<path fill="none" stroke="black" d="M368.39,-575.87C368.39,-551.67 368.39,-507.21 368.39,-478.39"/>
+<polygon fill="black" stroke="black" points="371.89,-478.19 368.39,-468.19 364.89,-478.19 371.89,-478.19"/>
</g>
<!-- ballista->ballista_scheduler -->
<g id="edge23" class="edge">
<title>ballista->ballista_scheduler</title>
-<path fill="none" stroke="black" d="M311.74,-575.7C311.74,-567.98 311.74,-558.71 311.74,-550.11"/>
-<polygon fill="black" stroke="black" points="315.24,-550.1 311.74,-540.1 308.24,-550.1 315.24,-550.1"/>
+<path fill="none" stroke="black" d="M393.28,-578.83C410.69,-568.93 434.25,-555.52 453.92,-544.32"/>
+<polygon fill="black" stroke="black" points="455.69,-547.34 462.65,-539.35 452.23,-541.26 455.69,-547.34"/>
</g>
<!-- ballista->ballista_executor -->
<g id="edge24" class="edge">
<title>ballista->ballista_executor</title>
-<path fill="none" stroke="black" d="M282.94,-582.97C251.02,-571.85 198.96,-553.72 160.42,-540.3"/>
-<polygon fill="black" stroke="black" points="161.53,-536.98 150.94,-537 159.23,-543.59 161.53,-536.98"/>
+<path fill="none" stroke="black" d="M344.22,-578.67C327.65,-568.83 305.37,-555.61 286.67,-544.52"/>
+<polygon fill="black" stroke="black" points="288.21,-541.37 277.83,-539.28 284.64,-547.39 288.21,-541.37"/>
</g>
<!-- datafusion_cli -->
<g id="node13" class="node">
<title>datafusion_cli</title>
-<ellipse fill="none" stroke="black" cx="381.74" cy="-666" rx="60.39" ry="18"/>
-<text text-anchor="middle" x="381.74" y="-662.3" font-family="Times,serif" font-size="14.00">datafusion_cli</text>
+<ellipse fill="none" stroke="black" cx="594.39" cy="-450" rx="74.99" ry="18"/>
+<text text-anchor="middle" x="594.39" y="-446.3" font-family="Times,serif" font-size="14.00">datafusion_cli</text>
</g>
<!-- datafusion_cli->datafusion -->
<g id="edge25" class="edge">
<title>datafusion_cli->datafusion</title>
-<path fill="none" stroke="black" d="M396.25,-648.39C416.75,-623.13 451.74,-572.75 451.74,-523 451.74,-523 451.74,-523 451.74,-449 451.74,-391.36 392.55,-348.14 351.98,-325.29"/>
-<polygon fill="black" stroke="black" points="353.58,-322.18 343.13,-320.46 350.23,-328.32 353.58,-322.18"/>
+<path fill="none" stroke="black" d="M581.21,-432.21C565.1,-412.65 536.18,-380.39 505.39,-360 481.01,-343.86 450.94,-331.26 425.76,-322.4"/>
+<polygon fill="black" stroke="black" points="426.61,-318.99 416.01,-319.07 424.35,-325.62 426.61,-318.99"/>
</g>
-<!-- datafusion_cli->ballista -->
+<!-- ballista_cli -->
+<g id="node14" class="node">
+<title>ballista_cli</title>
+<ellipse fill="none" stroke="black" cx="614.39" cy="-666" rx="59.59" ry="18"/>
+<text text-anchor="middle" x="614.39" y="-662.3" font-family="Times,serif" font-size="14.00">ballista_cli</text>
+</g>
+<!-- ballista_cli->datafusion -->
<g id="edge26" class="edge">
-<title>datafusion_cli->ballista</title>
-<path fill="none" stroke="black" d="M365.15,-648.41C356,-639.26 344.51,-627.76 334.52,-617.78"/>
-<polygon fill="black" stroke="black" points="336.8,-615.11 327.25,-610.51 331.85,-620.06 336.8,-615.11"/>
+<title>ballista_cli->datafusion</title>
+<path fill="none" stroke="black" d="M631.99,-648.48C656.35,-623.72 697.39,-574.4 697.39,-523 697.39,-523 697.39,-523 697.39,-449 697.39,-339.22 539.08,-313.55 444.13,-308.03"/>
+<polygon fill="black" stroke="black" points="444.18,-304.53 434.01,-307.51 443.81,-311.52 444.18,-304.53"/>
+</g>
+<!-- ballista_cli->ballista -->
+<g id="edge28" class="edge">
+<title>ballista_cli->ballista</title>
+<path fill="none" stroke="black" d="M572.38,-653.05C528.24,-640.49 459.04,-620.8 413.7,-607.89"/>
+<polygon fill="black" stroke="black" points="414.42,-604.46 403.84,-605.09 412.5,-611.19 414.42,-604.46"/>
+</g>
+<!-- ballista_cli->datafusion_cli -->
+<g id="edge27" class="edge">
+<title>ballista_cli->datafusion_cli</title>
+<path fill="none" stroke="black" d="M612.79,-647.85C609.33,-610.83 601.13,-523.18 596.95,-478.39"/>
+<polygon fill="black" stroke="black" points="600.41,-477.86 596,-468.23 593.45,-478.51 600.41,-477.86"/>
</g>
</g>
</svg>
diff --git a/dev/update_ballista_versions.py b/dev/update_ballista_versions.py
index d30f027d7..fa2c8bc18 100755
--- a/dev/update_ballista_versions.py
+++ b/dev/update_ballista_versions.py
@@ -35,7 +35,7 @@ def update_cargo_toml(cargo_toml: str, new_version: str):
data = f.read()
doc = tomlkit.parse(data)
- if "ballista/" in cargo_toml:
+ if "ballista/" in cargo_toml or "ballista-cli/" in cargo_toml:
doc.get('package')['version'] = new_version
# ballista crates also depend on each other
@@ -77,12 +77,12 @@ def main():
ballista_crates = set([
os.path.join(repo_root, rel_path, "Cargo.toml")
for rel_path in [
+ 'ballista-cli',
'ballista-examples',
'ballista/rust/core',
'ballista/rust/scheduler',
'ballista/rust/executor',
'ballista/rust/client',
- 'datafusion-cli',
]
])
new_version = args.new_version
diff --git a/docs/source/cli/index.rst b/docs/source/cli/index.rst
index 89f7f5da7..bc22bf6c8 100644
--- a/docs/source/cli/index.rst
+++ b/docs/source/cli/index.rst
@@ -23,10 +23,15 @@ The Arrow DataFusion CLI is a command-line interactive SQL utility that allows
queries to be executed against CSV and Parquet files. It is a convenient way to
try DataFusion out with your own data sources.
+Install and run using Cargo
+===========================
+
+The easiest way to install DataFusion CLI a spin is via `cargo install datafusion-cli`.
+
Install and run using Homebrew (on MacOS)
=========================================
-The easiest way to give DataFusion CLI a spin is via Homebrew (on MacOS). Install it as any other pre-built software like this:
+DataFusion CLI can also be installed via Homebrew (on MacOS). Install it as any other pre-built software like this:
.. code-block:: bash
@@ -41,26 +46,19 @@ The easiest way to give DataFusion CLI a spin is via Homebrew (on MacOS). Instal
datafusion-cli
-Run using Cargo
-===============
-
-Use the following commands to clone this repository and run the CLI. This will require the Rust toolchain to be installed. Rust can be installed from `https://rustup.rs <https://rustup.rs/>`_.
-
-.. code-block:: bash
-
- git clone https://github.com/apache/arrow-datafusion
- cd arrow-datafusion/datafusion-cli
- cargo run --release
-
Run using Docker
================
+There is no officially published Docker image for the DataFusion CLI, so it is necessary to build from source
+instead.
+
Use the following commands to clone this repository and build a Docker image containing the CLI tool. Note that there is :code:`.dockerignore` file in the root of the repository that may need to be deleted in order for this to work.
.. code-block:: bash
git clone https://github.com/apache/arrow-datafusion
+ git checkout 7.0.0
cd arrow-datafusion
docker build -f datafusion-cli/Dockerfile . --tag datafusion-cli
docker run -it -v $(your_data_location):/data datafusion-cli
@@ -71,25 +69,22 @@ Usage
.. code-block:: bash
- DataFusion 5.1.0-SNAPSHOT
- DataFusion is an in-memory query engine that uses Apache Arrow as the memory model. It supports executing SQL queries
- against CSV and Parquet files as well as querying directly against in-memory data.
+ Apache Arrow <de...@arrow.apache.org>
+ Command Line Client for DataFusion query engine and Ballista distributed computation engine.
USAGE:
- datafusion-cli [FLAGS] [OPTIONS]
-
- FLAGS:
- -h, --help Prints help information
- -q, --quiet Reduce printing other than the results and work quietly
- -V, --version Prints version information
+ datafusion-cli [OPTIONS]
OPTIONS:
- -c, --batch-size <batch-size> The batch size of each query, or use DataFusion default
- -p, --data-path <data-path> Path to your data, default to current directory
- -f, --file <file>... Execute commands from file(s), then exit
- --format <format> Output format [default: table] [possible values: csv, tsv, table, json, ndjson]
- --host <host> Ballista scheduler host
- --port <port> Ballista scheduler port
+ -c, --batch-size <BATCH_SIZE> The batch size of each query, or use DataFusion default
+ -f, --file <FILE>... Execute commands from file(s), then exit
+ --format <FORMAT> [default: table] [possible values: csv, tsv, table, json,
+ nd-json]
+ -h, --help Print help information
+ -p, --data-path <DATA_PATH> Path to your data, default to current directory
+ -q, --quiet Reduce printing other than the results and work quietly
+ -r, --rc <RC>... Run the provided files on startup instead of ~/.datafusionrc
+ -V, --version Print version information
Type `exit` or `quit` to exit the CLI.
diff --git a/docs/source/user-guide/cli.md b/docs/source/user-guide/cli.md
index 9561b5e1d..ec38aad9a 100644
--- a/docs/source/user-guide/cli.md
+++ b/docs/source/user-guide/cli.md
@@ -51,7 +51,7 @@ $ echo "1,2" > data.csv
```bash
$ datafusion-cli
-DataFusion CLI v5.1.0-SNAPSHOT
+DataFusion CLI v7.0.0
> CREATE EXTERNAL TABLE foo (a INT, b INT) STORED AS CSV LOCATION 'data.csv';
0 rows in set. Query took 0.001 seconds.
diff --git a/docs/source/cli/index.rst b/docs/source/user-guide/distributed/clients/cli.rst
similarity index 52%
copy from docs/source/cli/index.rst
copy to docs/source/user-guide/distributed/clients/cli.rst
index 89f7f5da7..715185082 100644
--- a/docs/source/cli/index.rst
+++ b/docs/source/user-guide/distributed/clients/cli.rst
@@ -16,54 +16,33 @@
.. under the License.
=======================
-DataFusion Command-line
+Ballista Command-line
=======================
-The Arrow DataFusion CLI is a command-line interactive SQL utility that allows
+The Arrow Ballista CLI is a command-line interactive SQL utility that allows
queries to be executed against CSV and Parquet files. It is a convenient way to
-try DataFusion out with your own data sources.
+try Ballista out with your own data sources.
-Install and run using Homebrew (on MacOS)
-=========================================
-
-The easiest way to give DataFusion CLI a spin is via Homebrew (on MacOS). Install it as any other pre-built software like this:
-
-.. code-block:: bash
-
- brew install datafusion
- # ==> Downloading https://ghcr.io/v2/homebrew/core/datafusion/manifests/5.0.0
- # ######################################################################## 100.0%
- # ==> Downloading https://ghcr.io/v2/homebrew/core/datafusion/blobs/sha256:9ecc8a01be47ceb9a53b39976696afa87c0a8
- # ==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:9ecc8a01be47ceb9a53b39976
- # ######################################################################## 100.0%
- # ==> Pouring datafusion--5.0.0.big_sur.bottle.tar.gz
- # 🍺 /usr/local/Cellar/datafusion/5.0.0: 9 files, 17.4MB
-
- datafusion-cli
-
-Run using Cargo
-===============
-
-Use the following commands to clone this repository and run the CLI. This will require the Rust toolchain to be installed. Rust can be installed from `https://rustup.rs <https://rustup.rs/>`_.
-
-.. code-block:: bash
-
- git clone https://github.com/apache/arrow-datafusion
- cd arrow-datafusion/datafusion-cli
- cargo run --release
+Install and run using Cargo
+===========================
+The easiest way to install Ballista CLI a spin is via `cargo install ballista-cli`.
Run using Docker
================
+There is no officially published Docker image for the Ballista CLI, so it is necessary to build from source
+instead.
+
Use the following commands to clone this repository and build a Docker image containing the CLI tool. Note that there is :code:`.dockerignore` file in the root of the repository that may need to be deleted in order for this to work.
.. code-block:: bash
git clone https://github.com/apache/arrow-datafusion
+ git checkout 7.0.0
cd arrow-datafusion
- docker build -f datafusion-cli/Dockerfile . --tag datafusion-cli
- docker run -it -v $(your_data_location):/data datafusion-cli
+ docker build -f ballista-cli/Dockerfile . --tag ballista-cli
+ docker run -it -v $(your_data_location):/data ballista-cli
Usage
@@ -71,25 +50,24 @@ Usage
.. code-block:: bash
- DataFusion 5.1.0-SNAPSHOT
- DataFusion is an in-memory query engine that uses Apache Arrow as the memory model. It supports executing SQL queries
- against CSV and Parquet files as well as querying directly against in-memory data.
+ Apache Arrow <de...@arrow.apache.org>
+ Command Line Client for Ballista distributed query engine.
USAGE:
- datafusion-cli [FLAGS] [OPTIONS]
-
- FLAGS:
- -h, --help Prints help information
- -q, --quiet Reduce printing other than the results and work quietly
- -V, --version Prints version information
+ ballista-cli [OPTIONS]
OPTIONS:
- -c, --batch-size <batch-size> The batch size of each query, or use DataFusion default
- -p, --data-path <data-path> Path to your data, default to current directory
- -f, --file <file>... Execute commands from file(s), then exit
- --format <format> Output format [default: table] [possible values: csv, tsv, table, json, ndjson]
- --host <host> Ballista scheduler host
- --port <port> Ballista scheduler port
+ -c, --batch-size <BATCH_SIZE> The batch size of each query, or use DataFusion default
+ -f, --file <FILE>... Execute commands from file(s), then exit
+ --format <FORMAT> [default: table] [possible values: csv, tsv, table, json,
+ nd-json]
+ -h, --help Print help information
+ --host <HOST> Ballista scheduler host
+ -p, --data-path <DATA_PATH> Path to your data, default to current directory
+ --port <PORT> Ballista scheduler port
+ -q, --quiet Reduce printing other than the results and work quietly
+ -r, --rc <RC>... Run the provided files on startup instead of ~/.datafusionrc
+ -V, --version Print version information
Type `exit` or `quit` to exit the CLI.
diff --git a/docs/source/user-guide/distributed/clients/index.rst b/docs/source/user-guide/distributed/clients/index.rst
index c9eb1e1f5..6199bca5e 100644
--- a/docs/source/user-guide/distributed/clients/index.rst
+++ b/docs/source/user-guide/distributed/clients/index.rst
@@ -21,5 +21,6 @@ Clients
.. toctree::
:maxdepth: 2
+ cli
rust
python