You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alexandreyc (via GitHub)" <gi...@apache.org> on 2024/04/16 12:46:54 UTC
[PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
alexandreyc opened a new pull request, #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725
Hey!
As discussed in #1723, here is the first PR of the new complete Rust implementation.
It includes:
- Traits and structs definitions for the public abstract API
- An implementation of a dummy driver for testing purposes (no test currently because we need the driver manager and exporter to have relevant tests)
- Constants from `adbc.h`
I put the code in `/rust2` but I can move it to `/rust` if you prefer.
CC @mbrobbel @paleolimbot @zeroshade @lidavidm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568857747
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
```rust
trait DatabaseOption {
fn uri(&self) -> Option<String> {
None
}
fn username(&self) -> Option<String> {
None
}
fn password(&self) -> Option<String> {
None
}
}
trait Database {
type ConnectionType: Connection;
type OptionType: DatabaseOption;
fn new_connection_with_opts(&self, opts: Self::OptionType) -> Result<Self::ConnectionType>;
}
struct MyDatabase;
struct MyDatabaseOptions {
uri: String,
other_key: bool,
}
impl DatabaseOption for MyDatabaseOptions {
fn uri(&self) -> Option<String> {
Some(self.uri.clone())
}
}
impl Database for MyDatabase {
type ConnectionType = ...;
type OptionType = MyDatabaseOptions;
...
}
let db = MyDatabase;
let connection = db.new_connection_with_opts(
MyDatabaseOptions { uri: "...".to_string(), other_key: false }
)?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1570101609
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,500 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+#[non_exhaustive]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
Review Comment:
I would prefer to keep `Double` because that's what the C spec uses.
Go is in a similar situation with a "double" type named `float64` and it uses `SetOptionDouble(key string, value float64) error`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569356502
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
Review Comment:
Note to myself: currently I use interior mutability in the driver manager. After switching to `&mut self` I was hoping to remove this interior mutability but it's not possible because even methods that look like "read-only" (`Statement::get_parameters_schema` for instance) need a mutable access to the underlying FFI type because all C functions take non-const pointer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1994455687
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
Next week I will open an issue/PR with examples in unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568501581
##########
rust2/core/Cargo.toml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "adbc_core"
+description = "Public abstract API, driver manager and driver exporter"
+version = { workspace = true }
+edition = { workspace = true }
+authors = { workspace = true }
+license = { workspace = true }
+
+[dependencies]
+arrow = { workspace = true }
+libloading = "0.8"
Review Comment:
For the moment we need to keep it here because we `impl From<libloading::Error> for Error` in `error.rs`.
But this raises an interesting question: how should we structure the workspace?
Currently, we have:
- A `core` crate that contains traits/structs for the API, the driver manager and the driver exporter
- A crate for each driver in `drivers/*`
Should we split the driver manager and the driver exporter into different crates?
One possible solution:
- Move the driver manager to `drivers/` (it's a driver after all)
- Keep the driver exporter into the core crate (because it does not introduce any new dependency and it's tightly coupled to the abstract API)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568362620
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
+ /// Additional metadata. Introduced in ADBC 1.1.0.
+ pub details: Option<Vec<(String, Vec<u8>)>>,
Review Comment:
Make sense. Let's keep the `Option` and `Vec`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568525770
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1570348776
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
So, I propose to go forward with this design for now and reconsider the question when we have more substance to discuss. Are you okay?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569003762
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
I've tried to come up with a similar design that provides all the required capabilities but I'm feeling stuck...
A few points:
- `new_connection_with_opts` takes *connection* options, not database options (same logic applies to other `new_` methods). This would forces us to duplicate the associated type into two traits.
- How do we get/set non-standard option in a generic way? That's required for the driver exporter. We also need to be able to set standard option in a generic way (but this can be done by adding `set_uri` method).
- According to the spec, we must be able to set an Int option and get it back as String (or any other combination of types)
- We must assume that each standard option key is associated with a specific type for its value. It seems to be the case as of now but it's not explicit in the spec.
I will put more thougth into it, but I've tried different designs before settling on this one. While not perfect, it's relatively simple and can accomodate the C API...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569003762
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
I've tried to come up with a similar design that provides all the required capabilities but I'm feeling stuck...
A few points:
- `new_connection_with_opts` takes *connection* options, not database options (same logic applies to other `new_` methods). This would forces us to duplicate the associated type into two traits.
- How do we get/set non-standard option in a generic way? That's required for the driver exporter. We also need to be able to set standard option in a generic way (but this can be done by adding `set_uri` method).
- According to the spec, we must be able to set an Int option and get it back as String (or any other combination of types)
- We must assume that each standard option key is associated with a specific type for its value. It seems to the case as of now but it's not explicit in the spec.
I will put more thougth into it, but I've tried different designs before settling on this one. While not perfect, it's relatively simple and can accomodate the C API...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568355944
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
+ /// Additional metadata. Introduced in ADBC 1.1.0.
+ pub details: Option<Vec<(String, Vec<u8>)>>,
Review Comment:
1. I think we cannot use a map here because the C API provides `AdbcErrorGetDetail(error, index)` which access the error detail by index rather than key... I'm not able to find any map usage in the Go implementation for this.
2. I would prefer to keep the `Option` here because `None` encodes the fact that this field does not exist in ADBC 1.0.0 rather than an empty collection. To me, it conveys a better semantics. No?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#issuecomment-2059646029
> I'm not very familiar with Github Actions so I need a bit more time to read doc before integrating the new folder to CI.
The final CI may be a bit different, but for the purposes of getting the review started, perhaps just modify:
https://github.com/apache/arrow-adbc/blob/aa04aadccd319e6fa3abb07154fa8d87b58d5c21/.github/workflows/rust.yml#L41
...such that it reads `working-directory: rust2`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568400988
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
Review Comment:
Agree. I was only suggesting this for fieldless enums.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569418062
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
Yes I was thinking of that too: we could build a higher level API which is less error-prone, more idiomatic and that does not have the constraint to accomodate the C API (it will probably not implement all the possible semantics of the C API though).
In that case, this API must only be used by consumer of Rust driver (driver implementors must implement the low-level API to be exportable to C).
Let's keep that in mind for the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1570343548
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
I've added the `new_ingest_statement` method, please take a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "aljazerzen (via GitHub)" <gi...@apache.org>.
aljazerzen commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569199195
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
> See here for instance: https://arrow.apache.org/adbc/current/format/specification.html#bulk-data-ingestion
Ahhh, this is the part of the spec that I was missing!
I like your proposal. It is a good trade off between the two extremes.
It would also not be hard to make a wrapper API that does not use options for such stuff at all and has idiomatic method structure that cannot be misused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569409474
##########
rust2/core/Cargo.toml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "adbc_core"
+description = "Public abstract API, driver manager and driver exporter"
+version = { workspace = true }
+edition = { workspace = true }
+authors = { workspace = true }
+license = { workspace = true }
+
+[dependencies]
+arrow = { workspace = true }
+libloading = "0.8"
Review Comment:
After trying to move the driver manager to its own crate I think that we should reconsider this. In doing so I had to publicly expose a lot of item (mostly related to FFI) which are just implementation details...
Alternatively, we could gate the driver manager behind a feature flag. Maybe we could do the same for the driver exporter even though it's less relevant because it does not introduce any additional dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "aljazerzen (via GitHub)" <gi...@apache.org>.
aljazerzen commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1570440217
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
Ok, great, we are on the same page.
Now that we also have this idea for a high-level API, there is less need for `new_ingest_statement`, since this API is not meant to be consumer-facing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569013032
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
> We must assume that each standard option key is associated with a specific type for its value. It seems to the case as of now but it's not explicit in the spec.
I don't think this will necessarily hold, either.
Also, re: the original proposal, not all drivers necessarily have or require uri, username, or password fields; the latter two would never apply to SQLite, for instance, and SQLite doesn't require a URI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567518935
##########
rust2/core/Cargo.toml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "adbc_core"
+description = "Public abstract API, driver manager and driver exporter"
+version = { workspace = true }
+edition = { workspace = true }
+authors = { workspace = true }
+license = { workspace = true }
+
+[dependencies]
+arrow = { workspace = true }
+libloading = "0.8"
Review Comment:
I think this should be added in a follow-up PR that adds the driver manager.
```suggestion
```
It would be nice if the core crate only defines the core traits and the used types in those traits.
##########
rust2/core/src/ffi/constants.rs:
##########
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::os::raw::c_int;
+
+use super::types::FFI_AdbcStatusCode;
+
+pub const ADBC_STATUS_OK: FFI_AdbcStatusCode = 0;
+pub const ADBC_STATUS_UNKNOWN: FFI_AdbcStatusCode = 1;
+pub const ADBC_STATUS_NOT_IMPLEMENTED: FFI_AdbcStatusCode = 2;
+pub const ADBC_STATUS_NOT_FOUND: FFI_AdbcStatusCode = 3;
+pub const ADBC_STATUS_ALREADY_EXISTS: FFI_AdbcStatusCode = 4;
+pub const ADBC_STATUS_INVALID_ARGUMENT: FFI_AdbcStatusCode = 5;
+pub const ADBC_STATUS_INVALID_STATE: FFI_AdbcStatusCode = 6;
+pub const ADBC_STATUS_INVALID_DATA: FFI_AdbcStatusCode = 7;
+pub const ADBC_STATUS_INTEGRITY: FFI_AdbcStatusCode = 8;
+pub const ADBC_STATUS_INTERNAL: FFI_AdbcStatusCode = 9;
+pub const ADBC_STATUS_IO: FFI_AdbcStatusCode = 10;
+pub const ADBC_STATUS_CANCELLED: FFI_AdbcStatusCode = 11;
+pub const ADBC_STATUS_TIMEOUT: FFI_AdbcStatusCode = 12;
+pub const ADBC_STATUS_UNAUTHENTICATED: FFI_AdbcStatusCode = 13;
+pub const ADBC_STATUS_UNAUTHORIZED: FFI_AdbcStatusCode = 14;
+
+pub const ADBC_VERSION_1_0_0: i32 = 1000000;
+pub const ADBC_VERSION_1_1_0: i32 = 1001000;
Review Comment:
Both Java and Go are using `i64` here.
```suggestion
pub const ADBC_VERSION_1_0_0: i64 = 1_000_000;
pub const ADBC_VERSION_1_1_0: i64 = 1_001_000;
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
Review Comment:
In order to cancel, the `get_` methods should probably also be `&mut self` to avoid having to do interior mutability to support the cancel method. A connection needs some state to support cancel, I think?
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
+ /// Catalogs, schemas, tables, and columns.
+ All,
+ /// Catalogs only.
+ Catalogs,
+ /// Catalogs and schemas.
+ Schemas,
+ /// Catalogs, schemas, and tables.
+ Tables,
+ /// Catalogs, schemas, tables, and columns. Identical to [ObjectDepth::All].
+ Columns,
+}
+
+impl From<ObjectDepth> for c_int {
+ fn from(value: ObjectDepth) -> Self {
+ match value {
+ ObjectDepth::All => constants::ADBC_OBJECT_DEPTH_ALL,
+ ObjectDepth::Catalogs => constants::ADBC_OBJECT_DEPTH_CATALOGS,
+ ObjectDepth::Schemas => constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS,
+ ObjectDepth::Tables => constants::ADBC_OBJECT_DEPTH_TABLES,
+ ObjectDepth::Columns => constants::ADBC_OBJECT_DEPTH_COLUMNS,
+ }
+ }
+}
+
+impl TryFrom<c_int> for ObjectDepth {
+ type Error = Error;
+
+ fn try_from(value: c_int) -> Result<Self, Error> {
+ match value {
+ constants::ADBC_OBJECT_DEPTH_ALL => Ok(ObjectDepth::All),
+ constants::ADBC_OBJECT_DEPTH_CATALOGS => Ok(ObjectDepth::Catalogs),
+ constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS => Ok(ObjectDepth::Schemas),
+ constants::ADBC_OBJECT_DEPTH_TABLES => Ok(ObjectDepth::Tables),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown object depth: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Database option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionDatabase {
+ /// Canonical option key for URIs.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Uri,
+ /// Canonical option key for usernames.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Username,
+ /// Canonical option key for passwords.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Password,
+ /// Driver-specific key.
+ Other(String),
+}
+
+impl AsRef<str> for OptionDatabase {
+ fn as_ref(&self) -> &str {
+ match self {
+ Self::Uri => constants::ADBC_OPTION_URI,
+ Self::Username => constants::ADBC_OPTION_USERNAME,
+ Self::Password => constants::ADBC_OPTION_PASSWORD,
+ Self::Other(key) => key,
+ }
+ }
+}
+
+impl From<&str> for OptionDatabase {
+ fn from(value: &str) -> Self {
+ match value {
+ constants::ADBC_OPTION_URI => Self::Uri,
+ constants::ADBC_OPTION_USERNAME => Self::Username,
+ constants::ADBC_OPTION_PASSWORD => Self::Password,
+ key => Self::Other(key.into()),
+ }
+ }
+}
+
+/// Connection option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionConnection {
+ /// Whether autocommit is enabled.
+ AutoCommit,
+ /// Whether the current connection should be restricted to being read-only.
+ ReadOnly,
+ /// The catalog used by the connection.
+ /// # Since
+ /// ADBC API revision 1.1.0
+ CurrentCatalog,
+ /// The database schema used by the connection.
+ /// # Since
+ /// ADBC API revision 1.1.0
+ CurrentSchema,
+ /// The isolation level of the connection. See [IsolationLevel].
+ IsolationLevel,
+ /// Driver-specific key.
+ Other(String),
+}
+
+impl AsRef<str> for OptionConnection {
+ fn as_ref(&self) -> &str {
+ match self {
+ Self::AutoCommit => constants::ADBC_CONNECTION_OPTION_AUTOCOMMIT,
+ Self::ReadOnly => constants::ADBC_CONNECTION_OPTION_READ_ONLY,
+ Self::CurrentCatalog => constants::ADBC_CONNECTION_OPTION_CURRENT_CATALOG,
+ Self::CurrentSchema => constants::ADBC_CONNECTION_OPTION_CURRENT_DB_SCHEMA,
+ Self::IsolationLevel => constants::ADBC_CONNECTION_OPTION_ISOLATION_LEVEL,
+ Self::Other(key) => key,
+ }
+ }
+}
+
+impl From<&str> for OptionConnection {
+ fn from(value: &str) -> Self {
+ match value {
+ constants::ADBC_CONNECTION_OPTION_AUTOCOMMIT => Self::AutoCommit,
+ constants::ADBC_CONNECTION_OPTION_READ_ONLY => Self::ReadOnly,
+ constants::ADBC_CONNECTION_OPTION_CURRENT_CATALOG => Self::CurrentCatalog,
+ constants::ADBC_CONNECTION_OPTION_CURRENT_DB_SCHEMA => Self::CurrentSchema,
+ constants::ADBC_CONNECTION_OPTION_ISOLATION_LEVEL => Self::IsolationLevel,
+ key => Self::Other(key.into()),
+ }
+ }
+}
+
+/// Statement option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionStatement {
Review Comment:
```suggestion
#[non_exhaustive]
#[derive(PartialEq, Eq, Hash, Debug)]
pub enum OptionStatement {
```
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
+ /// Additional metadata. Introduced in ADBC 1.1.0.
+ pub details: Option<Vec<(String, Vec<u8>)>>,
+}
+
+/// Result type wrapping [Error].
+pub type Result<T> = std::result::Result<T, Error>;
+
+impl Error {
+ pub fn with_message_and_status(message: &str, status: Status) -> Self {
Review Comment:
Nit: this would allow moving a `String` (vs making a copy).
```suggestion
pub fn with_message_and_status(message: impl Into<String>, status: Status) -> Self {
```
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
+ /// Additional metadata. Introduced in ADBC 1.1.0.
+ pub details: Option<Vec<(String, Vec<u8>)>>,
+}
+
+/// Result type wrapping [Error].
+pub type Result<T> = std::result::Result<T, Error>;
+
+impl Error {
+ pub fn with_message_and_status(message: &str, status: Status) -> Self {
+ Self {
+ message: message.into(),
+ status,
+ vendor_code: 0,
+ sqlstate: [0; 5],
+ details: None,
+ }
+ }
+}
+
+impl Display for Error {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(
+ f,
+ "{:?}: {} (sqlstate: {:?}, vendor_code: {})",
+ self.status, self.message, self.sqlstate, self.vendor_code
+ )
+ }
+}
+
+impl std::error::Error for Error {}
+
+impl From<ArrowError> for Error {
+ fn from(value: ArrowError) -> Self {
+ Self {
+ message: value.to_string(),
+ status: Status::Internal,
+ vendor_code: 0,
+ sqlstate: [0; 5],
+ details: None,
+ }
+ }
+}
+
+impl From<NulError> for Error {
+ fn from(value: NulError) -> Self {
+ Self {
+ message: format!(
+ "Interior null byte was found at position {}",
+ value.nul_position()
+ ),
+ status: Status::InvalidData,
+ vendor_code: 0,
+ sqlstate: [0; 5],
+ details: None,
+ }
+ }
+}
+
+impl From<libloading::Error> for Error {
+ fn from(value: libloading::Error) -> Self {
+ Self {
+ message: format!("Error with dynamic library: {}", value),
+ status: Status::Internal,
+ vendor_code: 0,
+ sqlstate: [0; 5],
+ details: None,
+ }
+ }
+}
+
+impl From<std::str::Utf8Error> for Error {
+ fn from(value: std::str::Utf8Error) -> Self {
+ Self {
+ message: format!("Error while decoding UTF-8: {}", value),
Review Comment:
Nit:
```suggestion
message: format!("Error while decoding UTF-8: {value}"),
```
##########
rust2/Cargo.toml:
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[workspace]
+members = ["core", "drivers/*"]
+resolver = "2"
+
+[workspace.package]
+version = "0.1.0"
+description = "Rust implementation of Arrow Database Connectivity (ADBC)"
+edition = "2021"
+authors = ["Apache Arrow <de...@arrow.apache.org>"]
+license = "Apache-2.0"
+
+[workspace.dependencies]
+arrow = { version = "51.0.0", default-features = false, features = ["ffi"] }
Review Comment:
Instead of depending on `arrow`, we can reduce the dependency tree by depending only on the crates that are required. It seems we only need `arrow-array` and `arrow-schema` for the core crate.
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
+ /// Additional metadata. Introduced in ADBC 1.1.0.
+ pub details: Option<Vec<(String, Vec<u8>)>>,
Review Comment:
Should we consider using a map here? It seems the spec is not strict about this, and this is what the other languages do, but what to do as consumer if there are duplicate keys?
We could remove the `Option` and use an empty vec instead of `None`:
```suggestion
pub details: Vec<(String, Vec<u8>)>,
```
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
Review Comment:
Why not `u8`?
```suggestion
pub sqlstate: [u8; 5],
```
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
Review Comment:
We could add the [`non_exhaustive`](https://doc.rust-lang.org/reference/attributes/type_system.html#the-non_exhaustive-attribute) attribute here to prevent breaking changes when more status variants are added.
```suggestion
#[non_exhaustive]
#[derive(Debug, Copy, Clone, PartialEq, Eq)]
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
Review Comment:
```suggestion
&mut self,
```
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
+ /// Catalogs, schemas, tables, and columns.
+ All,
+ /// Catalogs only.
+ Catalogs,
+ /// Catalogs and schemas.
+ Schemas,
+ /// Catalogs, schemas, and tables.
+ Tables,
+ /// Catalogs, schemas, tables, and columns. Identical to [ObjectDepth::All].
+ Columns,
+}
+
+impl From<ObjectDepth> for c_int {
+ fn from(value: ObjectDepth) -> Self {
+ match value {
+ ObjectDepth::All => constants::ADBC_OBJECT_DEPTH_ALL,
+ ObjectDepth::Catalogs => constants::ADBC_OBJECT_DEPTH_CATALOGS,
+ ObjectDepth::Schemas => constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS,
+ ObjectDepth::Tables => constants::ADBC_OBJECT_DEPTH_TABLES,
+ ObjectDepth::Columns => constants::ADBC_OBJECT_DEPTH_COLUMNS,
+ }
+ }
+}
+
+impl TryFrom<c_int> for ObjectDepth {
+ type Error = Error;
+
+ fn try_from(value: c_int) -> Result<Self, Error> {
+ match value {
+ constants::ADBC_OBJECT_DEPTH_ALL => Ok(ObjectDepth::All),
+ constants::ADBC_OBJECT_DEPTH_CATALOGS => Ok(ObjectDepth::Catalogs),
+ constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS => Ok(ObjectDepth::Schemas),
+ constants::ADBC_OBJECT_DEPTH_TABLES => Ok(ObjectDepth::Tables),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown object depth: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Database option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionDatabase {
+ /// Canonical option key for URIs.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Uri,
+ /// Canonical option key for usernames.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Username,
+ /// Canonical option key for passwords.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Password,
+ /// Driver-specific key.
+ Other(String),
+}
+
+impl AsRef<str> for OptionDatabase {
+ fn as_ref(&self) -> &str {
+ match self {
+ Self::Uri => constants::ADBC_OPTION_URI,
+ Self::Username => constants::ADBC_OPTION_USERNAME,
+ Self::Password => constants::ADBC_OPTION_PASSWORD,
+ Self::Other(key) => key,
+ }
+ }
+}
+
+impl From<&str> for OptionDatabase {
+ fn from(value: &str) -> Self {
+ match value {
+ constants::ADBC_OPTION_URI => Self::Uri,
+ constants::ADBC_OPTION_USERNAME => Self::Username,
+ constants::ADBC_OPTION_PASSWORD => Self::Password,
+ key => Self::Other(key.into()),
+ }
+ }
+}
+
+/// Connection option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionConnection {
+ /// Whether autocommit is enabled.
+ AutoCommit,
+ /// Whether the current connection should be restricted to being read-only.
+ ReadOnly,
+ /// The catalog used by the connection.
+ /// # Since
+ /// ADBC API revision 1.1.0
+ CurrentCatalog,
+ /// The database schema used by the connection.
+ /// # Since
+ /// ADBC API revision 1.1.0
+ CurrentSchema,
+ /// The isolation level of the connection. See [IsolationLevel].
+ IsolationLevel,
+ /// Driver-specific key.
+ Other(String),
+}
+
+impl AsRef<str> for OptionConnection {
+ fn as_ref(&self) -> &str {
+ match self {
+ Self::AutoCommit => constants::ADBC_CONNECTION_OPTION_AUTOCOMMIT,
+ Self::ReadOnly => constants::ADBC_CONNECTION_OPTION_READ_ONLY,
+ Self::CurrentCatalog => constants::ADBC_CONNECTION_OPTION_CURRENT_CATALOG,
+ Self::CurrentSchema => constants::ADBC_CONNECTION_OPTION_CURRENT_DB_SCHEMA,
+ Self::IsolationLevel => constants::ADBC_CONNECTION_OPTION_ISOLATION_LEVEL,
+ Self::Other(key) => key,
+ }
+ }
+}
+
+impl From<&str> for OptionConnection {
+ fn from(value: &str) -> Self {
+ match value {
+ constants::ADBC_CONNECTION_OPTION_AUTOCOMMIT => Self::AutoCommit,
+ constants::ADBC_CONNECTION_OPTION_READ_ONLY => Self::ReadOnly,
+ constants::ADBC_CONNECTION_OPTION_CURRENT_CATALOG => Self::CurrentCatalog,
+ constants::ADBC_CONNECTION_OPTION_CURRENT_DB_SCHEMA => Self::CurrentSchema,
+ constants::ADBC_CONNECTION_OPTION_ISOLATION_LEVEL => Self::IsolationLevel,
+ key => Self::Other(key.into()),
+ }
+ }
+}
+
+/// Statement option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionStatement {
+ /// The ingest mode for a bulk insert. See [IngestMode].
+ IngestMode,
+ /// The name of the target table for a bulk insert.
+ TargetTable,
+ /// Whether query execution is nonblocking. By default, execution is blocking.
+ ///
+ /// When enabled, [execute_partitions][crate::Statement::execute_partitions]
+ /// will return partitions as soon as they are available, instead of returning
+ /// them all at the end. When there are no more to return, it will return an
+ /// empty set of partitions. The methods [execute][crate::Statement::execute],
+ /// [execute_schema][crate::Statement::execute_schema] and
+ /// [execute_update][crate::Statement::execute_update] are not affected.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Incremental,
+ /// Get the progress of a query. It's a read-only option that should be
+ /// read with [get_option_double][crate::Optionable::get_option_double].
+ ///
+ /// The value is not necessarily in any particular range or have any
+ /// particular units. For example, it might be a percentage, bytes of data,
+ /// rows of data, number of workers, etc. The max value can be retrieved
+ /// via [OptionStatement::MaxProgress]. This represents the progress of
+ /// execution, not of consumption (i.e., it is independent of how much of the
+ /// result set has been read by the client).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Progress,
+ /// Get the maximum progress of a query. It's a read-only option that should be
+ /// read with [get_option_double][crate::Optionable::get_option_double].
+ ///
+ /// This is the value of [OptionStatement::Progress] for a completed query.
+ /// If not supported, or if the value is nonpositive, then the maximum is not
+ /// known. For instance, the query may be fully streaming and the driver
+ /// does not know when the result set will end.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ MaxProgress,
+ /// Driver-specific key.
+ Other(String),
+}
+
+impl AsRef<str> for OptionStatement {
+ fn as_ref(&self) -> &str {
+ match self {
+ Self::IngestMode => constants::ADBC_INGEST_OPTION_MODE,
+ Self::TargetTable => constants::ADBC_INGEST_OPTION_TARGET_TABLE,
+ Self::Incremental => constants::ADBC_STATEMENT_OPTION_INCREMENTAL,
+ Self::Progress => constants::ADBC_STATEMENT_OPTION_PROGRESS,
+ Self::MaxProgress => constants::ADBC_STATEMENT_OPTION_MAX_PROGRESS,
+ Self::Other(key) => key,
+ }
+ }
+}
+
+impl From<&str> for OptionStatement {
+ fn from(value: &str) -> Self {
+ match value {
+ constants::ADBC_INGEST_OPTION_MODE => Self::IngestMode,
+ constants::ADBC_INGEST_OPTION_TARGET_TABLE => Self::TargetTable,
+ constants::ADBC_STATEMENT_OPTION_INCREMENTAL => Self::Incremental,
+ constants::ADBC_STATEMENT_OPTION_PROGRESS => Self::Progress,
+ constants::ADBC_STATEMENT_OPTION_MAX_PROGRESS => Self::MaxProgress,
+ key => Self::Other(key.into()),
+ }
+ }
+}
+
+/// Isolation level value for key [OptionConnection::IsolationLevel].
+#[derive(Debug)]
+pub enum IsolationLevel {
+ /// Use database or driver default isolation level.
+ Default,
+ /// The lowest isolation level. Dirty reads are allowed, so one transaction
+ /// may see not-yet-committed changes made by others.
+ ReadUncommitted,
+ /// Lock-based concurrency control keeps write locks until the end of the
+ /// transaction, but read locks are released as soon as a SELECT is
+ /// performed. Non-repeatable reads can occur in this isolation level.
+ ///
+ /// More simply put, `ReadCommitted` is an isolation level that guarantees
+ /// that any data read is committed at the moment it is read. It simply
+ /// restricts the reader from seeing any intermediate, uncommitted,
+ /// 'dirty' reads. It makes no promise whatsoever that if the transaction
+ /// re-issues the read, it will find the same data; data is free to change
+ /// after it is read.
+ ReadCommitted,
+ /// Lock-based concurrency control keeps read AND write locks (acquired on
+ /// selection data) until the end of the transaction.
+ ///
+ /// However, range-locks are not managed, so phantom reads can occur.
+ /// Write skew is possible at this isolation level in some systems.
+ RepeatableRead,
+ /// This isolation guarantees that all reads in the transaction will see a
+ /// consistent snapshot of the database and the transaction should only
+ /// successfully commit if no updates conflict with any concurrent updates
+ /// made since that snapshot.
+ Snapshot,
+ /// Serializability requires read and write locks to be released only at the
+ /// end of the transaction. This includes acquiring range-locks when a
+ /// select query uses a ranged WHERE clause to avoid phantom reads.
+ Serializable,
+ /// The central distinction between serializability and linearizability is
+ /// that serializability is a global property; a property of an entire
+ /// history of operations and transactions. Linearizability is a local
+ /// property; a property of a single operation/transaction.
+ ///
+ /// Linearizability can be viewed as a special case of strict serializability
+ /// where transactions are restricted to consist of a single operation applied
+ /// to a single object.
+ Linearizable,
+}
+
+impl From<IsolationLevel> for String {
+ fn from(value: IsolationLevel) -> Self {
+ match value {
+ IsolationLevel::Default => constants::ADBC_OPTION_ISOLATION_LEVEL_DEFAULT.into(),
+ IsolationLevel::ReadUncommitted => {
+ constants::ADBC_OPTION_ISOLATION_LEVEL_READ_UNCOMMITTED.into()
+ }
+ IsolationLevel::ReadCommitted => {
+ constants::ADBC_OPTION_ISOLATION_LEVEL_READ_COMMITTED.into()
+ }
+ IsolationLevel::RepeatableRead => {
+ constants::ADBC_OPTION_ISOLATION_LEVEL_REPEATABLE_READ.into()
+ }
+ IsolationLevel::Snapshot => constants::ADBC_OPTION_ISOLATION_LEVEL_SNAPSHOT.into(),
+ IsolationLevel::Serializable => {
+ constants::ADBC_OPTION_ISOLATION_LEVEL_SERIALIZABLE.into()
+ }
+ IsolationLevel::Linearizable => {
+ constants::ADBC_OPTION_ISOLATION_LEVEL_LINEARIZABLE.into()
+ }
+ }
+ }
+}
+
+impl From<IsolationLevel> for OptionValue {
+ fn from(value: IsolationLevel) -> Self {
+ Self::String(value.into())
+ }
+}
+
+/// Ingestion mode value for key [OptionStatement::IngestMode].
+#[derive(Debug)]
+pub enum IngestMode {
Review Comment:
```suggestion
#[non_exhaustive]
#[derive(Debug)]
pub enum IngestMode {
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
Review Comment:
How about changing these to:
```suggestion
fn new_database(&mut self) -> Result<Self::DatabaseType>;
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
Review Comment:
```suggestion
fn new_connection(&mut self) -> Result<Self::ConnectionType>;
```
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
Review Comment:
What if we add explicit discriminant values:
```suggestion
#[repr(u8)]
pub enum Status {
/// No error.
Ok = 0,
/// An unknown error occurred.
Unknown = 1,
/// The operation is not implemented or supported.
NotImplemented = 2,
/// A requested resource was not found.
NotFound = 3,
/// A requested resource already exists.
AlreadyExists = 4,
/// The arguments are invalid, likely a programming error.
/// For instance, they may be of the wrong format, or out of range.
InvalidArguments = 5,
/// The preconditions for the operation are not met, likely a programming error.
/// For instance, the object may be uninitialized, or may have not
/// been fully configured.
InvalidState = 6,
/// Invalid data was processed (not a programming error).
/// For instance, a division by zero may have occurred during query
/// execution.
InvalidData = 7,
/// The database's integrity was affected.
/// For instance, a foreign key check may have failed, or a uniqueness
/// constraint may have been violated.
Integrity = 8,
/// An error internal to the driver or database occurred.
Internal = 9,
/// An I/O error occurred.
/// For instance, a remote service may be unavailable.
IO = 10,
/// The operation was cancelled, not due to a timeout.
Cancelled = 11,
/// The operation was cancelled due to a timeout.
Timeout = 12,
/// Authentication failed.
Unauthenticated = 13,
/// The client is not authorized to perform the given operation.
Unauthorized = 14,
}
```
Then we can drop the all those constants:
```rust
pub const ADBC_STATUS_OK: FFI_AdbcStatusCode = 0;
pub const ADBC_STATUS_UNKNOWN: FFI_AdbcStatusCode = 1;
pub const ADBC_STATUS_NOT_IMPLEMENTED: FFI_AdbcStatusCode = 2;
pub const ADBC_STATUS_NOT_FOUND: FFI_AdbcStatusCode = 3;
pub const ADBC_STATUS_ALREADY_EXISTS: FFI_AdbcStatusCode = 4;
pub const ADBC_STATUS_INVALID_ARGUMENT: FFI_AdbcStatusCode = 5;
pub const ADBC_STATUS_INVALID_STATE: FFI_AdbcStatusCode = 6;
pub const ADBC_STATUS_INVALID_DATA: FFI_AdbcStatusCode = 7;
pub const ADBC_STATUS_INTEGRITY: FFI_AdbcStatusCode = 8;
pub const ADBC_STATUS_INTERNAL: FFI_AdbcStatusCode = 9;
pub const ADBC_STATUS_IO: FFI_AdbcStatusCode = 10;
pub const ADBC_STATUS_CANCELLED: FFI_AdbcStatusCode = 11;
pub const ADBC_STATUS_TIMEOUT: FFI_AdbcStatusCode = 12;
pub const ADBC_STATUS_UNAUTHENTICATED: FFI_AdbcStatusCode = 13;
pub const ADBC_STATUS_UNAUTHORIZED: FFI_AdbcStatusCode = 14;
```
And for ffi we can then use a numeric cast e.g.:
```rust
assert_eq!(0, Status::Ok as u8);
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
Review Comment:
```suggestion
&mut self,
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
Review Comment:
Maybe use a product type with bools (for enable/disable) or a set?
```suggestion
codes: Option<HashSet<options::InfoCode>>,
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
How about defining a struct instead:
```rust
/// Database options.
pub struct DatabaseOptions {
/// Canonical option for URIs.
///
/// # Since
///
/// ADBC API revision 1.1.0
uri: String, // we could even use url::Url here?
/// Canonical option for usernames.
///
/// # Since
///
/// ADBC API revision 1.1.0
username: String,
/// Canonical option key for passwords.
///
/// # Since
///
/// ADBC API revision 1.1.0
password: String,
/// Driver-specific options.
others: HashMap<String, OptionValue>,
}
```
We could also define a non-fieldless enum and accept an iterator, but that allows duplicate values for the same key, so I really think a product type is better here.
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
Review Comment:
```suggestion
fn new_statement(&mut self) -> Result<Self::StatementType>;
```
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
Review Comment:
```suggestion
#[non_exhaustive]
#[derive(Clone, Copy)]
```
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
+ /// Catalogs, schemas, tables, and columns.
+ All,
+ /// Catalogs only.
+ Catalogs,
+ /// Catalogs and schemas.
+ Schemas,
+ /// Catalogs, schemas, and tables.
+ Tables,
+ /// Catalogs, schemas, tables, and columns. Identical to [ObjectDepth::All].
+ Columns,
+}
+
+impl From<ObjectDepth> for c_int {
+ fn from(value: ObjectDepth) -> Self {
+ match value {
+ ObjectDepth::All => constants::ADBC_OBJECT_DEPTH_ALL,
+ ObjectDepth::Catalogs => constants::ADBC_OBJECT_DEPTH_CATALOGS,
+ ObjectDepth::Schemas => constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS,
+ ObjectDepth::Tables => constants::ADBC_OBJECT_DEPTH_TABLES,
+ ObjectDepth::Columns => constants::ADBC_OBJECT_DEPTH_COLUMNS,
+ }
+ }
+}
+
+impl TryFrom<c_int> for ObjectDepth {
+ type Error = Error;
+
+ fn try_from(value: c_int) -> Result<Self, Error> {
+ match value {
+ constants::ADBC_OBJECT_DEPTH_ALL => Ok(ObjectDepth::All),
+ constants::ADBC_OBJECT_DEPTH_CATALOGS => Ok(ObjectDepth::Catalogs),
+ constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS => Ok(ObjectDepth::Schemas),
+ constants::ADBC_OBJECT_DEPTH_TABLES => Ok(ObjectDepth::Tables),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown object depth: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Database option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionDatabase {
Review Comment:
```suggestion
#[non_exhaustive]
#[derive(PartialEq, Eq, Hash, Debug)]
pub enum OptionDatabase {
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
Why is there a `Send` bound here? Should we maybe use an associated type so users can specify the bound only if needed?
```rust
pub trait Connection {
type GetInfoResult: RecordBatchReader;
fn get_info(&self, ...) -> Result<Self::GetInfoResult>;
}
```
```rust
struct SendConnection<C: Connection>(C) where <C as Connection>::GetInfoResult: Send
```
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
Review Comment:
```suggestion
#[non_exhaustive]
#[derive(Debug)]
#[repr(i32)]
pub enum ObjectDepth {
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
Review Comment:
Nit:
```suggestion
fn read_partition(&self, partition: impl AsRef<[u8]>) -> Result<impl RecordBatchReader + Send>;
```
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
Review Comment:
Should we consider using an associated type here?
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
Review Comment:
```suggestion
#[non_exhaustive]
#[derive(Debug)]
```
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
Review Comment:
```suggestion
#[repr(u32)]
pub enum InfoCode {
```
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
+ /// Catalogs, schemas, tables, and columns.
+ All,
+ /// Catalogs only.
+ Catalogs,
+ /// Catalogs and schemas.
+ Schemas,
+ /// Catalogs, schemas, and tables.
+ Tables,
+ /// Catalogs, schemas, tables, and columns. Identical to [ObjectDepth::All].
+ Columns,
+}
+
+impl From<ObjectDepth> for c_int {
+ fn from(value: ObjectDepth) -> Self {
+ match value {
+ ObjectDepth::All => constants::ADBC_OBJECT_DEPTH_ALL,
+ ObjectDepth::Catalogs => constants::ADBC_OBJECT_DEPTH_CATALOGS,
+ ObjectDepth::Schemas => constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS,
+ ObjectDepth::Tables => constants::ADBC_OBJECT_DEPTH_TABLES,
+ ObjectDepth::Columns => constants::ADBC_OBJECT_DEPTH_COLUMNS,
+ }
+ }
+}
+
+impl TryFrom<c_int> for ObjectDepth {
+ type Error = Error;
+
+ fn try_from(value: c_int) -> Result<Self, Error> {
+ match value {
+ constants::ADBC_OBJECT_DEPTH_ALL => Ok(ObjectDepth::All),
+ constants::ADBC_OBJECT_DEPTH_CATALOGS => Ok(ObjectDepth::Catalogs),
+ constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS => Ok(ObjectDepth::Schemas),
+ constants::ADBC_OBJECT_DEPTH_TABLES => Ok(ObjectDepth::Tables),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown object depth: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Database option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionDatabase {
+ /// Canonical option key for URIs.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Uri,
+ /// Canonical option key for usernames.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Username,
+ /// Canonical option key for passwords.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Password,
+ /// Driver-specific key.
+ Other(String),
+}
+
+impl AsRef<str> for OptionDatabase {
+ fn as_ref(&self) -> &str {
+ match self {
+ Self::Uri => constants::ADBC_OPTION_URI,
+ Self::Username => constants::ADBC_OPTION_USERNAME,
+ Self::Password => constants::ADBC_OPTION_PASSWORD,
+ Self::Other(key) => key,
+ }
+ }
+}
+
+impl From<&str> for OptionDatabase {
+ fn from(value: &str) -> Self {
+ match value {
+ constants::ADBC_OPTION_URI => Self::Uri,
+ constants::ADBC_OPTION_USERNAME => Self::Username,
+ constants::ADBC_OPTION_PASSWORD => Self::Password,
+ key => Self::Other(key.into()),
+ }
+ }
+}
+
+/// Connection option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionConnection {
+ /// Whether autocommit is enabled.
+ AutoCommit,
+ /// Whether the current connection should be restricted to being read-only.
+ ReadOnly,
+ /// The catalog used by the connection.
+ /// # Since
+ /// ADBC API revision 1.1.0
+ CurrentCatalog,
+ /// The database schema used by the connection.
+ /// # Since
+ /// ADBC API revision 1.1.0
+ CurrentSchema,
+ /// The isolation level of the connection. See [IsolationLevel].
+ IsolationLevel,
+ /// Driver-specific key.
+ Other(String),
+}
+
+impl AsRef<str> for OptionConnection {
+ fn as_ref(&self) -> &str {
+ match self {
+ Self::AutoCommit => constants::ADBC_CONNECTION_OPTION_AUTOCOMMIT,
+ Self::ReadOnly => constants::ADBC_CONNECTION_OPTION_READ_ONLY,
+ Self::CurrentCatalog => constants::ADBC_CONNECTION_OPTION_CURRENT_CATALOG,
+ Self::CurrentSchema => constants::ADBC_CONNECTION_OPTION_CURRENT_DB_SCHEMA,
+ Self::IsolationLevel => constants::ADBC_CONNECTION_OPTION_ISOLATION_LEVEL,
+ Self::Other(key) => key,
+ }
+ }
+}
+
+impl From<&str> for OptionConnection {
+ fn from(value: &str) -> Self {
+ match value {
+ constants::ADBC_CONNECTION_OPTION_AUTOCOMMIT => Self::AutoCommit,
+ constants::ADBC_CONNECTION_OPTION_READ_ONLY => Self::ReadOnly,
+ constants::ADBC_CONNECTION_OPTION_CURRENT_CATALOG => Self::CurrentCatalog,
+ constants::ADBC_CONNECTION_OPTION_CURRENT_DB_SCHEMA => Self::CurrentSchema,
+ constants::ADBC_CONNECTION_OPTION_ISOLATION_LEVEL => Self::IsolationLevel,
+ key => Self::Other(key.into()),
+ }
+ }
+}
+
+/// Statement option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionStatement {
+ /// The ingest mode for a bulk insert. See [IngestMode].
+ IngestMode,
+ /// The name of the target table for a bulk insert.
+ TargetTable,
+ /// Whether query execution is nonblocking. By default, execution is blocking.
+ ///
+ /// When enabled, [execute_partitions][crate::Statement::execute_partitions]
+ /// will return partitions as soon as they are available, instead of returning
+ /// them all at the end. When there are no more to return, it will return an
+ /// empty set of partitions. The methods [execute][crate::Statement::execute],
+ /// [execute_schema][crate::Statement::execute_schema] and
+ /// [execute_update][crate::Statement::execute_update] are not affected.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Incremental,
+ /// Get the progress of a query. It's a read-only option that should be
+ /// read with [get_option_double][crate::Optionable::get_option_double].
+ ///
+ /// The value is not necessarily in any particular range or have any
+ /// particular units. For example, it might be a percentage, bytes of data,
+ /// rows of data, number of workers, etc. The max value can be retrieved
+ /// via [OptionStatement::MaxProgress]. This represents the progress of
+ /// execution, not of consumption (i.e., it is independent of how much of the
+ /// result set has been read by the client).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Progress,
+ /// Get the maximum progress of a query. It's a read-only option that should be
+ /// read with [get_option_double][crate::Optionable::get_option_double].
+ ///
+ /// This is the value of [OptionStatement::Progress] for a completed query.
+ /// If not supported, or if the value is nonpositive, then the maximum is not
+ /// known. For instance, the query may be fully streaming and the driver
+ /// does not know when the result set will end.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ MaxProgress,
+ /// Driver-specific key.
+ Other(String),
+}
+
+impl AsRef<str> for OptionStatement {
+ fn as_ref(&self) -> &str {
+ match self {
+ Self::IngestMode => constants::ADBC_INGEST_OPTION_MODE,
+ Self::TargetTable => constants::ADBC_INGEST_OPTION_TARGET_TABLE,
+ Self::Incremental => constants::ADBC_STATEMENT_OPTION_INCREMENTAL,
+ Self::Progress => constants::ADBC_STATEMENT_OPTION_PROGRESS,
+ Self::MaxProgress => constants::ADBC_STATEMENT_OPTION_MAX_PROGRESS,
+ Self::Other(key) => key,
+ }
+ }
+}
+
+impl From<&str> for OptionStatement {
+ fn from(value: &str) -> Self {
+ match value {
+ constants::ADBC_INGEST_OPTION_MODE => Self::IngestMode,
+ constants::ADBC_INGEST_OPTION_TARGET_TABLE => Self::TargetTable,
+ constants::ADBC_STATEMENT_OPTION_INCREMENTAL => Self::Incremental,
+ constants::ADBC_STATEMENT_OPTION_PROGRESS => Self::Progress,
+ constants::ADBC_STATEMENT_OPTION_MAX_PROGRESS => Self::MaxProgress,
+ key => Self::Other(key.into()),
+ }
+ }
+}
+
+/// Isolation level value for key [OptionConnection::IsolationLevel].
+#[derive(Debug)]
+pub enum IsolationLevel {
Review Comment:
```suggestion
#[non_exhaustive]
#[derive(Debug)]
pub enum IsolationLevel {
```
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
+ /// Catalogs, schemas, tables, and columns.
+ All,
+ /// Catalogs only.
+ Catalogs,
+ /// Catalogs and schemas.
+ Schemas,
+ /// Catalogs, schemas, and tables.
+ Tables,
+ /// Catalogs, schemas, tables, and columns. Identical to [ObjectDepth::All].
+ Columns,
+}
+
+impl From<ObjectDepth> for c_int {
+ fn from(value: ObjectDepth) -> Self {
+ match value {
+ ObjectDepth::All => constants::ADBC_OBJECT_DEPTH_ALL,
+ ObjectDepth::Catalogs => constants::ADBC_OBJECT_DEPTH_CATALOGS,
+ ObjectDepth::Schemas => constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS,
+ ObjectDepth::Tables => constants::ADBC_OBJECT_DEPTH_TABLES,
+ ObjectDepth::Columns => constants::ADBC_OBJECT_DEPTH_COLUMNS,
+ }
+ }
+}
+
+impl TryFrom<c_int> for ObjectDepth {
+ type Error = Error;
+
+ fn try_from(value: c_int) -> Result<Self, Error> {
+ match value {
+ constants::ADBC_OBJECT_DEPTH_ALL => Ok(ObjectDepth::All),
+ constants::ADBC_OBJECT_DEPTH_CATALOGS => Ok(ObjectDepth::Catalogs),
+ constants::ADBC_OBJECT_DEPTH_DB_SCHEMAS => Ok(ObjectDepth::Schemas),
+ constants::ADBC_OBJECT_DEPTH_TABLES => Ok(ObjectDepth::Tables),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown object depth: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Database option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionDatabase {
+ /// Canonical option key for URIs.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Uri,
+ /// Canonical option key for usernames.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Username,
+ /// Canonical option key for passwords.
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ Password,
+ /// Driver-specific key.
+ Other(String),
+}
+
+impl AsRef<str> for OptionDatabase {
+ fn as_ref(&self) -> &str {
+ match self {
+ Self::Uri => constants::ADBC_OPTION_URI,
+ Self::Username => constants::ADBC_OPTION_USERNAME,
+ Self::Password => constants::ADBC_OPTION_PASSWORD,
+ Self::Other(key) => key,
+ }
+ }
+}
+
+impl From<&str> for OptionDatabase {
+ fn from(value: &str) -> Self {
+ match value {
+ constants::ADBC_OPTION_URI => Self::Uri,
+ constants::ADBC_OPTION_USERNAME => Self::Username,
+ constants::ADBC_OPTION_PASSWORD => Self::Password,
+ key => Self::Other(key.into()),
+ }
+ }
+}
+
+/// Connection option key.
+#[derive(PartialEq, Eq, Hash, Debug)]
+pub enum OptionConnection {
Review Comment:
```suggestion
#[non_exhaustive]
#[derive(PartialEq, Eq, Hash, Debug)]
pub enum OptionConnection {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567904488
##########
rust2/Cargo.toml:
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[workspace]
+members = ["core", "drivers/*"]
+resolver = "2"
+
+[workspace.package]
+version = "0.1.0"
+description = "Rust implementation of Arrow Database Connectivity (ADBC)"
+edition = "2021"
+authors = ["Apache Arrow <de...@arrow.apache.org>"]
+license = "Apache-2.0"
+
+[workspace.dependencies]
+arrow = { version = "51.0.0", default-features = false, features = ["ffi"] }
Review Comment:
According to my analysis, we have to depend on `arrow` because the following items are not defined in a sub-crate:
- `arrow::ffi_stream::FFI_ArrowArrayStream`
- `arrow::ffi_stream::ArrowArrayStreamReader`
- `arrow::ffi::from_ffi`
- `arrow::ffi::to_ffi`
Maybe we can open a PR in arrow-rs to fix that (I don't know if this is easy or not)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568455458
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
Review Comment:
I would rather avoid making FFI so central, we should use native Rust conventions where possible. (Of course it is easier to structure things in a way that maps cleanly but we shouldn't force everything to be 1:1, it defeats the point.)
FWIW the Go definitions mostly do not make concessions like this to FFI considerations but FFI works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1570746494
##########
rust2/drivers/dummy/src/lib.rs:
##########
@@ -0,0 +1,890 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+use std::{collections::HashMap, fmt::Debug, hash::Hash};
+
+use arrow::array::{
+ Array, ArrayRef, BinaryArray, BooleanArray, Float64Array, Int16Array, Int32Array, Int64Array,
+ ListArray, MapArray, StringArray, StructArray, UInt32Array, UInt64Array, UnionArray,
+};
+use arrow::buffer::{Buffer, OffsetBuffer, ScalarBuffer};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use adbc_core::{
+ error::{Error, Result, Status},
+ ffi::constants,
+ options::{
+ InfoCode, ObjectDepth, OptionConnection, OptionDatabase, OptionStatement, OptionValue,
+ },
+ schemas, Connection, Database, Driver, Optionable, PartitionedResult, Statement,
+};
+
+#[derive(Debug)]
+pub struct SingleBatchReader {
+ batch: Option<RecordBatch>,
+ schema: SchemaRef,
+}
+
+impl SingleBatchReader {
+ pub fn new(batch: RecordBatch) -> Self {
+ let schema = batch.schema();
+ Self {
+ batch: Some(batch),
+ schema,
+ }
+ }
+}
+
+impl Iterator for SingleBatchReader {
+ type Item = std::result::Result<RecordBatch, ArrowError>;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ Ok(self.batch.take()).transpose()
+ }
+}
+
+impl RecordBatchReader for SingleBatchReader {
+ fn schema(&self) -> SchemaRef {
+ self.schema.clone()
+ }
+}
+
+fn get_table_schema() -> Schema {
+ Schema::new(vec![
+ Field::new("a", DataType::UInt32, true),
+ Field::new("b", DataType::Float64, false),
+ Field::new("c", DataType::Utf8, true),
+ ])
+}
+
+fn get_table_data() -> RecordBatch {
+ RecordBatch::try_new(
+ Arc::new(get_table_schema()),
+ vec![
+ Arc::new(UInt32Array::from(vec![1, 2, 3])),
+ Arc::new(Float64Array::from(vec![1.5, 2.5, 3.5])),
+ Arc::new(StringArray::from(vec!["A", "B", "C"])),
+ ],
+ )
+ .unwrap()
+}
+
+fn set_option<T>(options: &mut HashMap<T, OptionValue>, key: T, value: OptionValue) -> Result<()>
+where
+ T: Eq + Hash,
+{
+ options.insert(key, value);
+ Ok(())
+}
+
+fn get_option_bytes<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<Vec<u8>>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Bytes(value) => Ok(value.clone()),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_double<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<f64>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Double(value) => Ok(*value),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_int<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<i64>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Int(value) => Ok(*value),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_string<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<String>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::String(value) => Ok(value.clone()),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+/// A dummy driver used for testing purposes.
+#[derive(Default)]
+pub struct DummyDriver {}
+
+impl Driver for DummyDriver {
+ type DatabaseType = DummyDatabase;
+
+ fn new_database(&mut self) -> Result<Self::DatabaseType> {
+ self.new_database_with_opts([].into_iter())
+ }
+
+ fn new_database_with_opts(
+ &mut self,
+ opts: impl Iterator<Item = (<Self::DatabaseType as Optionable>::Option, OptionValue)>,
+ ) -> Result<Self::DatabaseType> {
+ let mut database = Self::DatabaseType {
+ options: HashMap::new(),
+ };
+ for (key, value) in opts {
+ database.set_option(key, value)?;
+ }
+ Ok(database)
+ }
+}
+
+pub struct DummyDatabase {
+ options: HashMap<OptionDatabase, OptionValue>,
+}
+
+impl Optionable for DummyDatabase {
+ type Option = OptionDatabase;
+
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()> {
+ set_option(&mut self.options, key, value)
+ }
+
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>> {
+ get_option_bytes(&self.options, key, "database")
+ }
+
+ fn get_option_double(&self, key: Self::Option) -> Result<f64> {
+ get_option_double(&self.options, key, "database")
+ }
+
+ fn get_option_int(&self, key: Self::Option) -> Result<i64> {
+ get_option_int(&self.options, key, "database")
+ }
+
+ fn get_option_string(&self, key: Self::Option) -> Result<String> {
+ get_option_string(&self.options, key, "database")
+ }
+}
+
+impl Database for DummyDatabase {
+ type ConnectionType = DummyConnection;
+
+ fn new_connection(&mut self) -> Result<Self::ConnectionType> {
+ self.new_connection_with_opts([].into_iter())
+ }
+
+ fn new_connection_with_opts(
+ &mut self,
+ opts: impl Iterator<Item = (<Self::ConnectionType as Optionable>::Option, OptionValue)>,
+ ) -> Result<Self::ConnectionType> {
+ let mut connection = Self::ConnectionType {
+ options: HashMap::new(),
+ };
+ for (key, value) in opts {
+ connection.set_option(key, value)?;
+ }
+ Ok(connection)
+ }
+}
+
+pub struct DummyConnection {
+ options: HashMap<OptionConnection, OptionValue>,
+}
+
+impl Optionable for DummyConnection {
+ type Option = OptionConnection;
+
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()> {
+ set_option(&mut self.options, key, value)
+ }
+
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>> {
+ get_option_bytes(&self.options, key, "connection")
+ }
+
+ fn get_option_double(&self, key: Self::Option) -> Result<f64> {
+ get_option_double(&self.options, key, "connection")
+ }
+
+ fn get_option_int(&self, key: Self::Option) -> Result<i64> {
+ get_option_int(&self.options, key, "connection")
+ }
+
+ fn get_option_string(&self, key: Self::Option) -> Result<String> {
+ get_option_string(&self.options, key, "connection")
+ }
+}
+
+impl Connection for DummyConnection {
+ type StatementType = DummyStatement;
+
+ fn new_statement(&mut self) -> Result<Self::StatementType> {
+ Ok(Self::StatementType {
+ options: HashMap::new(),
+ })
+ }
+
+ // This method is used to test that errors round-trip correctly.
+ fn cancel(&mut self) -> Result<()> {
+ let mut error = Error::with_message_and_status("message", Status::Cancelled);
+ error.vendor_code = constants::ADBC_ERROR_VENDOR_CODE_PRIVATE_DATA;
+ error.sqlstate = [1, 2, 3, 4, 5];
+ error.details = Some(vec![
+ ("key1".into(), b"AAA".into()),
+ ("key2".into(), b"ZZZZZ".into()),
+ ]);
+ Err(error)
+ }
+
+ fn commit(&mut self) -> Result<()> {
+ Ok(())
+ }
+
+ fn get_info(&self, _codes: Option<HashSet<InfoCode>>) -> Result<impl RecordBatchReader> {
Review Comment:
I could try to artifically use the codes but it won't provide much value. If we want to usefully use them it will take a fair amount of work I think.
The goal here is just to test that data round-trip correctly between Rust -> C -> Rust (testing both the driver exporter & manager).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568395445
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
Review Comment:
I've chosen to not rely on integer representation of enums (and instead write explicit TryFrom/From) because of this:
```rust
use num_enum::{FromPrimitive, IntoPrimitive};
#[repr(u32)]
#[derive(FromPrimitive, IntoPrimitive)]
pub enum InfoCode {
VendorName = 0,
VendorVersion = 1,
VendorArrowVersion = 2,
DriverName = 100,
DriverVersion = 101,
DriverArrowVersion = 102,
#[num_enum(catch_all)]
Other(u32),
}
fn main() {
println!("size = {}", std::mem::size_of::<InfoCode>());
// size = 8
// not u32!
}
```
This snippet of code is present is the current version (the one in `/rust`). When crossing FFI boundaries by passing for instance an `&[InfoCode]` it introduces a memory corruption...
So, I we stay away from `#[num_enum(catch_all)]` I'm OK to rely on `#[repr(integer)]` instead of defining constants and conversion traits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "aljazerzen (via GitHub)" <gi...@apache.org>.
aljazerzen commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568706748
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,500 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+#[non_exhaustive]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
Review Comment:
```suggestion
Float(f64),
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568866777
##########
.github/workflows/rust.yml:
##########
@@ -38,7 +38,7 @@ permissions:
defaults:
run:
- working-directory: rust
+ working-directory: rust2
Review Comment:
I can help figure out CI in a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "aljazerzen (via GitHub)" <gi...@apache.org>.
aljazerzen commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567526774
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
Review Comment:
Maybe return `None` if affected row count is unknown or unsupported?
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
I'm not familiar with ADBC C api, but aren't these two required? And calling any other function before calling either of these two results in an error?
I would expect these two to be `fn new_sql(...) -> Self` and `fn new_substrait(...) -> Self`, but somehow associated with `Connection`. Maybe something like this:
```rust
enum Query<'a> {
SQL(&'a str),
Substrait(&'a [u8]),
}
trait Connection {
...
fn new_statement(&self, query: Query) -> Result<Self::StatementType>;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568826885
##########
.github/workflows/rust.yml:
##########
@@ -38,7 +38,7 @@ permissions:
defaults:
run:
- working-directory: rust
+ working-directory: rust2
Review Comment:
It looks like my suggestion didn't work here (e.g., I don't see the Rust workflow run at the end). There might be some constraint on newer contributors modifying these files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569070430
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
> I will put more thougth into it, but I've tried different designs before settling on this one. While not perfect, it's relatively simple and can accomodate the C API...
Sounds good, it's probably easier to discuss API designs here once we have some real implementations.
> Also, re: the original proposal, not all drivers necessarily have or require uri, username, or password fields; the latter two would never apply to SQLite, for instance, and SQLite doesn't require a URI
If you are referring to what I suggested, then yes, the field types should all be wrapped in `Option` as suggested by @alexandreyc:
> (we could wrap all the fields in Option though).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1986813316
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
> If I try to Box the impl RecordBatchReader returned by adbc_core::Connection methods, rustc complains that the boxed value may outlive the references I'm passing into the calls.
Can you give an example? I don't follow how making the method generic over a lifetime `'a`, without using that lifetime for any of the arguments helps here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1986204168
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
This thread was mentioned in the source code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1982104007
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
ADBC connection should probably be thread-confined since ordering of statements executed on them matters.
And considering that a batch reader might be sharing resources with the connection that created it, this annotation should be:
```rust
fn get_info<'a>(
&'a self,
codes: Option<Vec<options::InfoCode>>,
) -> Result<impl RecordBatchReader + 'a>;
```
What you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#issuecomment-2059229798
Thanks for taking time to review!
I've added the licence header to all files. The CI is now green.
I'm not very familiar with Github Actions so I need a bit more time to read doc before integrating the new folder to CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568791830
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
> Although it does that by setting an option on the connection
Hmm that's not what I read. The option is set on the statement not the connection. See here for instance: https://arrow.apache.org/adbc/current/format/specification.html#bulk-data-ingestion
> I would prefer something more explicit, similar to the Java API
After looking at the Java API I would say that it's very similar. The `AdbcStatement` interface has `setSqlQuery` and `setSubstraitPlan` and the `AdbcConnection` interface allows creating "empty" statements via `createStatement` (equivalent to our `new_statement`).
But you're right that the Java API is a bit more user-friendly for the ingestion use case because the `AdbcConnection` interface has an additonal method to create a pre-configured statement for ingestion via the `bulkIngest` method.
I propose to add a `new_ingest_statement(target_table, ingest_mode) -> Statement` to the `Connection` trait (this method could probably have a default implementation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567958000
##########
rust2/Cargo.toml:
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[workspace]
+members = ["core", "drivers/*"]
+resolver = "2"
+
+[workspace.package]
+version = "0.1.0"
+description = "Rust implementation of Arrow Database Connectivity (ADBC)"
+edition = "2021"
+authors = ["Apache Arrow <de...@arrow.apache.org>"]
+license = "Apache-2.0"
+
+[workspace.dependencies]
+arrow = { version = "51.0.0", default-features = false, features = ["ffi"] }
Review Comment:
Ah I missed that. I'll try to create a PR to introduce an `arrow-ffi` crate. For now we can just keep using `arrow` here then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568791830
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
> Although it does that by setting an option on the connection
Hmm that's not what I read. The option is set on the statement not the connection. See here for instance: https://arrow.apache.org/adbc/current/format/specification.html#bulk-data-ingestion
> I would prefer something more explicit, similar to the Java API
After looking at the Java API I would say that it's very similar. The `AdbcStatement` interface has `setSqlQuery` and `setSubstraitPlan` and the `AdbcConnection` interface allows creating "empty" statements via `createStatement` (equivalent to our `new_statement`).
But you're right that the Java API is a bit more user-friendly for the ingestion use case because the `AdbcConnection` interface has an additonal method to create a pre-configured statement for ingestion via the `bulkIngest` method.
I propose to add a `new_ingest_statement(target_table, ingest_mode) -> Statement` to the `Connection` trait
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "aljazerzen (via GitHub)" <gi...@apache.org>.
aljazerzen commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568737932
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
You are right, C API supports not setting sql query or substrait plan. Although it does that by setting an option on the connection:
https://github.com/apache/arrow-adbc/blob/main/adbc.h#L735-L755
I don't think this is a good design, I would prefer something more explicit, similar to the Java API: https://arrow.apache.org/adbc/current/java/api/org/apache/arrow/adbc/core/AdbcConnection.html#bulkIngest(java.lang.String,org.apache.arrow.adbc.core.BulkIngestMode)
```rust
enum Operation<'a> {
QuerySQL(&'a str),
QuerySubstrait(&'a [u8]),
BulkIngest { target_table_name: &'a str },
}
trait Connection {
...
fn new_statement(&self, op: Operation) -> Result<Self::StatementType>;
}
```
I can also forgo 2, creating a new statement should not be expensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1570211715
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
This is hypothetical but here is my intuition at the moment:
- The low-level API can accomodate all the semantics of the C API
- If your driver implement this low-level API, you can automatically (thanks to the driver exporter) build it as an object file (static or dynamic) that other languages can use (via their own driver manager).
- The high-level API is built on the low-level API and is more idiomatic and less error-prone BUT it cannot be shared with other languages (it might possible to automatically implement this high-level API from the low-level API, ala driver exporter) and probably cannot perform all operations made possible by the C API.
From a driver implementor perspective, you should first implement the low-level API, so that you can share your driver with others, which is kind of the point of ADBC. If it's possible to automatically implement the high-level API then this is a win-win. If you decide to only implement the high-level API, only Rust users will be able to use your driver.
From a driver consumer perspective, you should almost always use the high-level API since it's more ergonomic (except if you need some semantics that's not availabe in the high-level API).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568861423
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
Review Comment:
+1 for a TODO for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "aljazerzen (via GitHub)" <gi...@apache.org>.
aljazerzen commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568700678
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
Review Comment:
That would mean that an implementation of `Statement` will have to define which impl of `RecordBatchReader` it will use here.
I would rather suggest making bind_stream generic with `R: RecordBatchReader + Send`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567840458
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
Review Comment:
Ah, I didn't consider the ffi use-case here. Maybe we can use `u8` here and convert to `c_char` only for ffi? (Because to support ffi directly here I think `message` should be `CString` I think? - and what about `details`?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567845939
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
Review Comment:
In the full version there is code for converting between the two yes. You can see it here:
- https://github.com/alexandreyc/adbc-rs/blob/main/core/src/ffi/types.rs#L488
- https://github.com/alexandreyc/adbc-rs/blob/main/core/src/ffi/types.rs#L545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568710105
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
> We could also define a non-fieldless enum and accept an iterator, but that allows duplicate values for the same key, so I really think a product type is better here.
That's what I did in a first version but it wasn't a great fit because we also need to query option values with `get_option_*` and in that case we only need the option key and not the value (that would force us to provide a sentinel in the non-fieldless enum variant).
Regarding the product type, IMO it's not a great fit either:
1. Databases not necessarily need all options. For instance, SQLite only needs `uri`, it even does not recognize `username` nor `password` (we could wrap all the fields in `Option` though).
2. Everytime a new "standard" key is added we'll have to add a new field to this struct.
3. It's harder for the user to split between pre-init and post-init options, because in that case he would need to define two instances of the struct with non-overlapping non-`None` fields.
The only downside I see with the current solution is the possibility to provide duplicated keys... We could document that "last write win" strategy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568835401
##########
.github/workflows/rust.yml:
##########
@@ -38,7 +38,7 @@ permissions:
defaults:
run:
- working-directory: rust
+ working-directory: rust2
Review Comment:
in fact this action has _never_ worked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#issuecomment-2072037773
> @mbrobbel If you have time, I think that you can work on the CI in parallel (there shouldn't be any conflict)
#1749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#issuecomment-2063892369
Should be ready to merge now!
Next PR: full FFI bindings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1986204314
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
That's for reference. Please open a new issue if you want to discuss a change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#issuecomment-2065488929
I'll merge this in a day or two if there's no further comments. Thanks @alexandreyc!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568477669
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
Review Comment:
Is this for avoiding querying the same info code multiple times?
I'm OK with using an `HashSet` but I'm not convinced a product type is the right fit here because it would impose to update it every time a variant is added to `InfoCode` (or maybe I mistunderstand your point).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568810336
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
Review Comment:
Yes, we can make this function generic but the bound would need to be `R: RecordBatchReader + Send + 'static`.
To fully grasp the reason behind this, we need to consider the driver manager, in particular this line: https://github.com/alexandreyc/adbc-rs/blob/24990185cc5ab37b35ce2e8157fbb16f5608c19d/core/src/driver_manager.rs#L973
The reason I choose this design is because `FFI_ArrowArrayStream::new` takes an `Box<dyn RecordBatchReader + Send>`. We can switch to a generic but we need to box it before giving it to `FFI_ArrowArrayStream::new` and in that case the compiler complains that the generic does not live long enough so the additional `'static` bound.
I'm open to reconsider this (and https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567748242) but I think you should first read the code for the driver manager and exporter. So maybe I could add a TODO and rediscuss it in follow-up PRs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568799063
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
I'm not sure to understand what you mean. Can you elaborate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567611640
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
Overall I agree with you that we need to find a good balance between being idiomatic and close to the C API. That means that trade-offs will need to be made.
1. According to my understanding of the C API, you can use a statement without ever calling `StatementSetSqlQuery` or `StatementSetSubstraitPlan`. For instance, I you want to ingest some data you will set some options on the statement, bind some data with `StatementBind` and finally call `StatementExecuteQuery` (even though there is no query per se).
2. If we go this route, we cannot reuse a statement with a different query (it might not be a big deal since we can always create a fresh statement).
I think we can live with 2, but 1 seems a bit harder. We could pass an `Option<Query>` for the specific case of data ingestion... Maybe there exist some other cases that won't fit your proposed change, I need to think more about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1570211715
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
This is hypothetical but here is my intuition at the moment:
- The low-level API can accomodate all the semantics of the C API
- If your driver implement this low-level API, you can automatically (thanks to the driver exporter) build it as an object file (static or dynamic) that other languages can use (via their own driver manager).
- The high-level API is built on the low-level API and is more idiomatic and less error-prone BUT it cannot be shared with other languages (it might possible to automatically implement this high-level API from the low-level API, ala driver exporter) and probably cannot perform all operations made possible by the C API.
From a driver implementor perspective you should first implement the low-level API, so that you can share your driver with others, which is kind of the point of ADBC. If it's possible to automatically implement the high-level API then this is a win-win. If you decide to only implement the high-level API, only Rust users will be able to use your driver.
From a driver consumer perspective you should almost always use the high-level API since it's more ergonomic (except if you need some semantics that's not availabe in the high-level API).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567611640
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
Overall I agree with you that we need to find a good balance between being idiomatic and close to the C API. That also means trade-offs will need to be made.
1. According to my understanding of the C API, you can use a statement without ever calling `StatementSetSqlQuery` or `StatementSetSubstraitPlan`. For instance, if you want to ingest some data you will set some options on the statement, bind some data with `StatementBind` and finally call `StatementExecuteQuery` (even though there is no query per se).
2. If we go this route, we cannot reuse a statement with a different query (it might not be a big deal since we can always create a fresh statement).
I think we can live with 2, but 1 seems a bit harder. We could pass an `Option<Query>` for the specific case of data ingestion... Maybe there exist some other cases that won't fit your proposed change, I need to think more about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1986197628
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
Please open a new issue to discuss this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#issuecomment-2068331318
Augh, I forgot to edit out the pings from the commit message...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm merged PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1571037221
##########
rust2/Cargo.toml:
##########
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[workspace]
+members = ["core", "drivers/*"]
+resolver = "2"
+
+[workspace.package]
+version = "0.1.0"
+description = "Rust implementation of Arrow Database Connectivity (ADBC)"
+edition = "2021"
+authors = ["Apache Arrow <de...@arrow.apache.org>"]
+license = "Apache-2.0"
+
+[workspace.dependencies]
+arrow = { version = "51.0.0", default-features = false, features = ["ffi"] }
Review Comment:
I've opened an issue upstream to ask for ideas: https://github.com/apache/arrow-rs/issues/5663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567840868
##########
rust2/core/src/ffi/constants.rs:
##########
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::os::raw::c_int;
+
+use super::types::FFI_AdbcStatusCode;
+
+pub const ADBC_STATUS_OK: FFI_AdbcStatusCode = 0;
+pub const ADBC_STATUS_UNKNOWN: FFI_AdbcStatusCode = 1;
+pub const ADBC_STATUS_NOT_IMPLEMENTED: FFI_AdbcStatusCode = 2;
+pub const ADBC_STATUS_NOT_FOUND: FFI_AdbcStatusCode = 3;
+pub const ADBC_STATUS_ALREADY_EXISTS: FFI_AdbcStatusCode = 4;
+pub const ADBC_STATUS_INVALID_ARGUMENT: FFI_AdbcStatusCode = 5;
+pub const ADBC_STATUS_INVALID_STATE: FFI_AdbcStatusCode = 6;
+pub const ADBC_STATUS_INVALID_DATA: FFI_AdbcStatusCode = 7;
+pub const ADBC_STATUS_INTEGRITY: FFI_AdbcStatusCode = 8;
+pub const ADBC_STATUS_INTERNAL: FFI_AdbcStatusCode = 9;
+pub const ADBC_STATUS_IO: FFI_AdbcStatusCode = 10;
+pub const ADBC_STATUS_CANCELLED: FFI_AdbcStatusCode = 11;
+pub const ADBC_STATUS_TIMEOUT: FFI_AdbcStatusCode = 12;
+pub const ADBC_STATUS_UNAUTHENTICATED: FFI_AdbcStatusCode = 13;
+pub const ADBC_STATUS_UNAUTHORIZED: FFI_AdbcStatusCode = 14;
+
+pub const ADBC_VERSION_1_0_0: i32 = 1000000;
+pub const ADBC_VERSION_1_1_0: i32 = 1001000;
Review Comment:
In `adbc.h`, `AdbcDriverInitFunc` takes `version` as `int`. Maybe we should use `c_int` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "aljazerzen (via GitHub)" <gi...@apache.org>.
aljazerzen commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569527028
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
> In that case, this API must only be used by consumer of Rust driver (driver implementors must implement the low-level API to be exportable to C).
I don't understand what you mean here.
Are you saying that implementations of ADBC drivers in Rust will implement this (low-level) API, so it can be accessible via ffi by C code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568529728
##########
rust2/core/Cargo.toml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "adbc_core"
+description = "Public abstract API, driver manager and driver exporter"
+version = { workspace = true }
+edition = { workspace = true }
+authors = { workspace = true }
+license = { workspace = true }
+
+[dependencies]
+arrow = { workspace = true }
+libloading = "0.8"
Review Comment:
Your suggestion sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568783103
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
And what about adding an associated type for options to the `Database` trait - with a supertrait for the "standard" keys (that return `Option`s and default to `None`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568810336
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
Review Comment:
Yes, we can make this function generic but the bound would need to be `R: RecordBatchReader + Send + 'static`.
To fully grasp the reason behind this, we need to consider the driver manager, in particular this line: https://github.com/alexandreyc/adbc-rs/blob/24990185cc5ab37b35ce2e8157fbb16f5608c19d/core/src/driver_manager.rs#L973
The reason I choose this design is because `FFI_ArrowArrayStream::new` takes an `Box<dyn RecordBatchReader + Send>`. We can switch to a generic but we need to box it before giving it to `FFI_ArrowArrayStream::new` and in that case the compiler complains that the generic does not live long enough.
I'm open to reconsider this (and https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567748242) but I think you should first read the code for the driver manager and exporter. So maybe I could add a TODO and rediscuss it in follow-up PRs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568835003
##########
.github/workflows/rust.yml:
##########
@@ -38,7 +38,7 @@ permissions:
defaults:
run:
- working-directory: rust
+ working-directory: rust2
Review Comment:
https://github.com/apache/arrow-adbc/actions/runs/8719547999
we can't use arbitrary actions. can we stick with whatever apache/arrow-rs uses?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568322398
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
Review Comment:
I switched to `c_char` for the moment and added a TODO to revisit it later. Currently `sqlstate` is only used when printing the error, so I guess it's not a big deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569030653
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
Review Comment:
Agreed. If we push this logic further, we should also update to `&mut self` the following methods:
- `Connection::commit`
- `Connection::rollback`
- All the `Statement` methods except `get_parameters_schema`
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567610718
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
Review Comment:
Yes! Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567611640
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
Overall I agree with you that we need to find a good balance between being idiomatic and close to the C API. That also means trade-offs will need to be made.
1. According to my understanding of the C API, you can use a statement without ever calling `StatementSetSqlQuery` or `StatementSetSubstraitPlan`. For instance, I you want to ingest some data you will set some options on the statement, bind some data with `StatementBind` and finally call `StatementExecuteQuery` (even though there is no query per se).
2. If we go this route, we cannot reuse a statement with a different query (it might not be a big deal since we can always create a fresh statement).
I think we can live with 2, but 1 seems a bit harder. We could pass an `Option<Query>` for the specific case of data ingestion... Maybe there exist some other cases that won't fit your proposed change, I need to think more about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568451821
##########
rust2/core/src/options.rs:
##########
@@ -0,0 +1,492 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Various option and configuration types.
+
+use std::os::raw::c_int;
+
+use crate::{
+ error::{Error, Status},
+ ffi::constants,
+};
+
+/// Option value.
+///
+/// Can be created with various implementations of [From].
+#[derive(Debug)]
+pub enum OptionValue {
+ String(String),
+ Bytes(Vec<u8>),
+ Int(i64),
+ Double(f64),
+}
+
+impl From<String> for OptionValue {
+ fn from(value: String) -> Self {
+ Self::String(value)
+ }
+}
+
+impl From<&str> for OptionValue {
+ fn from(value: &str) -> Self {
+ Self::String(value.into())
+ }
+}
+
+impl From<i64> for OptionValue {
+ fn from(value: i64) -> Self {
+ Self::Int(value)
+ }
+}
+
+impl From<f64> for OptionValue {
+ fn from(value: f64) -> Self {
+ Self::Double(value)
+ }
+}
+
+impl From<Vec<u8>> for OptionValue {
+ fn from(value: Vec<u8>) -> Self {
+ Self::Bytes(value)
+ }
+}
+
+impl From<&[u8]> for OptionValue {
+ fn from(value: &[u8]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<[u8; N]> for OptionValue {
+ fn from(value: [u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+impl<const N: usize> From<&[u8; N]> for OptionValue {
+ fn from(value: &[u8; N]) -> Self {
+ Self::Bytes(value.into())
+ }
+}
+
+/// ADBC revision versions.
+#[derive(Clone, Copy)]
+pub enum AdbcVersion {
+ /// Version 1.0.0.
+ V100,
+ /// Version 1.1.0.
+ V110,
+}
+
+impl From<AdbcVersion> for c_int {
+ fn from(value: AdbcVersion) -> Self {
+ match value {
+ AdbcVersion::V100 => constants::ADBC_VERSION_1_0_0,
+ AdbcVersion::V110 => constants::ADBC_VERSION_1_1_0,
+ }
+ }
+}
+
+/// Info codes for database/driver metadata.
+#[derive(Debug)]
+pub enum InfoCode {
+ /// The database vendor/product name (type: utf8).
+ VendorName,
+ /// The database vendor/product version (type: utf8).
+ VendorVersion,
+ /// The database vendor/product Arrow library version (type: utf8).
+ VendorArrowVersion,
+ /// The driver name (type: utf8).
+ DriverName,
+ /// The driver version (type: utf8).
+ DriverVersion,
+ /// The driver Arrow library version (type: utf8).
+ DriverArrowVersion,
+ /// The driver ADBC API version (type: int64).
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ DriverAdbcVersion,
+ // TODO(alexandreyc): add new codes (see https://github.com/apache/arrow-adbc/commit/aa04aadccd319e6fa3abb07154fa8d87b58d5c21)
+}
+
+impl From<&InfoCode> for u32 {
+ fn from(value: &InfoCode) -> Self {
+ match value {
+ InfoCode::VendorName => constants::ADBC_INFO_VENDOR_NAME,
+ InfoCode::VendorVersion => constants::ADBC_INFO_VENDOR_VERSION,
+ InfoCode::VendorArrowVersion => constants::ADBC_INFO_VENDOR_ARROW_VERSION,
+ InfoCode::DriverName => constants::ADBC_INFO_DRIVER_NAME,
+ InfoCode::DriverVersion => constants::ADBC_INFO_DRIVER_VERSION,
+ InfoCode::DriverArrowVersion => constants::ADBC_INFO_DRIVER_ARROW_VERSION,
+ InfoCode::DriverAdbcVersion => constants::ADBC_INFO_DRIVER_ADBC_VERSION,
+ }
+ }
+}
+
+impl TryFrom<u32> for InfoCode {
+ type Error = Error;
+
+ fn try_from(value: u32) -> Result<Self, Self::Error> {
+ match value {
+ constants::ADBC_INFO_VENDOR_NAME => Ok(InfoCode::VendorName),
+ constants::ADBC_INFO_VENDOR_VERSION => Ok(InfoCode::VendorVersion),
+ constants::ADBC_INFO_VENDOR_ARROW_VERSION => Ok(InfoCode::VendorArrowVersion),
+ constants::ADBC_INFO_DRIVER_NAME => Ok(InfoCode::DriverName),
+ constants::ADBC_INFO_DRIVER_VERSION => Ok(InfoCode::DriverVersion),
+ constants::ADBC_INFO_DRIVER_ARROW_VERSION => Ok(InfoCode::DriverArrowVersion),
+ constants::ADBC_INFO_DRIVER_ADBC_VERSION => Ok(InfoCode::DriverAdbcVersion),
+ v => Err(Error::with_message_and_status(
+ &format!("Unknown info code: {}", v),
+ Status::InvalidData,
+ )),
+ }
+ }
+}
+
+/// Depth parameter for [get_objects][crate::Connection::get_objects] method.
+#[derive(Debug)]
+pub enum ObjectDepth {
Review Comment:
Well, after reconsideration here are my conclusions:
1. `Status` and `InfoCode` are represented as fixed-width integers in C (`u8` and `u32` respectively). In this case there should be no probleme using `#[repr(u8/u32)]`.
2. `ObjectDepth` and `AdbcVersion` are represented in C as `int` but we cannot `#[repr(c_int)]` so we must choose an `i*` but this is platform dependent... One solution would be to choose `i16` (because it's the minimum size imposed by the C standard for `int` and we will probably never have more than 2^16 variants) and widen values to `c_int` when required.
Personally I find this unecessary mental overhead and explicit conversion between "native" enums and integer is simpler (and it has the added benefit of handling invalid values as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1568491088
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
Review Comment:
> Is this for avoiding querying the same info code multiple times?
Yes, that's why I was suggesting that change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1569011077
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
Review Comment:
> we must be able to set an Int option and get it back as String (or any other combination of types)
this isn't strictly required, fwiw
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1570211715
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and
+ /// columns.
+ ///
+ /// # Arguments
+ ///
+ /// - `depth` - The level of nesting to query.
+ /// - `catalog` - Only show tables in the given catalog. If `None`,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern.
+ /// - `db_schema` - Only show tables in the given database schema. If
+ /// `None`, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern.
+ /// - `table_name` - Only show tables with the given name. If `None`, do not
+ /// filter by name. May be a search pattern.
+ /// - `table_type` - Only show tables matching one of the given table
+ /// types. If `None`, show tables of any type. Valid table types can be fetched
+ /// from [Connection::get_table_types].
+ /// - `column_name` - Only show columns with the given name. If
+ /// `None`, do not filter by name. May be a search pattern..
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|--------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> |
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_tables | list\<TABLE_SCHEMA\> |
+ ///
+ /// TABLE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|---------------------------|
+ /// | table_name | utf8 not null |
+ /// | table_type | utf8 not null |
+ /// | table_columns | list\<COLUMN_SCHEMA\> |
+ /// | table_constraints | list\<CONSTRAINT_SCHEMA\> |
+ ///
+ /// COLUMN_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | column_name | utf8 not null | |
+ /// | ordinal_position | int32 | (1) |
+ /// | remarks | utf8 | (2) |
+ /// | xdbc_data_type | int16 | (3) |
+ /// | xdbc_type_name | utf8 | (3) |
+ /// | xdbc_column_size | int32 | (3) |
+ /// | xdbc_decimal_digits | int16 | (3) |
+ /// | xdbc_num_prec_radix | int16 | (3) |
+ /// | xdbc_nullable | int16 | (3) |
+ /// | xdbc_column_def | utf8 | (3) |
+ /// | xdbc_sql_data_type | int16 | (3) |
+ /// | xdbc_datetime_sub | int16 | (3) |
+ /// | xdbc_char_octet_length | int32 | (3) |
+ /// | xdbc_is_nullable | utf8 | (3) |
+ /// | xdbc_scope_catalog | utf8 | (3) |
+ /// | xdbc_scope_schema | utf8 | (3) |
+ /// | xdbc_scope_table | utf8 | (3) |
+ /// | xdbc_is_autoincrement | bool | (3) |
+ /// | xdbc_is_generatedcolumn | bool | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// `xdbc_` values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// CONSTRAINT_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|-------------------------|----------|
+ /// | constraint_name | utf8 | |
+ /// | constraint_type | utf8 not null | (1) |
+ /// | constraint_column_names | list\<utf8\> not null | (2) |
+ /// | constraint_column_usage | list\<USAGE_SCHEMA\> | (3) |
+ ///
+ /// 1. One of `CHECK`, `FOREIGN KEY`, `PRIMARY KEY`, or `UNIQUE`.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For `FOREIGN KEY` only, the referenced table and columns.
+ ///
+ /// USAGE_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-------------------------|
+ /// | fk_catalog | utf8 |
+ /// | fk_db_schema | utf8 |
+ /// | fk_table | utf8 not null |
+ /// | fk_column_name | utf8 not null |
+ ///
+ fn get_objects(
+ &self,
+ depth: options::ObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: Option<Vec<&str>>,
+ column_name: Option<&str>,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable).
+ /// - `db_schema` - The database schema (or `None` if not applicable).
+ /// - `table_name` - The table name.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema>;
+
+ /// Get a list of table types in the database.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|--------------
+ /// table_type | utf8 not null
+ fn get_table_types(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get the names of statistics specific to this driver.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ---------------|----------------
+ /// statistic_name | utf8 not null
+ /// statistic_key | int16 not null
+ ///
+ /// # Since
+ /// ADBC API revision 1.1.0
+ fn get_statistic_names(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Get statistics about the data distribution of table(s).
+ ///
+ /// # Arguments
+ ///
+ /// - `catalog` - The catalog (or `None` if not applicable). May be a search pattern.
+ /// - `db_schema` - The database schema (or `None` if not applicable). May be a search pattern
+ /// - `table_name` - The table name (or `None` if not applicable). May be a search pattern
+ /// - `approximate` - If false, request exact values of statistics, else
+ /// allow for best-effort, approximate, or cached values. The database may
+ /// return approximate values regardless, as indicated in the result.
+ /// Requesting exact values may be expensive or unsupported.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | catalog_name | utf8 |
+ /// | catalog_db_schemas | list\<DB_SCHEMA_SCHEMA\> not null|
+ ///
+ /// DB_SCHEMA_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|-----------------------------------|
+ /// | db_schema_name | utf8 |
+ /// | db_schema_statistics | list\<STATISTICS_SCHEMA\> not null|
+ ///
+ /// STATISTICS_SCHEMA is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |--------------------------|----------------------------------| -------- |
+ /// | table_name | utf8 not null | |
+ /// | column_name | utf8 | (1) |
+ /// | statistic_key | int16 not null | (2) |
+ /// | statistic_value | VALUE_SCHEMA not null | |
+ /// | statistic_is_approximate | bool not null | (3) |
+ ///
+ /// 1. If null, then the statistic applies to the entire table.
+ /// 2. A dictionary-encoded statistic name (although we do not use the Arrow
+ /// dictionary type). Values in [0, 1024) are reserved for ADBC. Other
+ /// values are for implementation-specific statistics. For the definitions
+ /// of predefined statistic types, TODO (change this when statistics enum is added )
+ /// see \ref adbc-table-statistics. To get
+ /// driver-specific statistic names, use [Connection::get_statistic_names].
+ /// 3. If true, then the value is approximate or best-effort.
+ ///
+ /// VALUE_SCHEMA is a dense union with members:
+ ///
+ /// | Field Name | Field Type |
+ /// |--------------------------|----------------------------------|
+ /// | int64 | int64 |
+ /// | uint64 | uint64 |
+ /// | float64 | float64 |
+ /// | binary | binary |
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn get_statistics(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ approximate: bool,
+ ) -> Result<impl RecordBatchReader + Send>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn commit(&self) -> Result<()>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ ///
+ /// Behavior is undefined if this is mixed with SQL transaction statements.
+ fn rollback(&self) -> Result<()>;
+
+ /// Retrieve a given partition of data.
+ ///
+ /// A partition can be retrieved from [Statement::execute_partitions].
+ ///
+ /// # Arguments
+ ///
+ /// - `partition` - The partition descriptor.
+ fn read_partition(&self, partition: &[u8]) -> Result<impl RecordBatchReader + Send>;
+}
+
+/// A handle to an ADBC statement.
+///
+/// A statement is a container for all state needed to execute a database query,
+/// such as the query itself, parameters for prepared statements, driver
+/// parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used concurrently
+/// (whether from a single thread or multiple threads).
+pub trait Statement: Optionable<Option = OptionStatement> {
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind(&self, batch: RecordBatch) -> Result<()>;
+
+ /// Bind Arrow data. This can be used for bulk inserts or prepared
+ /// statements.
+ fn bind_stream(&self, reader: Box<dyn RecordBatchReader + Send>) -> Result<()>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// This invalidates any prior result sets.
+ fn execute(&self) -> Result<impl RecordBatchReader + Send>;
+
+ /// Execute a statement that doesn’t have a result set and get the number
+ /// of affected rows.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// # Result
+ ///
+ /// Will return the number of rows affected. If the affected row count is
+ /// unknown or unsupported by the database, will return `Ok(-1)`.
+ fn execute_update(&self) -> Result<i64>;
+
+ /// Get the schema of the result set of a query without executing it.
+ ///
+ /// This invalidates any prior result sets.
+ ///
+ /// Depending on the driver, this may require first executing
+ /// [Statement::prepare].
+ ///
+ /// # Since
+ ///
+ /// ADBC API revision 1.1.0
+ fn execute_schema(&self) -> Result<Schema>;
+
+ /// Execute a statement and get the results as a partitioned result set.
+ fn execute_partitions(&self) -> Result<PartitionedResult>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should be called after [Statement::prepare].
+ fn get_parameters_schema(&self) -> Result<Schema>;
+
+ /// Turn this statement into a prepared statement to be executed multiple
+ /// times.
+ ///
+ /// This invalidates any prior result sets.
+ fn prepare(&self) -> Result<()>;
+
+ /// Set the SQL query to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_sql_query(&self, query: &str) -> Result<()>;
+
+ /// Set the Substrait plan to execute.
+ ///
+ /// The query can then be executed with [Statement::execute]. For queries
+ /// expected to be executed repeatedly, call [Statement::prepare] first.
+ fn set_substrait_plan(&self, plan: &[u8]) -> Result<()>;
Review Comment:
This is hypothetical but here is my intuition at the moment:
- The low-level API can accomodate all the semantics of the C API
- If your driver implement this low-level API, you can automatically (thanks to the driver exporter) build it as an object file (static or dynamic) that other languages can use (via their own driver manager).
- The high-level API is built on the low-level API and is more idiomatic and less error-prone BUT it cannot be shared with other languages (it might possible to automatically implement this high-level API from the low-level API, ala driver exporter) and probably cannot perform all operations made possible by the C API.
From a driver implementor perspective you should first implement the low-level API, so that you can share your driver with others, which is kind of the point of ADBC. If it's possible to automatically implement the high-level API then this is a win-win. If you decide to only implement the high-level API, only Rust users will be able to use it.
From a driver consumer perspective you should almost always use the high-level API since it's more ergonomic (except if you need some semantics that's not availabe in the high-level API).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1570637817
##########
rust2/drivers/dummy/src/lib.rs:
##########
@@ -0,0 +1,890 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+use std::{collections::HashMap, fmt::Debug, hash::Hash};
+
+use arrow::array::{
+ Array, ArrayRef, BinaryArray, BooleanArray, Float64Array, Int16Array, Int32Array, Int64Array,
+ ListArray, MapArray, StringArray, StructArray, UInt32Array, UInt64Array, UnionArray,
+};
+use arrow::buffer::{Buffer, OffsetBuffer, ScalarBuffer};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use adbc_core::{
+ error::{Error, Result, Status},
+ ffi::constants,
+ options::{
+ InfoCode, ObjectDepth, OptionConnection, OptionDatabase, OptionStatement, OptionValue,
+ },
+ schemas, Connection, Database, Driver, Optionable, PartitionedResult, Statement,
+};
+
+#[derive(Debug)]
+pub struct SingleBatchReader {
+ batch: Option<RecordBatch>,
+ schema: SchemaRef,
+}
+
+impl SingleBatchReader {
+ pub fn new(batch: RecordBatch) -> Self {
+ let schema = batch.schema();
+ Self {
+ batch: Some(batch),
+ schema,
+ }
+ }
+}
+
+impl Iterator for SingleBatchReader {
+ type Item = std::result::Result<RecordBatch, ArrowError>;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ Ok(self.batch.take()).transpose()
+ }
+}
+
+impl RecordBatchReader for SingleBatchReader {
+ fn schema(&self) -> SchemaRef {
+ self.schema.clone()
+ }
+}
+
+fn get_table_schema() -> Schema {
+ Schema::new(vec![
+ Field::new("a", DataType::UInt32, true),
+ Field::new("b", DataType::Float64, false),
+ Field::new("c", DataType::Utf8, true),
+ ])
+}
+
+fn get_table_data() -> RecordBatch {
+ RecordBatch::try_new(
+ Arc::new(get_table_schema()),
+ vec![
+ Arc::new(UInt32Array::from(vec![1, 2, 3])),
+ Arc::new(Float64Array::from(vec![1.5, 2.5, 3.5])),
+ Arc::new(StringArray::from(vec!["A", "B", "C"])),
+ ],
+ )
+ .unwrap()
+}
+
+fn set_option<T>(options: &mut HashMap<T, OptionValue>, key: T, value: OptionValue) -> Result<()>
+where
+ T: Eq + Hash,
+{
+ options.insert(key, value);
+ Ok(())
+}
+
+fn get_option_bytes<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<Vec<u8>>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Bytes(value) => Ok(value.clone()),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_double<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<f64>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Double(value) => Ok(*value),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_int<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<i64>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Int(value) => Ok(*value),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_string<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<String>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::String(value) => Ok(value.clone()),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+/// A dummy driver used for testing purposes.
+#[derive(Default)]
+pub struct DummyDriver {}
+
+impl Driver for DummyDriver {
+ type DatabaseType = DummyDatabase;
+
+ fn new_database(&mut self) -> Result<Self::DatabaseType> {
+ self.new_database_with_opts([].into_iter())
+ }
+
+ fn new_database_with_opts(
+ &mut self,
+ opts: impl Iterator<Item = (<Self::DatabaseType as Optionable>::Option, OptionValue)>,
Review Comment:
Nit: looking at the example here, `impl IntoIterator<Item = ...>` would be little bit nicer.
##########
rust2/drivers/dummy/src/lib.rs:
##########
@@ -0,0 +1,890 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+use std::{collections::HashMap, fmt::Debug, hash::Hash};
+
+use arrow::array::{
+ Array, ArrayRef, BinaryArray, BooleanArray, Float64Array, Int16Array, Int32Array, Int64Array,
+ ListArray, MapArray, StringArray, StructArray, UInt32Array, UInt64Array, UnionArray,
+};
+use arrow::buffer::{Buffer, OffsetBuffer, ScalarBuffer};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use adbc_core::{
+ error::{Error, Result, Status},
+ ffi::constants,
+ options::{
+ InfoCode, ObjectDepth, OptionConnection, OptionDatabase, OptionStatement, OptionValue,
+ },
+ schemas, Connection, Database, Driver, Optionable, PartitionedResult, Statement,
+};
+
+#[derive(Debug)]
+pub struct SingleBatchReader {
+ batch: Option<RecordBatch>,
+ schema: SchemaRef,
+}
+
+impl SingleBatchReader {
+ pub fn new(batch: RecordBatch) -> Self {
+ let schema = batch.schema();
+ Self {
+ batch: Some(batch),
+ schema,
+ }
+ }
+}
+
+impl Iterator for SingleBatchReader {
+ type Item = std::result::Result<RecordBatch, ArrowError>;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ Ok(self.batch.take()).transpose()
+ }
+}
+
+impl RecordBatchReader for SingleBatchReader {
+ fn schema(&self) -> SchemaRef {
+ self.schema.clone()
+ }
+}
+
+fn get_table_schema() -> Schema {
+ Schema::new(vec![
+ Field::new("a", DataType::UInt32, true),
+ Field::new("b", DataType::Float64, false),
+ Field::new("c", DataType::Utf8, true),
+ ])
+}
+
+fn get_table_data() -> RecordBatch {
+ RecordBatch::try_new(
+ Arc::new(get_table_schema()),
+ vec![
+ Arc::new(UInt32Array::from(vec![1, 2, 3])),
+ Arc::new(Float64Array::from(vec![1.5, 2.5, 3.5])),
+ Arc::new(StringArray::from(vec!["A", "B", "C"])),
+ ],
+ )
+ .unwrap()
+}
+
+fn set_option<T>(options: &mut HashMap<T, OptionValue>, key: T, value: OptionValue) -> Result<()>
+where
+ T: Eq + Hash,
+{
+ options.insert(key, value);
+ Ok(())
+}
+
+fn get_option_bytes<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<Vec<u8>>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Bytes(value) => Ok(value.clone()),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_double<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<f64>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Double(value) => Ok(*value),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_int<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<i64>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Int(value) => Ok(*value),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_string<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<String>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::String(value) => Ok(value.clone()),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+/// A dummy driver used for testing purposes.
+#[derive(Default)]
+pub struct DummyDriver {}
+
+impl Driver for DummyDriver {
+ type DatabaseType = DummyDatabase;
+
+ fn new_database(&mut self) -> Result<Self::DatabaseType> {
+ self.new_database_with_opts([].into_iter())
+ }
+
+ fn new_database_with_opts(
+ &mut self,
+ opts: impl Iterator<Item = (<Self::DatabaseType as Optionable>::Option, OptionValue)>,
+ ) -> Result<Self::DatabaseType> {
+ let mut database = Self::DatabaseType {
+ options: HashMap::new(),
+ };
+ for (key, value) in opts {
+ database.set_option(key, value)?;
+ }
+ Ok(database)
+ }
+}
+
+pub struct DummyDatabase {
+ options: HashMap<OptionDatabase, OptionValue>,
+}
+
+impl Optionable for DummyDatabase {
+ type Option = OptionDatabase;
+
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()> {
+ set_option(&mut self.options, key, value)
+ }
+
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>> {
+ get_option_bytes(&self.options, key, "database")
+ }
+
+ fn get_option_double(&self, key: Self::Option) -> Result<f64> {
+ get_option_double(&self.options, key, "database")
+ }
+
+ fn get_option_int(&self, key: Self::Option) -> Result<i64> {
+ get_option_int(&self.options, key, "database")
+ }
+
+ fn get_option_string(&self, key: Self::Option) -> Result<String> {
+ get_option_string(&self.options, key, "database")
+ }
+}
+
+impl Database for DummyDatabase {
+ type ConnectionType = DummyConnection;
+
+ fn new_connection(&mut self) -> Result<Self::ConnectionType> {
+ self.new_connection_with_opts([].into_iter())
+ }
+
+ fn new_connection_with_opts(
+ &mut self,
+ opts: impl Iterator<Item = (<Self::ConnectionType as Optionable>::Option, OptionValue)>,
Review Comment:
Same here.
##########
rust2/drivers/dummy/src/lib.rs:
##########
@@ -0,0 +1,890 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+use std::{collections::HashMap, fmt::Debug, hash::Hash};
+
+use arrow::array::{
+ Array, ArrayRef, BinaryArray, BooleanArray, Float64Array, Int16Array, Int32Array, Int64Array,
+ ListArray, MapArray, StringArray, StructArray, UInt32Array, UInt64Array, UnionArray,
+};
+use arrow::buffer::{Buffer, OffsetBuffer, ScalarBuffer};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use arrow::error::ArrowError;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use adbc_core::{
+ error::{Error, Result, Status},
+ ffi::constants,
+ options::{
+ InfoCode, ObjectDepth, OptionConnection, OptionDatabase, OptionStatement, OptionValue,
+ },
+ schemas, Connection, Database, Driver, Optionable, PartitionedResult, Statement,
+};
+
+#[derive(Debug)]
+pub struct SingleBatchReader {
+ batch: Option<RecordBatch>,
+ schema: SchemaRef,
+}
+
+impl SingleBatchReader {
+ pub fn new(batch: RecordBatch) -> Self {
+ let schema = batch.schema();
+ Self {
+ batch: Some(batch),
+ schema,
+ }
+ }
+}
+
+impl Iterator for SingleBatchReader {
+ type Item = std::result::Result<RecordBatch, ArrowError>;
+
+ fn next(&mut self) -> Option<Self::Item> {
+ Ok(self.batch.take()).transpose()
+ }
+}
+
+impl RecordBatchReader for SingleBatchReader {
+ fn schema(&self) -> SchemaRef {
+ self.schema.clone()
+ }
+}
+
+fn get_table_schema() -> Schema {
+ Schema::new(vec![
+ Field::new("a", DataType::UInt32, true),
+ Field::new("b", DataType::Float64, false),
+ Field::new("c", DataType::Utf8, true),
+ ])
+}
+
+fn get_table_data() -> RecordBatch {
+ RecordBatch::try_new(
+ Arc::new(get_table_schema()),
+ vec![
+ Arc::new(UInt32Array::from(vec![1, 2, 3])),
+ Arc::new(Float64Array::from(vec![1.5, 2.5, 3.5])),
+ Arc::new(StringArray::from(vec!["A", "B", "C"])),
+ ],
+ )
+ .unwrap()
+}
+
+fn set_option<T>(options: &mut HashMap<T, OptionValue>, key: T, value: OptionValue) -> Result<()>
+where
+ T: Eq + Hash,
+{
+ options.insert(key, value);
+ Ok(())
+}
+
+fn get_option_bytes<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<Vec<u8>>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Bytes(value) => Ok(value.clone()),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_double<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<f64>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Double(value) => Ok(*value),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_int<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<i64>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::Int(value) => Ok(*value),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+fn get_option_string<T>(options: &HashMap<T, OptionValue>, key: T, kind: &str) -> Result<String>
+where
+ T: Eq + Hash + Debug,
+{
+ let value = options.get(&key);
+ match value {
+ None => Err(Error::with_message_and_status(
+ format!("Unrecognized {kind} option: {key:?}"),
+ Status::NotFound,
+ )),
+ Some(value) => match value {
+ OptionValue::String(value) => Ok(value.clone()),
+ _ => Err(Error::with_message_and_status(
+ format!("Incorrect value for {kind} option: {key:?}"),
+ Status::InvalidData,
+ )),
+ },
+ }
+}
+
+/// A dummy driver used for testing purposes.
+#[derive(Default)]
+pub struct DummyDriver {}
+
+impl Driver for DummyDriver {
+ type DatabaseType = DummyDatabase;
+
+ fn new_database(&mut self) -> Result<Self::DatabaseType> {
+ self.new_database_with_opts([].into_iter())
+ }
+
+ fn new_database_with_opts(
+ &mut self,
+ opts: impl Iterator<Item = (<Self::DatabaseType as Optionable>::Option, OptionValue)>,
+ ) -> Result<Self::DatabaseType> {
+ let mut database = Self::DatabaseType {
+ options: HashMap::new(),
+ };
+ for (key, value) in opts {
+ database.set_option(key, value)?;
+ }
+ Ok(database)
+ }
+}
+
+pub struct DummyDatabase {
+ options: HashMap<OptionDatabase, OptionValue>,
+}
+
+impl Optionable for DummyDatabase {
+ type Option = OptionDatabase;
+
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()> {
+ set_option(&mut self.options, key, value)
+ }
+
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>> {
+ get_option_bytes(&self.options, key, "database")
+ }
+
+ fn get_option_double(&self, key: Self::Option) -> Result<f64> {
+ get_option_double(&self.options, key, "database")
+ }
+
+ fn get_option_int(&self, key: Self::Option) -> Result<i64> {
+ get_option_int(&self.options, key, "database")
+ }
+
+ fn get_option_string(&self, key: Self::Option) -> Result<String> {
+ get_option_string(&self.options, key, "database")
+ }
+}
+
+impl Database for DummyDatabase {
+ type ConnectionType = DummyConnection;
+
+ fn new_connection(&mut self) -> Result<Self::ConnectionType> {
+ self.new_connection_with_opts([].into_iter())
+ }
+
+ fn new_connection_with_opts(
+ &mut self,
+ opts: impl Iterator<Item = (<Self::ConnectionType as Optionable>::Option, OptionValue)>,
+ ) -> Result<Self::ConnectionType> {
+ let mut connection = Self::ConnectionType {
+ options: HashMap::new(),
+ };
+ for (key, value) in opts {
+ connection.set_option(key, value)?;
+ }
+ Ok(connection)
+ }
+}
+
+pub struct DummyConnection {
+ options: HashMap<OptionConnection, OptionValue>,
+}
+
+impl Optionable for DummyConnection {
+ type Option = OptionConnection;
+
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()> {
+ set_option(&mut self.options, key, value)
+ }
+
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>> {
+ get_option_bytes(&self.options, key, "connection")
+ }
+
+ fn get_option_double(&self, key: Self::Option) -> Result<f64> {
+ get_option_double(&self.options, key, "connection")
+ }
+
+ fn get_option_int(&self, key: Self::Option) -> Result<i64> {
+ get_option_int(&self.options, key, "connection")
+ }
+
+ fn get_option_string(&self, key: Self::Option) -> Result<String> {
+ get_option_string(&self.options, key, "connection")
+ }
+}
+
+impl Connection for DummyConnection {
+ type StatementType = DummyStatement;
+
+ fn new_statement(&mut self) -> Result<Self::StatementType> {
+ Ok(Self::StatementType {
+ options: HashMap::new(),
+ })
+ }
+
+ // This method is used to test that errors round-trip correctly.
+ fn cancel(&mut self) -> Result<()> {
+ let mut error = Error::with_message_and_status("message", Status::Cancelled);
+ error.vendor_code = constants::ADBC_ERROR_VENDOR_CODE_PRIVATE_DATA;
+ error.sqlstate = [1, 2, 3, 4, 5];
+ error.details = Some(vec![
+ ("key1".into(), b"AAA".into()),
+ ("key2".into(), b"ZZZZZ".into()),
+ ]);
+ Err(error)
+ }
+
+ fn commit(&mut self) -> Result<()> {
+ Ok(())
+ }
+
+ fn get_info(&self, _codes: Option<HashSet<InfoCode>>) -> Result<impl RecordBatchReader> {
Review Comment:
Nit: would be nice if the codes were used in this example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567840868
##########
rust2/core/src/ffi/constants.rs:
##########
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::os::raw::c_int;
+
+use super::types::FFI_AdbcStatusCode;
+
+pub const ADBC_STATUS_OK: FFI_AdbcStatusCode = 0;
+pub const ADBC_STATUS_UNKNOWN: FFI_AdbcStatusCode = 1;
+pub const ADBC_STATUS_NOT_IMPLEMENTED: FFI_AdbcStatusCode = 2;
+pub const ADBC_STATUS_NOT_FOUND: FFI_AdbcStatusCode = 3;
+pub const ADBC_STATUS_ALREADY_EXISTS: FFI_AdbcStatusCode = 4;
+pub const ADBC_STATUS_INVALID_ARGUMENT: FFI_AdbcStatusCode = 5;
+pub const ADBC_STATUS_INVALID_STATE: FFI_AdbcStatusCode = 6;
+pub const ADBC_STATUS_INVALID_DATA: FFI_AdbcStatusCode = 7;
+pub const ADBC_STATUS_INTEGRITY: FFI_AdbcStatusCode = 8;
+pub const ADBC_STATUS_INTERNAL: FFI_AdbcStatusCode = 9;
+pub const ADBC_STATUS_IO: FFI_AdbcStatusCode = 10;
+pub const ADBC_STATUS_CANCELLED: FFI_AdbcStatusCode = 11;
+pub const ADBC_STATUS_TIMEOUT: FFI_AdbcStatusCode = 12;
+pub const ADBC_STATUS_UNAUTHENTICATED: FFI_AdbcStatusCode = 13;
+pub const ADBC_STATUS_UNAUTHORIZED: FFI_AdbcStatusCode = 14;
+
+pub const ADBC_VERSION_1_0_0: i32 = 1000000;
+pub const ADBC_VERSION_1_1_0: i32 = 1001000;
Review Comment:
In `adbc.h`, `AdbcDriverInitFunc` take `version` as `c_int`. Maybe we should use `c_int` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "alexandreyc (via GitHub)" <gi...@apache.org>.
alexandreyc commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1567822352
##########
rust2/core/src/error.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Error, status and result types.
+
+use std::{ffi::NulError, fmt::Display};
+
+use arrow::error::ArrowError;
+
+/// Status of an operation.
+#[derive(Debug, Copy, Clone, PartialEq, Eq)]
+pub enum Status {
+ /// No error.
+ Ok,
+ /// An unknown error occurred.
+ Unknown,
+ /// The operation is not implemented or supported.
+ NotImplemented,
+ /// A requested resource was not found.
+ NotFound,
+ /// A requested resource already exists.
+ AlreadyExists,
+ /// The arguments are invalid, likely a programming error.
+ /// For instance, they may be of the wrong format, or out of range.
+ InvalidArguments,
+ /// The preconditions for the operation are not met, likely a programming error.
+ /// For instance, the object may be uninitialized, or may have not
+ /// been fully configured.
+ InvalidState,
+ /// Invalid data was processed (not a programming error).
+ /// For instance, a division by zero may have occurred during query
+ /// execution.
+ InvalidData,
+ /// The database's integrity was affected.
+ /// For instance, a foreign key check may have failed, or a uniqueness
+ /// constraint may have been violated.
+ Integrity,
+ /// An error internal to the driver or database occurred.
+ Internal,
+ /// An I/O error occurred.
+ /// For instance, a remote service may be unavailable.
+ IO,
+ /// The operation was cancelled, not due to a timeout.
+ Cancelled,
+ /// The operation was cancelled due to a timeout.
+ Timeout,
+ /// Authentication failed.
+ Unauthenticated,
+ /// The client is not authorized to perform the given operation.
+ Unauthorized,
+}
+
+/// An ADBC error.
+#[derive(Debug, PartialEq, Eq)]
+pub struct Error {
+ /// The error message.
+ pub message: String,
+ /// The status of the operation.
+ pub status: Status,
+ /// A vendor-specific error code, if applicable.
+ pub vendor_code: i32,
+ /// A SQLSTATE error code, if provided, as defined by the SQL:2003 standard.
+ /// If not set, it should be set to `\0\0\0\0\0`.
+ pub sqlstate: [i8; 5],
Review Comment:
Because `c_char` [translates](https://doc.rust-lang.org/std/ffi/type.c_char.html) to `i8` on my platform. Maybe we can switch to `c_char`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1982186406
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
> ADBC connection should probably be thread-confined since ordering of statements executed on them matters.
Does it always matter? My suggestion here was to make this opt-in instead of required.
Can you give an example of a reader sharing resources with the connection like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1986131177
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
I changed my mind regarding
> ADBC connection should probably be thread-confined since ordering of statements executed on them matters.
after reading the `adbc.h` comments and noticing that it explicitly says every object should be accessible by different threads, but access must be synchronized.
## But that then means the current annotations are too restrictive...
There is a problem with the current annotations in the Rust wrapper. If I try to `Box` the `impl RecordBatchReader` returned by `adbc_core::Connection` methods, `rustc` complains that the boxed value may outlive the references I'm passing into the calls. To make that acceptable, the annotations has to change to:
```rust
fn get_info<'a>(
&self, // 'a not added here because result can outlive connection
codes: Option<Vec<options::InfoCode>>,
) -> Result<impl RecordBatchReader + 'a>;
```
This is even more critical in functions taking many references
```diff
- fn get_statistics(
+ fn get_statistics<'a>(
&self,
catalog: Option<&str>,
db_schema: Option<&str>,
table_name: Option<&str>,
approximate: bool,
- ) -> Result<impl RecordBatchReader + Send>;
+ ) -> Result<impl RecordBatchReader + Send + 'a>;
```
This would let me `Box`/`Rc`/`Arc` the returned record batch reader without `rustc` complaining that lifetime of the reader may be longer than the lifetime of the references I passed in.
> Does it always matter? My suggestion here was to make this opt-in instead of required.
In conclusion: to make it opt-in, lifetime annotations should change. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(rust): add public abstract API and dummy driver implementation [arrow-adbc]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #1725:
URL: https://github.com/apache/arrow-adbc/pull/1725#discussion_r1986206868
##########
rust2/core/src/lib.rs:
##########
@@ -0,0 +1,520 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! ADBC: Arrow Database Connectivity
+//!
+//! ADBC is a set of APIs and libraries for [Arrow](https://arrow.apache.org/)-native
+//! access to databases. Execute SQL and [Substrait](https://substrait.io/)
+//! queries, query database catalogs, and more, all using Arrow data to
+//! eliminate unnecessary data copies, speed up access, and make it more
+//! convenient to build analytical applications.
+//!
+//! Read more about ADBC at <https://arrow.apache.org/adbc/>
+//!
+//! This library currently provides:
+//! - An abstract Rust API to be implemented by vendor-specific drivers.
+//! - A driver manager which implements this same API, but dynamically loads
+//! drivers internally and forwards calls appropriately using the [C API](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+//!
+//! We hope to provide the ability to expose Rust drivers as C drivers in the
+//! near future permitting other languages to use Rust drivers.
+//!
+//! # Native Rust drivers
+//!
+//! Native Rust drivers will implement the abstract API consisting of the traits:
+//! - [Driver]
+//! - [Database]
+//! - [Connection]
+//! - [Statement]
+//!
+//! For drivers implemented in Rust, using these will be more efficient and
+//! safe, since it avoids the overhead of going through C FFI.
+//!
+//! # Driver Manager
+//!
+//! The [driver_manager] module allows loading drivers exposing the C API,
+//! either from an initialization function (link-time, either static or dynamic)
+//! or by dynamically finding such a function in a dynamic library (run-time).
+
+// TODO(alexandreyc): uncomment these lines during follow-up PRs
+// pub mod driver_exporter;
+// pub mod driver_manager;
+// pub use ffi::FFI_AdbcDriverInitFunc as AdbcDriverInitFunc;
+
+pub mod error;
+pub mod ffi;
+pub mod options;
+pub mod schemas;
+
+use arrow::datatypes::Schema;
+use arrow::record_batch::{RecordBatch, RecordBatchReader};
+
+use error::Result;
+use options::{OptionConnection, OptionDatabase, OptionStatement, OptionValue};
+
+/// Ability to configure an object by setting/getting options.
+pub trait Optionable {
+ type Option: AsRef<str>;
+
+ /// Set a post-init option.
+ fn set_option(&mut self, key: Self::Option, value: OptionValue) -> Result<()>;
+
+ /// Get a string option value by key.
+ fn get_option_string(&self, key: Self::Option) -> Result<String>;
+
+ /// Get a bytes option value by key.
+ fn get_option_bytes(&self, key: Self::Option) -> Result<Vec<u8>>;
+
+ /// Get an integer option value by key.
+ fn get_option_int(&self, key: Self::Option) -> Result<i64>;
+
+ /// Get a float option value by key.
+ fn get_option_double(&self, key: Self::Option) -> Result<f64>;
+}
+
+/// A handle to an ADBC driver.
+pub trait Driver {
+ type DatabaseType: Database;
+
+ /// Allocate and initialize a new database without pre-init options.
+ fn new_database(&self) -> Result<Self::DatabaseType>;
+
+ /// Allocate and initialize a new database with pre-init options.
+ fn new_database_with_opts(
+ &self,
+ opts: impl Iterator<Item = (OptionDatabase, OptionValue)>,
+ ) -> Result<Self::DatabaseType>;
+}
+
+/// A handle to an ADBC database.
+///
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Databases must be kept alive as long as any connections exist.
+pub trait Database: Optionable<Option = OptionDatabase> {
+ type ConnectionType: Connection;
+
+ /// Allocate and initialize a new connection without pre-init options.
+ fn new_connection(&self) -> Result<Self::ConnectionType>;
+
+ /// Allocate and initialize a new connection with pre-init options.
+ fn new_connection_with_opts(
+ &self,
+ opts: impl Iterator<Item = (options::OptionConnection, OptionValue)>,
+ ) -> Result<Self::ConnectionType>;
+}
+
+/// A handle to an ADBC connection.
+///
+/// Connections provide methods for query execution, managing prepared
+/// statements, using transactions, and so on.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting [options::OptionConnection::AutoCommit] to "false". Turning off
+/// autocommit allows customizing the isolation level.
+pub trait Connection: Optionable<Option = OptionConnection> {
+ type StatementType: Statement;
+
+ /// Allocate and initialize a new statement.
+ fn new_statement(&self) -> Result<Self::StatementType>;
+
+ /// Cancel the in-progress operation on a connection.
+ fn cancel(&self) -> Result<()>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// # Arguments
+ ///
+ /// - `codes` - Requested metadata. If `None`, retrieve all available metadata.
+ ///
+ /// # Result
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// info_name | uint32 not null
+ /// info_value | INFO_SCHEMA
+ ///
+ /// INFO_SCHEMA is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ----------------------------|------------------------
+ /// string_value (0) | utf8
+ /// bool_value (1) | bool
+ /// int64_value (2) | int64
+ /// int32_bitmask (3) | int32
+ /// string_list (4) | list\<utf8\>
+ /// int32_to_int32_list_map (5) | map\<int32, list\<int32\>\>
+ fn get_info(
+ &self,
+ codes: Option<Vec<options::InfoCode>>,
+ ) -> Result<impl RecordBatchReader + Send>;
Review Comment:
I'm understanding first what is it that I want to suggest. I don't need (yet) all the attention an issue brings and the wave of negative comments towards something that might be a breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org