You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "wjones127 (via GitHub)" <gi...@apache.org> on 2023/01/21 22:31:15 UTC
[GitHub] [arrow-rs] wjones127 opened a new pull request, #3582: Start new ADBC crate
wjones127 opened a new pull request, #3582:
URL: https://github.com/apache/arrow-rs/pull/3582
# Which issue does this PR close?
Closes #3540.
# Rationale for this change
Rust is a good candidate for implementing an ADBC driver, but doing so involves quite a bit of `unsafe`. It would be nice to keep that `unsafe` code consolidated in one place, and allow implementors to interact instead with APIs that are more canonical Rust.
# What changes are included in this PR?
The PR adds a new `arrow_adbc` crate with three modules:
* `ffi`: contains the FFI types corresponding to each of the structs in [adbc.h](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
* `error`: contains traits relevant to translating Rust error enums into the `FFI_AdbcError` struct and `AdbcStatusCode` enum.
* `interface`: provides Rust traits to define behavior for each struct and a macro `adbc_api!()` that generates the ADBC API. Users simply have to implement each of the traits and call the macro.
# Are there any user-facing changes?
These are entirely new APIs. This PR will include documentation for all APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #3582: Start new ADBC crate
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1410210860
> Although I suspect there are users who may wish to write a driver in Rust over C/C++ purely out of desire for stronger memory-safety guarantees. (As someone whose day job involves debugging memory bugs in C++ code, I sure feel that way 😅.)
I certainly agree that rust has reduced the amount of time I spend debugging memory bugs 🎉
But I was thinking if the underlying system is already in C/C++ using Rust for the ADBC interface might not give enough extra benefit to warrant the additional set of dependencies / integration / build complexity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 commented on pull request #3582: Start new ADBC crate
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1422062813
Closing in favor of https://github.com/apache/arrow-adbc/pull/416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 commented on a diff in pull request #3582: Start new ADBC crate
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#discussion_r1084407045
##########
arrow-adbc/src/interface/mod.rs:
##########
@@ -0,0 +1,402 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implement ADBC driver without unsafe code.
+//!
+//! Allows implementing ADBC driver in Rust without directly interacting with
+//! FFI types, which requires unsafe code. Implement [AdbcDatabase],
+//! [AdbcConnection], [AdbcStatement], and [AdbcError] first. Then pass the
+//! statement type to [adbc_api] to generate C FFI interface for ADBC.
+
+use std::{rc::Rc, sync::Arc};
+
+use arrow::{array::ArrayRef, datatypes::Schema, record_batch::RecordBatchReader};
+
+use crate::{error::AdbcError, ffi::AdbcObjectDepth};
+
+pub mod internal;
+
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Because it is shared by multiple connections, the implementation must be
+/// thread safe. Internally, it is held with an [std::sync::Arc] by each connection.
+pub trait AdbcDatabase {
+ type Error: AdbcError;
+
+ /// Set an option on the database.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the database.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self) -> Result<(), Self::Error>;
+}
+
+/// A connection is a single connection to a database.
+///
+/// It is never accessed concurrently from multiple threads.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting `"adbc.connection.autocommit"` to `"false"` (using
+/// [AdbcConnection::set_option]). Turning off autocommit allows customizing
+/// the isolation level. Read more in [adbc.h](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+pub trait AdbcConnection {
+ type Error: AdbcError;
+ type DatabaseType: AdbcDatabase + Default;
+
+ /// Set an option on the connection.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the connection.
+ ///
+ /// The Arc to the database should be stored within your struct.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self, database: Arc<Self::DatabaseType>) -> Result<(), Self::Error>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// `info_name` | `uint32 not null`
+ /// `info_value` | `INFO_SCHEMA`
+ ///
+ /// `INFO_SCHEMA` is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ------------------------------|------------------------
+ /// `string_value` (0) | `utf8`
+ /// `bool_value` (1) | `bool`
+ /// `int64_value` (2) | `int64`
+ /// `int32_bitmask` (3) | `int32`
+ /// `string_list` (4) | `list<utf8>`
+ /// `int32_to_int32_list_map` (5) | `map<int32, list<int32>>`
+ ///
+ /// Each metadatum is identified by an integer code. The recognized
+ /// codes are defined as constants. Codes [0, 10_000) are reserved
+ /// for ADBC usage. Drivers/vendors will ignore requests for
+ /// unrecognized codes (the row will be omitted from the result).
+ ///
+ /// For definitions of known ADBC codes, see <https://github.com/apache/arrow-adbc/blob/main/adbc.h>
+ fn get_info(
+ &self,
+ info_codes: &[u32],
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and columns.
+ ///
+ /// # Schema
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `catalog_name` | `utf8` |
+ /// | `catalog_db_schemas` | `list<DB_SCHEMA_SCHEMA>` |
+ ///
+ /// `DB_SCHEMA_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `db_schema_name` | `utf8` |
+ /// | `db_schema_tables` | `list<TABLE_SCHEMA>` |
+ ///
+ /// `TABLE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `table_name` | `utf8 not null` |
+ /// | `table_type` | `utf8 not null` |
+ /// | `table_columns` | `list<COLUMN_SCHEMA>` |
+ /// | `table_constraints` | `list<CONSTRAINT_SCHEMA>` |
+ ///
+ /// `COLUMN_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `column_name` | `utf8 not null` | |
+ /// | `ordinal_position` | `int32` | (1) |
+ /// | `remarks` | `utf8` | (2) |
+ /// | `xdbc_data_type` | `int16` | (3) |
+ /// | `xdbc_type_name` | `utf8` | (3) |
+ /// | `xdbc_column_size` | `int32` | (3) |
+ /// | `xdbc_decimal_digits` | `int16` | (3) |
+ /// | `xdbc_num_prec_radix` | `int16` | (3) |
+ /// | `xdbc_nullable` | `int16` | (3) |
+ /// | `xdbc_column_def` | `utf8` | (3) |
+ /// | `xdbc_sql_data_type` | `int16` | (3) |
+ /// | `xdbc_datetime_sub` | `int16` | (3) |
+ /// | `xdbc_char_octet_length` | `int32` | (3) |
+ /// | `xdbc_is_nullable` | `utf8` | (3) |
+ /// | `xdbc_scope_catalog` | `utf8` | (3) |
+ /// | `xdbc_scope_schema` | `utf8` | (3) |
+ /// | `xdbc_scope_table` | `utf8` | (3) |
+ /// | `xdbc_is_autoincrement` | `bool` | (3) |
+ /// | `xdbc_is_generatedcolumn` | `bool` | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// xdbc_ values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// `CONSTRAINT_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `constraint_name` | `utf8` | |
+ /// | `constraint_type` | `utf8 not null` | (1) |
+ /// | `constraint_column_names` | `list<utf8> not null` | (2) |
+ /// | `constraint_column_usage` | `list<USAGE_SCHEMA>` | (3) |
+ ///
+ /// 1. One of 'CHECK', 'FOREIGN KEY', 'PRIMARY KEY', or 'UNIQUE'.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For FOREIGN KEY only, the referenced table and columns.
+ ///
+ /// `USAGE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|-------------------------|
+ /// | `fk_catalog` | `utf8` |
+ /// | `fk_db_schema` | `utf8` |
+ /// | `fk_table` | `utf8 not null` |
+ /// | `fk_column_name` | `utf8 not null` |
+ ///
+ /// # Parameters
+ ///
+ /// * **depth**: The level of nesting to display. If [AdbcObjectDepth::All], display
+ /// all levels. If [AdbcObjectDepth::Catalogs], display only catalogs (i.e. `catalog_schemas`
+ /// will be null). If [AdbcObjectDepth::DBSchemas], display only catalogs and schemas
+ /// (i.e. `db_schema_tables` will be null), and so on.
+ /// * **catalog**: Only show tables in the given catalog. If None,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern (see next section).
+ /// * **db_schema**: Only show tables in the given database schema. If
+ /// None, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern (see next section).
+ /// * **table_name**: Only show tables with the given name. If None, do not
+ /// filter by name. May be a search pattern (see next section).
+ /// * **table_type**: Only show tables matching one of the given table
+ /// types. If None, show tables of any type. Valid table types should
+ /// match those returned by [AdbcConnection::get_table_schema].
+ /// * **column_name**: Only show columns with the given name. If
+ /// None, do not filter by name. May be a search pattern (see next section).
+ ///
+ /// # Search patterns
+ ///
+ /// Some parameters accept "search patterns", which are
+ /// strings that can contain the special character `"%"` to match zero
+ /// or more characters, or `"_"` to match exactly one character. (See
+ /// the documentation of DatabaseMetaData in JDBC or "Pattern Value
+ /// Arguments" in the ODBC documentation.)
+ fn get_objects(
+ &self,
+ depth: AdbcObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: &[&str],
+ column_name: Option<&str>,
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// `catalog` or `db_schema` may be `None` when not applicable.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema, Self::Error>;
+
+ // TODO: should this just return &[&str]?
+ /// Get a list of table types in the database.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// -----------------|--------------
+ /// `table_type` | `utf8 not null`
+ fn get_table_types(&self) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
Review Comment:
Yeah I'm thinking of having the Rust trait users implement with a return type of `impl Iterator<Item=&str>` instead, and then build the Arrow array and stream for them in the FFI adapter layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 commented on pull request #3582: Start new ADBC crate
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1408946429
> Can you confirm that the usecase of this project (adbc in arrow-rs) is to make it easier for users to implement (in Rust) the ADBC interface for a specific database)?
There are two use cases for this project, but this PR will only add the first:
1. Implement ADBC drivers as an interface for querying a particular database.
2. (Not for this PR) Provide a driver manager, that allows loading and using an ADBC driver. ADBC drivers can either be initialized from a specific function (if statically linking to a particular driver) or by dynamically loading the init function (if dynamically loading drivers).
I am starting with the first since that's my use case, but will probably add the second in a follow-up soon because it makes testing a lot easier. 😄
> It seems this is likely the most useful for databases that were already implemented in Rust and thus their "native" client drivers and codebase are written in Rust as well (for example GreptimeDB, IOx, and Ballista).
Yes, but except for in-memory ones, there's not a hard rule that you have to write the client in the same language as the server. The main requirement is that the language can produce dynamic libraries, which I think excludes Java for example. The recommended stack for an Arrow-native database is use ADBC as the client-facing API, and the client-server communication in FlightSQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] lidavidm commented on pull request #3582: Start new ADBC crate
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1408864007
Yes, that's the intent. (Ideally we should let you consume ADBC from Rust as well. But I'm not up to speed with the state of dynamic linking and such in Rust.) I believe Will's immediate use case is an [ADBC driver for Delta Lake](https://lists.apache.org/thread/g50jvvqgy1y6vvh8d8dbg2n2jgs7hz07), which would take advantage of delta-rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 commented on pull request #3582: Start new ADBC crate
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1409654463
> Thanks @wjones127 -- do you have a specific database / usecase in mind? I ask because I still don't fully undertand what advantage that a Rust implementation of ADBC would have over the C/C++ implementation (except for when the native client was already written in Rust)
My particular use case is somewhat different than the primary one, as it's for a storage layer rather than a full database. My goal is to implement an ADBC driver for Delta Lake, allowing the [delta-rs](https://github.com/delta-io/delta-rs) project to be used in a greater variety of contexts. ([Design doc here](https://docs.google.com/document/d/1ud-iBPg8VVz2N3HxySz9qbrffw6a9I7TiGZJ2MBs7ZE/edit#heading=h.6h2fefkepofr)) So you're right that in my use case, it's mostly that the library it's integrating with is already written in Rust.
Although I suspect there are users who may wish to write a driver in Rust over C/C++ purely out of desire for stronger memory-safety guarantees. (As someone whose day job involves debugging memory bugs in C++ code, I sure feel that way 😅.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #3582: Start new ADBC crate
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1403639716
This looks awesome -I plan to check it out over the next few days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 commented on a diff in pull request #3582: Start new ADBC crate
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#discussion_r1084407045
##########
arrow-adbc/src/interface/mod.rs:
##########
@@ -0,0 +1,402 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implement ADBC driver without unsafe code.
+//!
+//! Allows implementing ADBC driver in Rust without directly interacting with
+//! FFI types, which requires unsafe code. Implement [AdbcDatabase],
+//! [AdbcConnection], [AdbcStatement], and [AdbcError] first. Then pass the
+//! statement type to [adbc_api] to generate C FFI interface for ADBC.
+
+use std::{rc::Rc, sync::Arc};
+
+use arrow::{array::ArrayRef, datatypes::Schema, record_batch::RecordBatchReader};
+
+use crate::{error::AdbcError, ffi::AdbcObjectDepth};
+
+pub mod internal;
+
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Because it is shared by multiple connections, the implementation must be
+/// thread safe. Internally, it is held with an [std::sync::Arc] by each connection.
+pub trait AdbcDatabase {
+ type Error: AdbcError;
+
+ /// Set an option on the database.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the database.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self) -> Result<(), Self::Error>;
+}
+
+/// A connection is a single connection to a database.
+///
+/// It is never accessed concurrently from multiple threads.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting `"adbc.connection.autocommit"` to `"false"` (using
+/// [AdbcConnection::set_option]). Turning off autocommit allows customizing
+/// the isolation level. Read more in [adbc.h](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+pub trait AdbcConnection {
+ type Error: AdbcError;
+ type DatabaseType: AdbcDatabase + Default;
+
+ /// Set an option on the connection.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the connection.
+ ///
+ /// The Arc to the database should be stored within your struct.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self, database: Arc<Self::DatabaseType>) -> Result<(), Self::Error>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// `info_name` | `uint32 not null`
+ /// `info_value` | `INFO_SCHEMA`
+ ///
+ /// `INFO_SCHEMA` is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ------------------------------|------------------------
+ /// `string_value` (0) | `utf8`
+ /// `bool_value` (1) | `bool`
+ /// `int64_value` (2) | `int64`
+ /// `int32_bitmask` (3) | `int32`
+ /// `string_list` (4) | `list<utf8>`
+ /// `int32_to_int32_list_map` (5) | `map<int32, list<int32>>`
+ ///
+ /// Each metadatum is identified by an integer code. The recognized
+ /// codes are defined as constants. Codes [0, 10_000) are reserved
+ /// for ADBC usage. Drivers/vendors will ignore requests for
+ /// unrecognized codes (the row will be omitted from the result).
+ ///
+ /// For definitions of known ADBC codes, see <https://github.com/apache/arrow-adbc/blob/main/adbc.h>
+ fn get_info(
+ &self,
+ info_codes: &[u32],
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and columns.
+ ///
+ /// # Schema
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `catalog_name` | `utf8` |
+ /// | `catalog_db_schemas` | `list<DB_SCHEMA_SCHEMA>` |
+ ///
+ /// `DB_SCHEMA_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `db_schema_name` | `utf8` |
+ /// | `db_schema_tables` | `list<TABLE_SCHEMA>` |
+ ///
+ /// `TABLE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `table_name` | `utf8 not null` |
+ /// | `table_type` | `utf8 not null` |
+ /// | `table_columns` | `list<COLUMN_SCHEMA>` |
+ /// | `table_constraints` | `list<CONSTRAINT_SCHEMA>` |
+ ///
+ /// `COLUMN_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `column_name` | `utf8 not null` | |
+ /// | `ordinal_position` | `int32` | (1) |
+ /// | `remarks` | `utf8` | (2) |
+ /// | `xdbc_data_type` | `int16` | (3) |
+ /// | `xdbc_type_name` | `utf8` | (3) |
+ /// | `xdbc_column_size` | `int32` | (3) |
+ /// | `xdbc_decimal_digits` | `int16` | (3) |
+ /// | `xdbc_num_prec_radix` | `int16` | (3) |
+ /// | `xdbc_nullable` | `int16` | (3) |
+ /// | `xdbc_column_def` | `utf8` | (3) |
+ /// | `xdbc_sql_data_type` | `int16` | (3) |
+ /// | `xdbc_datetime_sub` | `int16` | (3) |
+ /// | `xdbc_char_octet_length` | `int32` | (3) |
+ /// | `xdbc_is_nullable` | `utf8` | (3) |
+ /// | `xdbc_scope_catalog` | `utf8` | (3) |
+ /// | `xdbc_scope_schema` | `utf8` | (3) |
+ /// | `xdbc_scope_table` | `utf8` | (3) |
+ /// | `xdbc_is_autoincrement` | `bool` | (3) |
+ /// | `xdbc_is_generatedcolumn` | `bool` | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// xdbc_ values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// `CONSTRAINT_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `constraint_name` | `utf8` | |
+ /// | `constraint_type` | `utf8 not null` | (1) |
+ /// | `constraint_column_names` | `list<utf8> not null` | (2) |
+ /// | `constraint_column_usage` | `list<USAGE_SCHEMA>` | (3) |
+ ///
+ /// 1. One of 'CHECK', 'FOREIGN KEY', 'PRIMARY KEY', or 'UNIQUE'.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For FOREIGN KEY only, the referenced table and columns.
+ ///
+ /// `USAGE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|-------------------------|
+ /// | `fk_catalog` | `utf8` |
+ /// | `fk_db_schema` | `utf8` |
+ /// | `fk_table` | `utf8 not null` |
+ /// | `fk_column_name` | `utf8 not null` |
+ ///
+ /// # Parameters
+ ///
+ /// * **depth**: The level of nesting to display. If [AdbcObjectDepth::All], display
+ /// all levels. If [AdbcObjectDepth::Catalogs], display only catalogs (i.e. `catalog_schemas`
+ /// will be null). If [AdbcObjectDepth::DBSchemas], display only catalogs and schemas
+ /// (i.e. `db_schema_tables` will be null), and so on.
+ /// * **catalog**: Only show tables in the given catalog. If None,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern (see next section).
+ /// * **db_schema**: Only show tables in the given database schema. If
+ /// None, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern (see next section).
+ /// * **table_name**: Only show tables with the given name. If None, do not
+ /// filter by name. May be a search pattern (see next section).
+ /// * **table_type**: Only show tables matching one of the given table
+ /// types. If None, show tables of any type. Valid table types should
+ /// match those returned by [AdbcConnection::get_table_schema].
+ /// * **column_name**: Only show columns with the given name. If
+ /// None, do not filter by name. May be a search pattern (see next section).
+ ///
+ /// # Search patterns
+ ///
+ /// Some parameters accept "search patterns", which are
+ /// strings that can contain the special character `"%"` to match zero
+ /// or more characters, or `"_"` to match exactly one character. (See
+ /// the documentation of DatabaseMetaData in JDBC or "Pattern Value
+ /// Arguments" in the ODBC documentation.)
+ fn get_objects(
+ &self,
+ depth: AdbcObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: &[&str],
+ column_name: Option<&str>,
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// `catalog` or `db_schema` may be `None` when not applicable.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema, Self::Error>;
+
+ // TODO: should this just return &[&str]?
+ /// Get a list of table types in the database.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// -----------------|--------------
+ /// `table_type` | `utf8 not null`
+ fn get_table_types(&self) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
Review Comment:
Yeah I'm thinking of having the Rust trait users implement just return `impl Iterator<Item=&str>` instead, and then build the Arrow array and stream for them in the FFI adapter layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 commented on a diff in pull request #3582: Start new ADBC crate
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#discussion_r1083557467
##########
arrow-adbc/src/interface/mod.rs:
##########
@@ -0,0 +1,402 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implement ADBC driver without unsafe code.
+//!
+//! Allows implementing ADBC driver in Rust without directly interacting with
+//! FFI types, which requires unsafe code. Implement [AdbcDatabase],
+//! [AdbcConnection], [AdbcStatement], and [AdbcError] first. Then pass the
+//! statement type to [adbc_api] to generate C FFI interface for ADBC.
+
+use std::{rc::Rc, sync::Arc};
+
+use arrow::{array::ArrayRef, datatypes::Schema, record_batch::RecordBatchReader};
+
+use crate::{error::AdbcError, ffi::AdbcObjectDepth};
+
+pub mod internal;
+
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Because it is shared by multiple connections, the implementation must be
+/// thread safe. Internally, it is held with an [std::sync::Arc] by each connection.
+pub trait AdbcDatabase {
+ type Error: AdbcError;
+
+ /// Set an option on the database.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the database.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self) -> Result<(), Self::Error>;
+}
+
+/// A connection is a single connection to a database.
+///
+/// It is never accessed concurrently from multiple threads.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting `"adbc.connection.autocommit"` to `"false"` (using
+/// [AdbcConnection::set_option]). Turning off autocommit allows customizing
+/// the isolation level. Read more in [adbc.h](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+pub trait AdbcConnection {
+ type Error: AdbcError;
+ type DatabaseType: AdbcDatabase + Default;
+
+ /// Set an option on the connection.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the connection.
+ ///
+ /// The Arc to the database should be stored within your struct.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self, database: Arc<Self::DatabaseType>) -> Result<(), Self::Error>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// `info_name` | `uint32 not null`
+ /// `info_value` | `INFO_SCHEMA`
+ ///
+ /// `INFO_SCHEMA` is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ------------------------------|------------------------
+ /// `string_value` (0) | `utf8`
+ /// `bool_value` (1) | `bool`
+ /// `int64_value` (2) | `int64`
+ /// `int32_bitmask` (3) | `int32`
+ /// `string_list` (4) | `list<utf8>`
+ /// `int32_to_int32_list_map` (5) | `map<int32, list<int32>>`
+ ///
+ /// Each metadatum is identified by an integer code. The recognized
+ /// codes are defined as constants. Codes [0, 10_000) are reserved
+ /// for ADBC usage. Drivers/vendors will ignore requests for
+ /// unrecognized codes (the row will be omitted from the result).
+ ///
+ /// For definitions of known ADBC codes, see <https://github.com/apache/arrow-adbc/blob/main/adbc.h>
+ fn get_info(
+ &self,
+ info_codes: &[u32],
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and columns.
+ ///
+ /// # Schema
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `catalog_name` | `utf8` |
+ /// | `catalog_db_schemas` | `list<DB_SCHEMA_SCHEMA>` |
+ ///
+ /// `DB_SCHEMA_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `db_schema_name` | `utf8` |
+ /// | `db_schema_tables` | `list<TABLE_SCHEMA>` |
+ ///
+ /// `TABLE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `table_name` | `utf8 not null` |
+ /// | `table_type` | `utf8 not null` |
+ /// | `table_columns` | `list<COLUMN_SCHEMA>` |
+ /// | `table_constraints` | `list<CONSTRAINT_SCHEMA>` |
+ ///
+ /// `COLUMN_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `column_name` | `utf8 not null` | |
+ /// | `ordinal_position` | `int32` | (1) |
+ /// | `remarks` | `utf8` | (2) |
+ /// | `xdbc_data_type` | `int16` | (3) |
+ /// | `xdbc_type_name` | `utf8` | (3) |
+ /// | `xdbc_column_size` | `int32` | (3) |
+ /// | `xdbc_decimal_digits` | `int16` | (3) |
+ /// | `xdbc_num_prec_radix` | `int16` | (3) |
+ /// | `xdbc_nullable` | `int16` | (3) |
+ /// | `xdbc_column_def` | `utf8` | (3) |
+ /// | `xdbc_sql_data_type` | `int16` | (3) |
+ /// | `xdbc_datetime_sub` | `int16` | (3) |
+ /// | `xdbc_char_octet_length` | `int32` | (3) |
+ /// | `xdbc_is_nullable` | `utf8` | (3) |
+ /// | `xdbc_scope_catalog` | `utf8` | (3) |
+ /// | `xdbc_scope_schema` | `utf8` | (3) |
+ /// | `xdbc_scope_table` | `utf8` | (3) |
+ /// | `xdbc_is_autoincrement` | `bool` | (3) |
+ /// | `xdbc_is_generatedcolumn` | `bool` | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// xdbc_ values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// `CONSTRAINT_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `constraint_name` | `utf8` | |
+ /// | `constraint_type` | `utf8 not null` | (1) |
+ /// | `constraint_column_names` | `list<utf8> not null` | (2) |
+ /// | `constraint_column_usage` | `list<USAGE_SCHEMA>` | (3) |
+ ///
+ /// 1. One of 'CHECK', 'FOREIGN KEY', 'PRIMARY KEY', or 'UNIQUE'.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For FOREIGN KEY only, the referenced table and columns.
+ ///
+ /// `USAGE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|-------------------------|
+ /// | `fk_catalog` | `utf8` |
+ /// | `fk_db_schema` | `utf8` |
+ /// | `fk_table` | `utf8 not null` |
+ /// | `fk_column_name` | `utf8 not null` |
+ ///
+ /// # Parameters
+ ///
+ /// * **depth**: The level of nesting to display. If [AdbcObjectDepth::All], display
+ /// all levels. If [AdbcObjectDepth::Catalogs], display only catalogs (i.e. `catalog_schemas`
+ /// will be null). If [AdbcObjectDepth::DBSchemas], display only catalogs and schemas
+ /// (i.e. `db_schema_tables` will be null), and so on.
+ /// * **catalog**: Only show tables in the given catalog. If None,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern (see next section).
+ /// * **db_schema**: Only show tables in the given database schema. If
+ /// None, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern (see next section).
+ /// * **table_name**: Only show tables with the given name. If None, do not
+ /// filter by name. May be a search pattern (see next section).
+ /// * **table_type**: Only show tables matching one of the given table
+ /// types. If None, show tables of any type. Valid table types should
+ /// match those returned by [AdbcConnection::get_table_schema].
+ /// * **column_name**: Only show columns with the given name. If
+ /// None, do not filter by name. May be a search pattern (see next section).
+ ///
+ /// # Search patterns
+ ///
+ /// Some parameters accept "search patterns", which are
+ /// strings that can contain the special character `"%"` to match zero
+ /// or more characters, or `"_"` to match exactly one character. (See
+ /// the documentation of DatabaseMetaData in JDBC or "Pattern Value
+ /// Arguments" in the ODBC documentation.)
+ fn get_objects(
+ &self,
+ depth: AdbcObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: &[&str],
+ column_name: Option<&str>,
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// `catalog` or `db_schema` may be `None` when not applicable.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema, Self::Error>;
+
+ // TODO: should this just return &[&str]?
+ /// Get a list of table types in the database.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// -----------------|--------------
+ /// `table_type` | `utf8 not null`
+ fn get_table_types(&self) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
Review Comment:
@lidavidm Are there situations where a user might want to implement this as a stream of data? Or is it always just a fixed set of strings?
##########
arrow-adbc/src/interface/mod.rs:
##########
@@ -0,0 +1,402 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implement ADBC driver without unsafe code.
+//!
+//! Allows implementing ADBC driver in Rust without directly interacting with
+//! FFI types, which requires unsafe code. Implement [AdbcDatabase],
+//! [AdbcConnection], [AdbcStatement], and [AdbcError] first. Then pass the
+//! statement type to [adbc_api] to generate C FFI interface for ADBC.
+
+use std::{rc::Rc, sync::Arc};
+
+use arrow::{array::ArrayRef, datatypes::Schema, record_batch::RecordBatchReader};
+
+use crate::{error::AdbcError, ffi::AdbcObjectDepth};
+
+pub mod internal;
+
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Because it is shared by multiple connections, the implementation must be
+/// thread safe. Internally, it is held with an [std::sync::Arc] by each connection.
+pub trait AdbcDatabase {
+ type Error: AdbcError;
+
+ /// Set an option on the database.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the database.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self) -> Result<(), Self::Error>;
+}
+
+/// A connection is a single connection to a database.
+///
+/// It is never accessed concurrently from multiple threads.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting `"adbc.connection.autocommit"` to `"false"` (using
+/// [AdbcConnection::set_option]). Turning off autocommit allows customizing
+/// the isolation level. Read more in [adbc.h](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+pub trait AdbcConnection {
+ type Error: AdbcError;
+ type DatabaseType: AdbcDatabase + Default;
+
+ /// Set an option on the connection.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the connection.
+ ///
+ /// The Arc to the database should be stored within your struct.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self, database: Arc<Self::DatabaseType>) -> Result<(), Self::Error>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// `info_name` | `uint32 not null`
+ /// `info_value` | `INFO_SCHEMA`
+ ///
+ /// `INFO_SCHEMA` is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ------------------------------|------------------------
+ /// `string_value` (0) | `utf8`
+ /// `bool_value` (1) | `bool`
+ /// `int64_value` (2) | `int64`
+ /// `int32_bitmask` (3) | `int32`
+ /// `string_list` (4) | `list<utf8>`
+ /// `int32_to_int32_list_map` (5) | `map<int32, list<int32>>`
+ ///
+ /// Each metadatum is identified by an integer code. The recognized
+ /// codes are defined as constants. Codes [0, 10_000) are reserved
+ /// for ADBC usage. Drivers/vendors will ignore requests for
+ /// unrecognized codes (the row will be omitted from the result).
+ ///
+ /// For definitions of known ADBC codes, see <https://github.com/apache/arrow-adbc/blob/main/adbc.h>
+ fn get_info(
+ &self,
+ info_codes: &[u32],
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and columns.
+ ///
+ /// # Schema
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `catalog_name` | `utf8` |
+ /// | `catalog_db_schemas` | `list<DB_SCHEMA_SCHEMA>` |
+ ///
+ /// `DB_SCHEMA_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `db_schema_name` | `utf8` |
+ /// | `db_schema_tables` | `list<TABLE_SCHEMA>` |
+ ///
+ /// `TABLE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `table_name` | `utf8 not null` |
+ /// | `table_type` | `utf8 not null` |
+ /// | `table_columns` | `list<COLUMN_SCHEMA>` |
+ /// | `table_constraints` | `list<CONSTRAINT_SCHEMA>` |
+ ///
+ /// `COLUMN_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `column_name` | `utf8 not null` | |
+ /// | `ordinal_position` | `int32` | (1) |
+ /// | `remarks` | `utf8` | (2) |
+ /// | `xdbc_data_type` | `int16` | (3) |
+ /// | `xdbc_type_name` | `utf8` | (3) |
+ /// | `xdbc_column_size` | `int32` | (3) |
+ /// | `xdbc_decimal_digits` | `int16` | (3) |
+ /// | `xdbc_num_prec_radix` | `int16` | (3) |
+ /// | `xdbc_nullable` | `int16` | (3) |
+ /// | `xdbc_column_def` | `utf8` | (3) |
+ /// | `xdbc_sql_data_type` | `int16` | (3) |
+ /// | `xdbc_datetime_sub` | `int16` | (3) |
+ /// | `xdbc_char_octet_length` | `int32` | (3) |
+ /// | `xdbc_is_nullable` | `utf8` | (3) |
+ /// | `xdbc_scope_catalog` | `utf8` | (3) |
+ /// | `xdbc_scope_schema` | `utf8` | (3) |
+ /// | `xdbc_scope_table` | `utf8` | (3) |
+ /// | `xdbc_is_autoincrement` | `bool` | (3) |
+ /// | `xdbc_is_generatedcolumn` | `bool` | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// xdbc_ values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// `CONSTRAINT_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `constraint_name` | `utf8` | |
+ /// | `constraint_type` | `utf8 not null` | (1) |
+ /// | `constraint_column_names` | `list<utf8> not null` | (2) |
+ /// | `constraint_column_usage` | `list<USAGE_SCHEMA>` | (3) |
+ ///
+ /// 1. One of 'CHECK', 'FOREIGN KEY', 'PRIMARY KEY', or 'UNIQUE'.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For FOREIGN KEY only, the referenced table and columns.
+ ///
+ /// `USAGE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|-------------------------|
+ /// | `fk_catalog` | `utf8` |
+ /// | `fk_db_schema` | `utf8` |
+ /// | `fk_table` | `utf8 not null` |
+ /// | `fk_column_name` | `utf8 not null` |
+ ///
+ /// # Parameters
+ ///
+ /// * **depth**: The level of nesting to display. If [AdbcObjectDepth::All], display
+ /// all levels. If [AdbcObjectDepth::Catalogs], display only catalogs (i.e. `catalog_schemas`
+ /// will be null). If [AdbcObjectDepth::DBSchemas], display only catalogs and schemas
+ /// (i.e. `db_schema_tables` will be null), and so on.
+ /// * **catalog**: Only show tables in the given catalog. If None,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern (see next section).
+ /// * **db_schema**: Only show tables in the given database schema. If
+ /// None, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern (see next section).
+ /// * **table_name**: Only show tables with the given name. If None, do not
+ /// filter by name. May be a search pattern (see next section).
+ /// * **table_type**: Only show tables matching one of the given table
+ /// types. If None, show tables of any type. Valid table types should
+ /// match those returned by [AdbcConnection::get_table_schema].
+ /// * **column_name**: Only show columns with the given name. If
+ /// None, do not filter by name. May be a search pattern (see next section).
+ ///
+ /// # Search patterns
+ ///
+ /// Some parameters accept "search patterns", which are
+ /// strings that can contain the special character `"%"` to match zero
+ /// or more characters, or `"_"` to match exactly one character. (See
+ /// the documentation of DatabaseMetaData in JDBC or "Pattern Value
+ /// Arguments" in the ODBC documentation.)
+ fn get_objects(
+ &self,
+ depth: AdbcObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: &[&str],
+ column_name: Option<&str>,
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// `catalog` or `db_schema` may be `None` when not applicable.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema, Self::Error>;
+
+ // TODO: should this just return &[&str]?
+ /// Get a list of table types in the database.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// -----------------|--------------
+ /// `table_type` | `utf8 not null`
+ fn get_table_types(&self) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ // TODO: if Connection is only meant to be access from a single thread, does
+ // that mean these partitions should be accessible across connections? Or
+ // That read_partition should be called on the main thread and then the
+ // RBR are then passed to other threads?
+ /// Read part of a partitioned result set.
+ fn read_partition(
+ &self,
+ partition: &[u8],
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ fn commit(&self) -> Result<(), Self::Error>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ fn rollback(&self) -> Result<(), Self::Error>;
+}
+
+/// A container for all state needed to execute a database query, such as the
+/// query itself, parameters for prepared statements, driver parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used
+/// concurrently (whether from a single thread or multiple threads).
+pub trait AdbcStatement {
+ type Error: AdbcError;
+ type ConnectionType: AdbcConnection + Default;
+
+ /// Create a new statement.
+ ///
+ /// The conn should be saved within the struct.
+ fn new_from_connection(conn: Rc<Self::ConnectionType>) -> Self;
+
+ /// Turn this statement into a prepared statement to be executed multiple times.
+ ///
+ /// This should return an error if called before [AdbcStatement::set_sql_query].
+ fn prepare(&self) -> Result<(), Self::Error>;
+
+ /// Set a string option on a statement.
+ fn set_option(&mut self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Set the SQL query to execute.
+ fn set_sql_query(&mut self, query: &str) -> Result<(), Self::Error>;
+
+ /// Set the Substrait plan to execute.
+ fn set_substrait_plan(&mut self, plan: &[u8]) -> Result<(), Self::Error>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should return an error if this was called before [AdbcStatement::prepare].
+ fn get_param_schema(&self) -> Result<Schema, Self::Error>;
+
+ /// Bind Arrow data, either for bulk inserts or prepared statements.
+ fn bind_data(&mut self, arr: ArrayRef) -> Result<(), Self::Error>;
+
+ /// Bind Arrow data, either for bulk inserts or prepared statements.
+ fn bind_stream(
+ &mut self,
+ stream: Box<dyn RecordBatchReader>,
+ ) -> Result<(), Self::Error>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// See [StatementResult].
+ fn execute(&self) -> Result<StatementResult, Self::Error>;
+
+ /// Execute a statement with a partitioned result set.
+ ///
+ /// This is not required to be implemented, as it only applies to backends
+ /// that internally partition results. These backends can use this method
+ /// to support threaded or distributed clients.
+ ///
+ /// See [PartitionedStatementResult].
+ fn execute_partitioned(&self) -> Result<PartitionedStatementResult, Self::Error>;
+}
+
+/// Result of calling [AdbcStatement::execute].
+///
+/// `result` may be None if there is no meaningful result.
+/// `row_affected` may be None if not applicable or if it is not supported.
+pub struct StatementResult {
+ pub result: Option<Box<dyn RecordBatchReader>>,
+ pub rows_affected: Option<i64>,
+}
+
+/// Partitioned results
+///
+/// [AdbcConnection::read_partition] will be called to get the output stream
+/// for each partition.
+///
+/// These may be used by a multi-threaded or a distributed client. Each partition
+/// will be retrieved by a separate connection. For in-memory databases, these
+/// may be connections on different threads that all reference the same database.
+/// For remote databases, these may be connections in different processes.
+#[derive(Debug, Clone)]
+pub struct PartitionedStatementResult {
+ pub schema: Schema,
+ pub partition_ids: Vec<Vec<u8>>,
+ pub rows_affected: Option<i64>,
+}
+
+/// Expose an ADBC driver entrypoint for the given statement type.
+///
+/// The type must implement [AdbcStatement].
+#[macro_export]
+macro_rules! adbc_api {
+ ($statement_type:ident) => {
+ mod _adbc_api {
+ use super::$statement_type;
+ use arrow_adbc::error::{AdbcStatusCode, FFI_AdbcError};
+ use arrow_adbc::ffi::FFI_AdbcDriver;
+
+ // TODO: Should this not be mangled? Is there a risk of symbol name collisions?
+ // No mangle, since this is the main entrypoint.
+ #[no_mangle]
+ pub extern "C" fn AdbcDriverInit(
Review Comment:
@lidavidm Perhaps a dumb question, but should this name be mangled or prefixed with something? Or is there not a risk of any symbol name collision?
https://github.com/apache/arrow-adbc/blob/5029f7b8ed1e60dd7d41f175af0abee6ccadb02e/adbc.h#L1185-L1186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] lidavidm commented on a diff in pull request #3582: Start new ADBC crate
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#discussion_r1083566482
##########
arrow-adbc/src/interface/mod.rs:
##########
@@ -0,0 +1,402 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implement ADBC driver without unsafe code.
+//!
+//! Allows implementing ADBC driver in Rust without directly interacting with
+//! FFI types, which requires unsafe code. Implement [AdbcDatabase],
+//! [AdbcConnection], [AdbcStatement], and [AdbcError] first. Then pass the
+//! statement type to [adbc_api] to generate C FFI interface for ADBC.
+
+use std::{rc::Rc, sync::Arc};
+
+use arrow::{array::ArrayRef, datatypes::Schema, record_batch::RecordBatchReader};
+
+use crate::{error::AdbcError, ffi::AdbcObjectDepth};
+
+pub mod internal;
+
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Because it is shared by multiple connections, the implementation must be
+/// thread safe. Internally, it is held with an [std::sync::Arc] by each connection.
+pub trait AdbcDatabase {
+ type Error: AdbcError;
+
+ /// Set an option on the database.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the database.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self) -> Result<(), Self::Error>;
+}
+
+/// A connection is a single connection to a database.
+///
+/// It is never accessed concurrently from multiple threads.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting `"adbc.connection.autocommit"` to `"false"` (using
+/// [AdbcConnection::set_option]). Turning off autocommit allows customizing
+/// the isolation level. Read more in [adbc.h](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+pub trait AdbcConnection {
+ type Error: AdbcError;
+ type DatabaseType: AdbcDatabase + Default;
+
+ /// Set an option on the connection.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the connection.
+ ///
+ /// The Arc to the database should be stored within your struct.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self, database: Arc<Self::DatabaseType>) -> Result<(), Self::Error>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// `info_name` | `uint32 not null`
+ /// `info_value` | `INFO_SCHEMA`
+ ///
+ /// `INFO_SCHEMA` is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ------------------------------|------------------------
+ /// `string_value` (0) | `utf8`
+ /// `bool_value` (1) | `bool`
+ /// `int64_value` (2) | `int64`
+ /// `int32_bitmask` (3) | `int32`
+ /// `string_list` (4) | `list<utf8>`
+ /// `int32_to_int32_list_map` (5) | `map<int32, list<int32>>`
+ ///
+ /// Each metadatum is identified by an integer code. The recognized
+ /// codes are defined as constants. Codes [0, 10_000) are reserved
+ /// for ADBC usage. Drivers/vendors will ignore requests for
+ /// unrecognized codes (the row will be omitted from the result).
+ ///
+ /// For definitions of known ADBC codes, see <https://github.com/apache/arrow-adbc/blob/main/adbc.h>
+ fn get_info(
+ &self,
+ info_codes: &[u32],
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and columns.
+ ///
+ /// # Schema
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `catalog_name` | `utf8` |
+ /// | `catalog_db_schemas` | `list<DB_SCHEMA_SCHEMA>` |
+ ///
+ /// `DB_SCHEMA_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `db_schema_name` | `utf8` |
+ /// | `db_schema_tables` | `list<TABLE_SCHEMA>` |
+ ///
+ /// `TABLE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `table_name` | `utf8 not null` |
+ /// | `table_type` | `utf8 not null` |
+ /// | `table_columns` | `list<COLUMN_SCHEMA>` |
+ /// | `table_constraints` | `list<CONSTRAINT_SCHEMA>` |
+ ///
+ /// `COLUMN_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `column_name` | `utf8 not null` | |
+ /// | `ordinal_position` | `int32` | (1) |
+ /// | `remarks` | `utf8` | (2) |
+ /// | `xdbc_data_type` | `int16` | (3) |
+ /// | `xdbc_type_name` | `utf8` | (3) |
+ /// | `xdbc_column_size` | `int32` | (3) |
+ /// | `xdbc_decimal_digits` | `int16` | (3) |
+ /// | `xdbc_num_prec_radix` | `int16` | (3) |
+ /// | `xdbc_nullable` | `int16` | (3) |
+ /// | `xdbc_column_def` | `utf8` | (3) |
+ /// | `xdbc_sql_data_type` | `int16` | (3) |
+ /// | `xdbc_datetime_sub` | `int16` | (3) |
+ /// | `xdbc_char_octet_length` | `int32` | (3) |
+ /// | `xdbc_is_nullable` | `utf8` | (3) |
+ /// | `xdbc_scope_catalog` | `utf8` | (3) |
+ /// | `xdbc_scope_schema` | `utf8` | (3) |
+ /// | `xdbc_scope_table` | `utf8` | (3) |
+ /// | `xdbc_is_autoincrement` | `bool` | (3) |
+ /// | `xdbc_is_generatedcolumn` | `bool` | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// xdbc_ values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// `CONSTRAINT_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `constraint_name` | `utf8` | |
+ /// | `constraint_type` | `utf8 not null` | (1) |
+ /// | `constraint_column_names` | `list<utf8> not null` | (2) |
+ /// | `constraint_column_usage` | `list<USAGE_SCHEMA>` | (3) |
+ ///
+ /// 1. One of 'CHECK', 'FOREIGN KEY', 'PRIMARY KEY', or 'UNIQUE'.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For FOREIGN KEY only, the referenced table and columns.
+ ///
+ /// `USAGE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|-------------------------|
+ /// | `fk_catalog` | `utf8` |
+ /// | `fk_db_schema` | `utf8` |
+ /// | `fk_table` | `utf8 not null` |
+ /// | `fk_column_name` | `utf8 not null` |
+ ///
+ /// # Parameters
+ ///
+ /// * **depth**: The level of nesting to display. If [AdbcObjectDepth::All], display
+ /// all levels. If [AdbcObjectDepth::Catalogs], display only catalogs (i.e. `catalog_schemas`
+ /// will be null). If [AdbcObjectDepth::DBSchemas], display only catalogs and schemas
+ /// (i.e. `db_schema_tables` will be null), and so on.
+ /// * **catalog**: Only show tables in the given catalog. If None,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern (see next section).
+ /// * **db_schema**: Only show tables in the given database schema. If
+ /// None, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern (see next section).
+ /// * **table_name**: Only show tables with the given name. If None, do not
+ /// filter by name. May be a search pattern (see next section).
+ /// * **table_type**: Only show tables matching one of the given table
+ /// types. If None, show tables of any type. Valid table types should
+ /// match those returned by [AdbcConnection::get_table_schema].
+ /// * **column_name**: Only show columns with the given name. If
+ /// None, do not filter by name. May be a search pattern (see next section).
+ ///
+ /// # Search patterns
+ ///
+ /// Some parameters accept "search patterns", which are
+ /// strings that can contain the special character `"%"` to match zero
+ /// or more characters, or `"_"` to match exactly one character. (See
+ /// the documentation of DatabaseMetaData in JDBC or "Pattern Value
+ /// Arguments" in the ODBC documentation.)
+ fn get_objects(
+ &self,
+ depth: AdbcObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: &[&str],
+ column_name: Option<&str>,
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// `catalog` or `db_schema` may be `None` when not applicable.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema, Self::Error>;
+
+ // TODO: should this just return &[&str]?
+ /// Get a list of table types in the database.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// -----------------|--------------
+ /// `table_type` | `utf8 not null`
+ fn get_table_types(&self) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ // TODO: if Connection is only meant to be access from a single thread, does
+ // that mean these partitions should be accessible across connections? Or
+ // That read_partition should be called on the main thread and then the
+ // RBR are then passed to other threads?
+ /// Read part of a partitioned result set.
+ fn read_partition(
+ &self,
+ partition: &[u8],
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Commit any pending transactions. Only used if autocommit is disabled.
+ fn commit(&self) -> Result<(), Self::Error>;
+
+ /// Roll back any pending transactions. Only used if autocommit is disabled.
+ fn rollback(&self) -> Result<(), Self::Error>;
+}
+
+/// A container for all state needed to execute a database query, such as the
+/// query itself, parameters for prepared statements, driver parameters, etc.
+///
+/// Statements may represent queries or prepared statements.
+///
+/// Statements may be used multiple times and can be reconfigured
+/// (e.g. they can be reused to execute multiple different queries).
+/// However, executing a statement (and changing certain other state)
+/// will invalidate result sets obtained prior to that execution.
+///
+/// Multiple statements may be created from a single connection.
+/// However, the driver may block or error if they are used
+/// concurrently (whether from a single thread or multiple threads).
+pub trait AdbcStatement {
+ type Error: AdbcError;
+ type ConnectionType: AdbcConnection + Default;
+
+ /// Create a new statement.
+ ///
+ /// The conn should be saved within the struct.
+ fn new_from_connection(conn: Rc<Self::ConnectionType>) -> Self;
+
+ /// Turn this statement into a prepared statement to be executed multiple times.
+ ///
+ /// This should return an error if called before [AdbcStatement::set_sql_query].
+ fn prepare(&self) -> Result<(), Self::Error>;
+
+ /// Set a string option on a statement.
+ fn set_option(&mut self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Set the SQL query to execute.
+ fn set_sql_query(&mut self, query: &str) -> Result<(), Self::Error>;
+
+ /// Set the Substrait plan to execute.
+ fn set_substrait_plan(&mut self, plan: &[u8]) -> Result<(), Self::Error>;
+
+ /// Get the schema for bound parameters.
+ ///
+ /// This retrieves an Arrow schema describing the number, names, and
+ /// types of the parameters in a parameterized statement. The fields
+ /// of the schema should be in order of the ordinal position of the
+ /// parameters; named parameters should appear only once.
+ ///
+ /// If the parameter does not have a name, or the name cannot be
+ /// determined, the name of the corresponding field in the schema will
+ /// be an empty string. If the type cannot be determined, the type of
+ /// the corresponding field will be NA (NullType).
+ ///
+ /// This should return an error if this was called before [AdbcStatement::prepare].
+ fn get_param_schema(&self) -> Result<Schema, Self::Error>;
+
+ /// Bind Arrow data, either for bulk inserts or prepared statements.
+ fn bind_data(&mut self, arr: ArrayRef) -> Result<(), Self::Error>;
+
+ /// Bind Arrow data, either for bulk inserts or prepared statements.
+ fn bind_stream(
+ &mut self,
+ stream: Box<dyn RecordBatchReader>,
+ ) -> Result<(), Self::Error>;
+
+ /// Execute a statement and get the results.
+ ///
+ /// See [StatementResult].
+ fn execute(&self) -> Result<StatementResult, Self::Error>;
+
+ /// Execute a statement with a partitioned result set.
+ ///
+ /// This is not required to be implemented, as it only applies to backends
+ /// that internally partition results. These backends can use this method
+ /// to support threaded or distributed clients.
+ ///
+ /// See [PartitionedStatementResult].
+ fn execute_partitioned(&self) -> Result<PartitionedStatementResult, Self::Error>;
+}
+
+/// Result of calling [AdbcStatement::execute].
+///
+/// `result` may be None if there is no meaningful result.
+/// `row_affected` may be None if not applicable or if it is not supported.
+pub struct StatementResult {
+ pub result: Option<Box<dyn RecordBatchReader>>,
+ pub rows_affected: Option<i64>,
+}
+
+/// Partitioned results
+///
+/// [AdbcConnection::read_partition] will be called to get the output stream
+/// for each partition.
+///
+/// These may be used by a multi-threaded or a distributed client. Each partition
+/// will be retrieved by a separate connection. For in-memory databases, these
+/// may be connections on different threads that all reference the same database.
+/// For remote databases, these may be connections in different processes.
+#[derive(Debug, Clone)]
+pub struct PartitionedStatementResult {
+ pub schema: Schema,
+ pub partition_ids: Vec<Vec<u8>>,
+ pub rows_affected: Option<i64>,
+}
+
+/// Expose an ADBC driver entrypoint for the given statement type.
+///
+/// The type must implement [AdbcStatement].
+#[macro_export]
+macro_rules! adbc_api {
+ ($statement_type:ident) => {
+ mod _adbc_api {
+ use super::$statement_type;
+ use arrow_adbc::error::{AdbcStatusCode, FFI_AdbcError};
+ use arrow_adbc::ffi::FFI_AdbcDriver;
+
+ // TODO: Should this not be mangled? Is there a risk of symbol name collisions?
+ // No mangle, since this is the main entrypoint.
+ #[no_mangle]
+ pub extern "C" fn AdbcDriverInit(
Review Comment:
The assumption is that if you have multiple drivers, you will be using the driver manager; this loads them with RTLD_LOCAL, so there shouldn't be collisions. (You do still need to make sure that the driver itself doesn't call any of the Adbc* functions, since those will get resolved by the dynamic loader to the functions defined by the _driver manager_.) What all the current implementations do is provide both a name-mangled and an unmangled variant; the driver manager uses the unmangled variant for convenience, and the name-mangled one may possibly be useful for an application that wants to poke multiple drivers directly itself, or if an application links a library that also provides a driver (e.g. what libarrow was planned to do)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #3582: Start new ADBC crate
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1409600584
> > Implement ADBC drivers as an interface for querying a particular database.
> I am starting with the first since that's my use case, but will probably add the second in a follow-up soon because it makes testing a lot easier. 😄
Thanks @wjones127 -- do you have a specific database / usecase in mind? I ask because I still don't fully undertand what advantage that a Rust implementation of ADBC would have over the C/C++ implementation (except for when the native client was already written in Rust)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #3582: Start new ADBC crate
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1410209060
> My goal is to implement an ADBC driver for Delta Lake
Makes sense! Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] lidavidm commented on pull request #3582: Start new ADBC crate
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1415969386
> But I was thinking if the underlying system is already in C/C++ using Rust for the ADBC interface might not give enough extra benefit to warrant the additional set of dependencies / integration / build complexity.
FWIW, we actually just rewrote one of the drivers - the Flight SQL one - in Go instead of C++, using a wrapper to expose the C API from Go, because C++ presented distribution/packaging issues that Go mostly sidesteps.
Rust may be useful in this manner, too. For instance, maybe [ConnectorX](https://sfu-db.github.io/connector-x/intro.html) could be leveraged to provide a driver.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] lidavidm commented on a diff in pull request #3582: Start new ADBC crate
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#discussion_r1083566010
##########
arrow-adbc/src/interface/mod.rs:
##########
@@ -0,0 +1,402 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implement ADBC driver without unsafe code.
+//!
+//! Allows implementing ADBC driver in Rust without directly interacting with
+//! FFI types, which requires unsafe code. Implement [AdbcDatabase],
+//! [AdbcConnection], [AdbcStatement], and [AdbcError] first. Then pass the
+//! statement type to [adbc_api] to generate C FFI interface for ADBC.
+
+use std::{rc::Rc, sync::Arc};
+
+use arrow::{array::ArrayRef, datatypes::Schema, record_batch::RecordBatchReader};
+
+use crate::{error::AdbcError, ffi::AdbcObjectDepth};
+
+pub mod internal;
+
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Because it is shared by multiple connections, the implementation must be
+/// thread safe. Internally, it is held with an [std::sync::Arc] by each connection.
+pub trait AdbcDatabase {
+ type Error: AdbcError;
+
+ /// Set an option on the database.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the database.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self) -> Result<(), Self::Error>;
+}
+
+/// A connection is a single connection to a database.
+///
+/// It is never accessed concurrently from multiple threads.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting `"adbc.connection.autocommit"` to `"false"` (using
+/// [AdbcConnection::set_option]). Turning off autocommit allows customizing
+/// the isolation level. Read more in [adbc.h](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+pub trait AdbcConnection {
+ type Error: AdbcError;
+ type DatabaseType: AdbcDatabase + Default;
+
+ /// Set an option on the connection.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the connection.
+ ///
+ /// The Arc to the database should be stored within your struct.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self, database: Arc<Self::DatabaseType>) -> Result<(), Self::Error>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// `info_name` | `uint32 not null`
+ /// `info_value` | `INFO_SCHEMA`
+ ///
+ /// `INFO_SCHEMA` is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ------------------------------|------------------------
+ /// `string_value` (0) | `utf8`
+ /// `bool_value` (1) | `bool`
+ /// `int64_value` (2) | `int64`
+ /// `int32_bitmask` (3) | `int32`
+ /// `string_list` (4) | `list<utf8>`
+ /// `int32_to_int32_list_map` (5) | `map<int32, list<int32>>`
+ ///
+ /// Each metadatum is identified by an integer code. The recognized
+ /// codes are defined as constants. Codes [0, 10_000) are reserved
+ /// for ADBC usage. Drivers/vendors will ignore requests for
+ /// unrecognized codes (the row will be omitted from the result).
+ ///
+ /// For definitions of known ADBC codes, see <https://github.com/apache/arrow-adbc/blob/main/adbc.h>
+ fn get_info(
+ &self,
+ info_codes: &[u32],
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and columns.
+ ///
+ /// # Schema
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `catalog_name` | `utf8` |
+ /// | `catalog_db_schemas` | `list<DB_SCHEMA_SCHEMA>` |
+ ///
+ /// `DB_SCHEMA_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `db_schema_name` | `utf8` |
+ /// | `db_schema_tables` | `list<TABLE_SCHEMA>` |
+ ///
+ /// `TABLE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `table_name` | `utf8 not null` |
+ /// | `table_type` | `utf8 not null` |
+ /// | `table_columns` | `list<COLUMN_SCHEMA>` |
+ /// | `table_constraints` | `list<CONSTRAINT_SCHEMA>` |
+ ///
+ /// `COLUMN_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `column_name` | `utf8 not null` | |
+ /// | `ordinal_position` | `int32` | (1) |
+ /// | `remarks` | `utf8` | (2) |
+ /// | `xdbc_data_type` | `int16` | (3) |
+ /// | `xdbc_type_name` | `utf8` | (3) |
+ /// | `xdbc_column_size` | `int32` | (3) |
+ /// | `xdbc_decimal_digits` | `int16` | (3) |
+ /// | `xdbc_num_prec_radix` | `int16` | (3) |
+ /// | `xdbc_nullable` | `int16` | (3) |
+ /// | `xdbc_column_def` | `utf8` | (3) |
+ /// | `xdbc_sql_data_type` | `int16` | (3) |
+ /// | `xdbc_datetime_sub` | `int16` | (3) |
+ /// | `xdbc_char_octet_length` | `int32` | (3) |
+ /// | `xdbc_is_nullable` | `utf8` | (3) |
+ /// | `xdbc_scope_catalog` | `utf8` | (3) |
+ /// | `xdbc_scope_schema` | `utf8` | (3) |
+ /// | `xdbc_scope_table` | `utf8` | (3) |
+ /// | `xdbc_is_autoincrement` | `bool` | (3) |
+ /// | `xdbc_is_generatedcolumn` | `bool` | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// xdbc_ values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// `CONSTRAINT_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `constraint_name` | `utf8` | |
+ /// | `constraint_type` | `utf8 not null` | (1) |
+ /// | `constraint_column_names` | `list<utf8> not null` | (2) |
+ /// | `constraint_column_usage` | `list<USAGE_SCHEMA>` | (3) |
+ ///
+ /// 1. One of 'CHECK', 'FOREIGN KEY', 'PRIMARY KEY', or 'UNIQUE'.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For FOREIGN KEY only, the referenced table and columns.
+ ///
+ /// `USAGE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|-------------------------|
+ /// | `fk_catalog` | `utf8` |
+ /// | `fk_db_schema` | `utf8` |
+ /// | `fk_table` | `utf8 not null` |
+ /// | `fk_column_name` | `utf8 not null` |
+ ///
+ /// # Parameters
+ ///
+ /// * **depth**: The level of nesting to display. If [AdbcObjectDepth::All], display
+ /// all levels. If [AdbcObjectDepth::Catalogs], display only catalogs (i.e. `catalog_schemas`
+ /// will be null). If [AdbcObjectDepth::DBSchemas], display only catalogs and schemas
+ /// (i.e. `db_schema_tables` will be null), and so on.
+ /// * **catalog**: Only show tables in the given catalog. If None,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern (see next section).
+ /// * **db_schema**: Only show tables in the given database schema. If
+ /// None, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern (see next section).
+ /// * **table_name**: Only show tables with the given name. If None, do not
+ /// filter by name. May be a search pattern (see next section).
+ /// * **table_type**: Only show tables matching one of the given table
+ /// types. If None, show tables of any type. Valid table types should
+ /// match those returned by [AdbcConnection::get_table_schema].
+ /// * **column_name**: Only show columns with the given name. If
+ /// None, do not filter by name. May be a search pattern (see next section).
+ ///
+ /// # Search patterns
+ ///
+ /// Some parameters accept "search patterns", which are
+ /// strings that can contain the special character `"%"` to match zero
+ /// or more characters, or `"_"` to match exactly one character. (See
+ /// the documentation of DatabaseMetaData in JDBC or "Pattern Value
+ /// Arguments" in the ODBC documentation.)
+ fn get_objects(
+ &self,
+ depth: AdbcObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: &[&str],
+ column_name: Option<&str>,
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// `catalog` or `db_schema` may be `None` when not applicable.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema, Self::Error>;
+
+ // TODO: should this just return &[&str]?
+ /// Get a list of table types in the database.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// -----------------|--------------
+ /// `table_type` | `utf8 not null`
+ fn get_table_types(&self) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
Review Comment:
Basically everything is returned via a reader for consistency; this lines up with what JDBC does, too. (In retrospect, get_info and get_table_types might have been better off using native types instead of Arrow.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] lidavidm commented on a diff in pull request #3582: Start new ADBC crate
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#discussion_r1083566821
##########
arrow-adbc/src/interface/mod.rs:
##########
@@ -0,0 +1,402 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implement ADBC driver without unsafe code.
+//!
+//! Allows implementing ADBC driver in Rust without directly interacting with
+//! FFI types, which requires unsafe code. Implement [AdbcDatabase],
+//! [AdbcConnection], [AdbcStatement], and [AdbcError] first. Then pass the
+//! statement type to [adbc_api] to generate C FFI interface for ADBC.
+
+use std::{rc::Rc, sync::Arc};
+
+use arrow::{array::ArrayRef, datatypes::Schema, record_batch::RecordBatchReader};
+
+use crate::{error::AdbcError, ffi::AdbcObjectDepth};
+
+pub mod internal;
+
+/// Databases hold state shared by multiple connections. This typically means
+/// configuration and caches. For in-memory databases, it provides a place to
+/// hold ownership of the in-memory database.
+///
+/// Because it is shared by multiple connections, the implementation must be
+/// thread safe. Internally, it is held with an [std::sync::Arc] by each connection.
+pub trait AdbcDatabase {
+ type Error: AdbcError;
+
+ /// Set an option on the database.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the database.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self) -> Result<(), Self::Error>;
+}
+
+/// A connection is a single connection to a database.
+///
+/// It is never accessed concurrently from multiple threads.
+///
+/// # Autocommit
+///
+/// Connections should start in autocommit mode. They can be moved out by
+/// setting `"adbc.connection.autocommit"` to `"false"` (using
+/// [AdbcConnection::set_option]). Turning off autocommit allows customizing
+/// the isolation level. Read more in [adbc.h](https://github.com/apache/arrow-adbc/blob/main/adbc.h).
+pub trait AdbcConnection {
+ type Error: AdbcError;
+ type DatabaseType: AdbcDatabase + Default;
+
+ /// Set an option on the connection.
+ fn set_option(&self, key: &str, value: &str) -> Result<(), Self::Error>;
+
+ /// Initialize the connection.
+ ///
+ /// The Arc to the database should be stored within your struct.
+ ///
+ /// Some drivers may choose not to support setting options after this has
+ /// been called.
+ fn init(&self, database: Arc<Self::DatabaseType>) -> Result<(), Self::Error>;
+
+ /// Get metadata about the database/driver.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// ----------------------------|------------------------
+ /// `info_name` | `uint32 not null`
+ /// `info_value` | `INFO_SCHEMA`
+ ///
+ /// `INFO_SCHEMA` is a dense union with members:
+ ///
+ /// Field Name (Type Code) | Field Type
+ /// ------------------------------|------------------------
+ /// `string_value` (0) | `utf8`
+ /// `bool_value` (1) | `bool`
+ /// `int64_value` (2) | `int64`
+ /// `int32_bitmask` (3) | `int32`
+ /// `string_list` (4) | `list<utf8>`
+ /// `int32_to_int32_list_map` (5) | `map<int32, list<int32>>`
+ ///
+ /// Each metadatum is identified by an integer code. The recognized
+ /// codes are defined as constants. Codes [0, 10_000) are reserved
+ /// for ADBC usage. Drivers/vendors will ignore requests for
+ /// unrecognized codes (the row will be omitted from the result).
+ ///
+ /// For definitions of known ADBC codes, see <https://github.com/apache/arrow-adbc/blob/main/adbc.h>
+ fn get_info(
+ &self,
+ info_codes: &[u32],
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get a hierarchical view of all catalogs, database schemas, tables, and columns.
+ ///
+ /// # Schema
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `catalog_name` | `utf8` |
+ /// | `catalog_db_schemas` | `list<DB_SCHEMA_SCHEMA>` |
+ ///
+ /// `DB_SCHEMA_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `db_schema_name` | `utf8` |
+ /// | `db_schema_tables` | `list<TABLE_SCHEMA>` |
+ ///
+ /// `TABLE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|---------------------------|
+ /// | `table_name` | `utf8 not null` |
+ /// | `table_type` | `utf8 not null` |
+ /// | `table_columns` | `list<COLUMN_SCHEMA>` |
+ /// | `table_constraints` | `list<CONSTRAINT_SCHEMA>` |
+ ///
+ /// `COLUMN_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `column_name` | `utf8 not null` | |
+ /// | `ordinal_position` | `int32` | (1) |
+ /// | `remarks` | `utf8` | (2) |
+ /// | `xdbc_data_type` | `int16` | (3) |
+ /// | `xdbc_type_name` | `utf8` | (3) |
+ /// | `xdbc_column_size` | `int32` | (3) |
+ /// | `xdbc_decimal_digits` | `int16` | (3) |
+ /// | `xdbc_num_prec_radix` | `int16` | (3) |
+ /// | `xdbc_nullable` | `int16` | (3) |
+ /// | `xdbc_column_def` | `utf8` | (3) |
+ /// | `xdbc_sql_data_type` | `int16` | (3) |
+ /// | `xdbc_datetime_sub` | `int16` | (3) |
+ /// | `xdbc_char_octet_length` | `int32` | (3) |
+ /// | `xdbc_is_nullable` | `utf8` | (3) |
+ /// | `xdbc_scope_catalog` | `utf8` | (3) |
+ /// | `xdbc_scope_schema` | `utf8` | (3) |
+ /// | `xdbc_scope_table` | `utf8` | (3) |
+ /// | `xdbc_is_autoincrement` | `bool` | (3) |
+ /// | `xdbc_is_generatedcolumn` | `bool` | (3) |
+ ///
+ /// 1. The column's ordinal position in the table (starting from 1).
+ /// 2. Database-specific description of the column.
+ /// 3. Optional value. Should be null if not supported by the driver.
+ /// xdbc_ values are meant to provide JDBC/ODBC-compatible metadata
+ /// in an agnostic manner.
+ ///
+ /// `CONSTRAINT_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type | Comments |
+ /// |----------------------------|---------------------------|----------|
+ /// | `constraint_name` | `utf8` | |
+ /// | `constraint_type` | `utf8 not null` | (1) |
+ /// | `constraint_column_names` | `list<utf8> not null` | (2) |
+ /// | `constraint_column_usage` | `list<USAGE_SCHEMA>` | (3) |
+ ///
+ /// 1. One of 'CHECK', 'FOREIGN KEY', 'PRIMARY KEY', or 'UNIQUE'.
+ /// 2. The columns on the current table that are constrained, in
+ /// order.
+ /// 3. For FOREIGN KEY only, the referenced table and columns.
+ ///
+ /// `USAGE_SCHEMA` is a Struct with fields:
+ ///
+ /// | Field Name | Field Type |
+ /// |----------------------------|-------------------------|
+ /// | `fk_catalog` | `utf8` |
+ /// | `fk_db_schema` | `utf8` |
+ /// | `fk_table` | `utf8 not null` |
+ /// | `fk_column_name` | `utf8 not null` |
+ ///
+ /// # Parameters
+ ///
+ /// * **depth**: The level of nesting to display. If [AdbcObjectDepth::All], display
+ /// all levels. If [AdbcObjectDepth::Catalogs], display only catalogs (i.e. `catalog_schemas`
+ /// will be null). If [AdbcObjectDepth::DBSchemas], display only catalogs and schemas
+ /// (i.e. `db_schema_tables` will be null), and so on.
+ /// * **catalog**: Only show tables in the given catalog. If None,
+ /// do not filter by catalog. If an empty string, only show tables
+ /// without a catalog. May be a search pattern (see next section).
+ /// * **db_schema**: Only show tables in the given database schema. If
+ /// None, do not filter by database schema. If an empty string, only show
+ /// tables without a database schema. May be a search pattern (see next section).
+ /// * **table_name**: Only show tables with the given name. If None, do not
+ /// filter by name. May be a search pattern (see next section).
+ /// * **table_type**: Only show tables matching one of the given table
+ /// types. If None, show tables of any type. Valid table types should
+ /// match those returned by [AdbcConnection::get_table_schema].
+ /// * **column_name**: Only show columns with the given name. If
+ /// None, do not filter by name. May be a search pattern (see next section).
+ ///
+ /// # Search patterns
+ ///
+ /// Some parameters accept "search patterns", which are
+ /// strings that can contain the special character `"%"` to match zero
+ /// or more characters, or `"_"` to match exactly one character. (See
+ /// the documentation of DatabaseMetaData in JDBC or "Pattern Value
+ /// Arguments" in the ODBC documentation.)
+ fn get_objects(
+ &self,
+ depth: AdbcObjectDepth,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: Option<&str>,
+ table_type: &[&str],
+ column_name: Option<&str>,
+ ) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ /// Get the Arrow schema of a table.
+ ///
+ /// `catalog` or `db_schema` may be `None` when not applicable.
+ fn get_table_schema(
+ &self,
+ catalog: Option<&str>,
+ db_schema: Option<&str>,
+ table_name: &str,
+ ) -> Result<Schema, Self::Error>;
+
+ // TODO: should this just return &[&str]?
+ /// Get a list of table types in the database.
+ ///
+ /// The result is an Arrow dataset with the following schema:
+ ///
+ /// Field Name | Field Type
+ /// -----------------|--------------
+ /// `table_type` | `utf8 not null`
+ fn get_table_types(&self) -> Result<Box<dyn RecordBatchReader>, Self::Error>;
+
+ // TODO: if Connection is only meant to be access from a single thread, does
+ // that mean these partitions should be accessible across connections? Or
+ // That read_partition should be called on the main thread and then the
+ // RBR are then passed to other threads?
Review Comment:
It is meant that the serialized partition should be deserializable by a different connection (basically, FlightInfo/FlightEndpoint). I believe this will work for BigQuery eventually as well (since read sessions are a first-class API object)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 commented on pull request #3582: Start new ADBC crate
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1408972859
> Ideally we should let you consume ADBC from Rust as well. But I'm not up to speed with the state of dynamic linking and such in Rust.
This crate looks promising: https://docs.rs/libloading/latest/libloading/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 closed pull request #3582: Start new ADBC crate
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 closed pull request #3582: Start new ADBC crate
URL: https://github.com/apache/arrow-rs/pull/3582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] wjones127 commented on pull request #3582: Start new ADBC crate
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #3582:
URL: https://github.com/apache/arrow-rs/pull/3582#issuecomment-1409658818
@alamb Don't feel the need to review too soon, actually. When I started the tests for the implementation API, I realized that it might be easier to implement the driver manager first. So I think I will refactor this PR into two PRs:
1. ADBC driver manager
2. ADBC implementation utilities
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org