[arrow-adbc] 10/14: docs: update prose for 1.1.0 and clarify cancellation (#932)

[li...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

lidavidm pushed a commit to branch spec-1.1.0
in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git

commit 581c2a1db002b0089d9cb4c380e7ea03796178f3
Author: David Li <li...@gmail.com>
AuthorDate: Mon Jul 24 15:44:05 2023 -0400

    docs: update prose for 1.1.0 and clarify cancellation (#932)
    
    Fixes #928.
    Fixes #929.
---
 adbc.h                               |  12 ++--
 docs/source/format/specification.rst | 110 +++++++++++++++++++++++++++++++++++
 docs/source/format/versioning.rst    |  28 +++++----
 go/adbc/drivermgr/adbc.h             |  12 ++--
 4 files changed, 143 insertions(+), 19 deletions(-)

diff --git a/adbc.h b/adbc.h
index badfc7d6..f7ea69af 100644
--- a/adbc.h
+++ b/adbc.h
@@ -1292,9 +1292,11 @@ AdbcStatusCode AdbcConnectionRelease(struct AdbcConnection* connection,
 /// or while consuming an ArrowArrayStream returned from such.
 /// Calling this function should make the other functions return
 /// ADBC_STATUS_CANCELLED (from ADBC functions) or ECANCELED (from
-/// methods of ArrowArrayStream).
+/// methods of ArrowArrayStream).  (It is not guaranteed to, for
+/// instance, the result set may be buffered in memory already.)
 ///
-/// This must always be thread-safe (other operations are not).
+/// This must always be thread-safe (other operations are not).  It is
+/// not necessarily signal-safe.
 ///
 /// \since ADBC API revision 1.1.0
 /// \addtogroup adbc-1.1.0
@@ -1947,9 +1949,11 @@ AdbcStatusCode AdbcStatementBindStream(struct AdbcStatement* statement,
 /// or while consuming an ArrowArrayStream returned from such.
 /// Calling this function should make the other functions return
 /// ADBC_STATUS_CANCELLED (from ADBC functions) or ECANCELED (from
-/// methods of ArrowArrayStream).
+/// methods of ArrowArrayStream).  (It is not guaranteed to, for
+/// instance, the result set may be buffered in memory already.)
 ///
-/// This must always be thread-safe (other operations are not).
+/// This must always be thread-safe (other operations are not).  It is
+/// not necessarily signal-safe.
 ///
 /// \since ADBC API revision 1.1.0
 /// \addtogroup adbc-1.1.0
diff --git a/docs/source/format/specification.rst b/docs/source/format/specification.rst
index e7a44a41..88515e41 100644
--- a/docs/source/format/specification.rst
+++ b/docs/source/format/specification.rst
@@ -57,6 +57,26 @@ implementations will support this.
 - Go: ``OptionKeyAutoCommit``
 - Java: ``org.apache.arrow.adbc.core.AdbcConnection#setAutoCommit(boolean)``
 
+Metadata
+--------
+
+ADBC exposes a variety of metadata about the database, such as what catalogs,
+schemas, and tables exist, the Arrow schema of tables, and so on.
+
+.. _specification-statistics:
+
+Statistics
+----------
+
+.. note:: Since API revision 1.1.0
+
+ADBC exposes table/column statistics, such as the (unique) row count, min/max
+values, and so on.  The goal here is to make ADBC work better in federation
+scenarios, where one query engine wants to read Arrow data from another
+database.  Having statistics available lets the "outer" query planner make
+better choices about things like join order, or even decide to skip reading
+data entirely.
+
 Statements
 ==========
 
@@ -84,6 +104,16 @@ frees the user from knowing the right SQL syntax for their database.
 - Go: ``OptionKeyIngestTargetTable``
 - Java: ``org.apache.arrow.adbc.core.AdbcConnection#bulkIngest(String, org.apache.arrow.adbc.core.BulkIngestMode)``
 
+.. _specification-cancellation:
+
+Cancellation
+------------
+
+.. note:: Since API revision 1.1.0
+
+Queries (and operations that implicitly represent queries, like fetching
+:ref:`specification-statistics`) can be cancelled.
+
 Partitioned Result Sets
 -----------------------
 
@@ -97,6 +127,16 @@ machines.
 - Go: ``Statement.ExecutePartitions``
 - Java: ``org.apache.arrow.adbc.core.AdbcStatement#executePartitioned()``
 
+.. _specification-incremental-execution:
+
+In principle, a vendor could return the results of partitioned execution as
+they are available, instead of all at once.  Incremental execution allows
+drivers to expose this.  When enabled, each call to ``ExecutePartitions`` will
+return available endpoints to read instead of blocking to retrieve all
+endpoints.
+
+.. note:: Since API revision 1.1.0
+
 Lifecycle & Usage
 -----------------
 
@@ -135,3 +175,73 @@ Partitioned Execution
 .. mermaid:: AdbcStatementPartitioned.mmd
    :caption: This is similar to fetching data in Arrow Flight RPC (by
              design). See :doc:`"Downloading Data" <arrow:format/Flight>`.
+
+Error Handling
+==============
+
+The error handling strategy varies by language.
+
+In C, most methods take a :cpp:class:`AdbcError`.  In Go, most methods return
+an error that can be cast to an ``AdbcError``.  In Java, most methods raise an
+``AdbcException``.
+
+In all cases, an error contains:
+
+- A status code,
+- An error message,
+- An optional vendor code (a vendor-specific status code),
+- An optional 5-character "SQLSTATE" code (a SQL-like vendor-specific code).
+
+.. _specification-rich-error-metadata:
+
+Rich Error Metadata
+-------------------
+
+.. note:: Since API revision 1.1.0
+
+Drivers can expose additional rich error metadata.  This can be used to return
+structured error information.  For example, a driver could use something like
+the `Googleapis ErrorDetails`_.
+
+In C, special option values can be read after receiving an error to get error
+metadata.  In Go and Java, ``AdbcError`` and ``AdbcException`` respectively
+expose a list of additional metadata.
+
+.. _Googleapis ErrorDetails: https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto
+
+Changelog
+=========
+
+Version 1.1.0
+-------------
+
+The info key ADBC_INFO_DRIVER_ADBC_VERSION can be used to retrieve the
+driver's supported ADBC version.
+
+The canonical options "uri", "username", and "password" were added to make
+configuration consistent between drivers.
+
+:ref:`specification-cancellation` and the ability to both get and set options
+of different types were added.  (Previously, you could set string options but
+could not get option values or get/set values of other types.)  This can be
+used to get and set the current active catalog and/or schema through a pair of
+new canonical options.
+
+:ref:`specification-bulk-ingestion` supports two additional modes:
+
+- "adbc.ingest.mode.replace" will drop existing data, then behave like
+  "create".
+- "adbc.ingest.mode.create_append" will behave like "create", except if the
+  table already exists, it will not error.
+
+:ref:`specification-rich-error-metadata` has been added, allowing clients to
+get additional error metadata.
+
+The ability to retrive table/column :ref:`statistics
+<specification-statistics>` was added.  The goal here is to make ADBC work
+better in federation scenarios, where one query engine wants to read Arrow
+data from another database.
+
+:ref:`Incremental execution <specification-incremental-execution>` allows
+streaming partitions of a result set as they are available instead of blocking
+and waiting for query execution to finish before reading results.
diff --git a/docs/source/format/versioning.rst b/docs/source/format/versioning.rst
index 3205b792..b255aeeb 100644
--- a/docs/source/format/versioning.rst
+++ b/docs/source/format/versioning.rst
@@ -29,14 +29,19 @@ choices were made:
 Of course, we can never add/remove/change struct members, and we can
 never change the signatures of existing functions.
 
-The main point of concern is compatibility of :cpp:class:`AdbcDriver`.
+In ADBC 1.1.0, it was decided this would only apply to the "public"
+API, and not the driver-internal API (:cpp:class:`AdbcDriver`).  New
+members were added to this struct in the 1.1.0 revision.
+Compatibility is handled as follows:
 
 The driver entrypoint, :cpp:type:`AdbcDriverInitFunc`, is given a
-version and a pointer to a table of function pointers to initialize.
-The type of the table will depend on the version; when a new version
-of ADBC is accepted, then a new table of function pointers will be
-added.  That way, the driver knows the type of the table.  If/when we
-add a new ADBC version, the following scenarios are possible:
+version and a pointer to a table of function pointers to initialize
+(the :cpp:class:`AdbcDriver`).  The size of the table will depend on
+the version; when a new version of ADBC is accepted, then a new table
+of function pointers may be expanded.  For each version, the driver
+knows the expected size of the table, and must not read/write fields
+beyond that size.  If/when we add a new ADBC version, the following
+scenarios are possible:
 
 - An updated client application uses an old driver library.  The
   client will pass a `version` field greater than what the driver
@@ -46,7 +51,8 @@ add a new ADBC version, the following scenarios are possible:
 - An old client application uses an updated driver library.  The
   client will pass a ``version`` lower than what the driver
   recognizes, so the driver can either error, or if it can still
-  implement the old API contract, initialize the older table.
+  implement the old API contract, initialize the subset of the table
+  corresponding to the older version.
 
 This approach does not let us change the signatures of existing
 functions, but we can add new functions and remove existing ones.
@@ -64,7 +70,7 @@ backwards-incompatible versions such as 2.0.0, but which still
 implement the API standard version 1.0.0.
 
 Similarly, this documentation describes the ADBC API standard version
-1.0.0.  If/when a compatible revision is made (e.g. new standard
-options are defined), the next version would be 1.1.0.  If
-incompatible changes are made (e.g. new API functions), the next
-version would be 2.0.0.
+1.1.0.  If/when a compatible revision is made (e.g. new standard
+options or API functions are defined), the next version would be
+1.2.0.  If incompatible changes are made (e.g. changing the signature
+or semantics of a function), the next version would be 2.0.0.
diff --git a/go/adbc/drivermgr/adbc.h b/go/adbc/drivermgr/adbc.h
index badfc7d6..f7ea69af 100644
--- a/go/adbc/drivermgr/adbc.h
+++ b/go/adbc/drivermgr/adbc.h
@@ -1292,9 +1292,11 @@ AdbcStatusCode AdbcConnectionRelease(struct AdbcConnection* connection,
 /// or while consuming an ArrowArrayStream returned from such.
 /// Calling this function should make the other functions return
 /// ADBC_STATUS_CANCELLED (from ADBC functions) or ECANCELED (from
-/// methods of ArrowArrayStream).
+/// methods of ArrowArrayStream).  (It is not guaranteed to, for
+/// instance, the result set may be buffered in memory already.)
 ///
-/// This must always be thread-safe (other operations are not).
+/// This must always be thread-safe (other operations are not).  It is
+/// not necessarily signal-safe.
 ///
 /// \since ADBC API revision 1.1.0
 /// \addtogroup adbc-1.1.0
@@ -1947,9 +1949,11 @@ AdbcStatusCode AdbcStatementBindStream(struct AdbcStatement* statement,
 /// or while consuming an ArrowArrayStream returned from such.
 /// Calling this function should make the other functions return
 /// ADBC_STATUS_CANCELLED (from ADBC functions) or ECANCELED (from
-/// methods of ArrowArrayStream).
+/// methods of ArrowArrayStream).  (It is not guaranteed to, for
+/// instance, the result set may be buffered in memory already.)
 ///
-/// This must always be thread-safe (other operations are not).
+/// This must always be thread-safe (other operations are not).  It is
+/// not necessarily signal-safe.
 ///
 /// \since ADBC API revision 1.1.0
 /// \addtogroup adbc-1.1.0