[jira] [Created] (SPARK-60255) [SPIP] ADBC Data Source

["Tornike Gurgenidze (Jira)" <ji...@apache.org>]

Tornike Gurgenidze created SPARK-60255:
------------------------------------------

             Summary: [SPIP] ADBC Data Source
                 Key: SPARK-60255
                 URL: https://issues.apache.org/jira/browse/SPARK-60255
             Project: Spark
          Issue Type: New Feature
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Tornike Gurgenidze


Summary

Add a built-in ADBC (Arrow Database Connectivity) data source to Spark, alongside the existing JDBC data source. It reads and writes through any ADBC driver and passes Arrow column batches straight into Spark as ColumnarBatches, so data stays in columns from the database to Spark.

*SPIP document:* [SPIP: ADBC Data Source|https://docs.google.com/document/d/1hewLmAc8yFCY4PYepE02Yz8iACCAWkRUZuhzHyyMqBY/edit?tab=t.0]
*Prototype:* [tokoko/spark-adbc|https://github.com/tokoko/spark-adbc]

Motivation

JDBC returns data one row at a time, and Spark reads each value separately. For analytical databases that store or send data in columns, results are converted from columns to rows at the driver, read value by value in Spark, and often converted back to columns by vectorized engines (Comet, Gluten). On large reads this conversion dominates. The spark-adbc prototype shows 3-10x read throughput over JDBC on column-oriented databases.

Proposal

* New optional data source: \{{spark.read.format("adbc")}}. No public API changes; JDBC is unchanged.
* Reads: Arrow batches wrapped as ColumnarBatch via ArrowColumnVector, without copying.
* Pushdown of column pruning, filters, limit, Top-N and aggregates.
* Partitioning: JDBC-style range partitioning, plus driver partitioning through ADBC executePartitioned.
* Writes: per-task appends through ADBC bulk ingest (same at-least-once guarantee as the JDBC writer).
* Pushdown semantics without per-database dialect code in Spark: SQL generated from driver-reported metadata, with an optional Substrait-based mode.

Non-goals

* Replacing or changing the JDBC data source.
* Atomic distributed writes in the first version (depends on apache/arrow-adbc#4317).
* Structured Streaming.

Related upstream work

* [apache/arrow-adbc#4317|https://github.com/apache/arrow-adbc/pull/4317] - partitioned bulk ingest
* [apache/arrow#49796|https://github.com/apache/arrow/pull/49796] - dialect-related Flight SQL SqlInfo codes
* [substrait-io/substrait#1283|https://github.com/substrait-io/substrait/issues/1283] - SqlQuery read type



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org