[jira] [Created] (SPARK-58998) [spark-connect-rust] Drop-in pyspark API fidelity + high-level API test coverage

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

Hyukjin Kwon created SPARK-58998:
------------------------------------

             Summary: [spark-connect-rust] Drop-in pyspark API fidelity + high-level API test coverage
                 Key: SPARK-58998
                 URL: https://issues.apache.org/jira/browse/SPARK-58998
             Project: Spark
          Issue Type: Improvement
          Components: Project Infra
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Follow-up to SPARK-58987 (which added the ours-only parity gate + coverage tooling).

Two findings from the coverage work:
h2. 1. The parity gate covers the wire, not our high-level API
The transport-injection harness swaps only the gRPC stub, so the official suite runs the REFERENCE pyspark DataFrame/Session/Column classes and only routes bytes through our Rust stub. Our high-level API (spark-connect DataFrame/Session/Catalog/... and the pyspark-rs bindings) is therefore NOT exercised by it - merged coverage is ~49%, with those modules near 0-20% while the wire layer (client/error/retries/channel/bytes_codec) is 90%+.

h2. 2. The drop-in pyspark API diverges from reference pyspark
The skin re-exports the Rust PyO3 classes directly, and their API is name-divergent and incomplete vs reference pyspark, e.g.:
* SparkSession: no `conf`, no `read` (only `readStream`); `catalog` is a method, not a property
* DataFrame: `where_` not `where`; missing `withColumns`, `hint`, `unpivot`/`melt`, `intersectAll`; `head()` takes no count; `createDataFrame` accepts only a list of column names (no StructType/DDL)
* Column: `is_null`/`is_not_null`/`get_field` (snake_case) not `isNull`/`isNotNull`/`getField`
* GroupedData: no `pivot`
The origin project reached full parity via a skin adapter layer; the fork appears to have lost it. `df.where(...)`, `col.isNull()`, `spark.read`, etc. would break for users.

h2. Work
* Restore a faithful drop-in skin (adapter wrappers: where_->where, is_null->isNull, add conf/read/pivot/withColumns/hint/unpivot/intersectAll/head(n)/StructType-createDataFrame, catalog as a property).
* Add high-level API tests (Rust golden+e2e for the spark-connect API; a drop-in Python exercise, scripts/e2e_wrapper.py, for the pyspark-rs bindings).
* Raise the rust-coverage CI gate (scripts/rust_coverage.sh, added in SPARK-58987) to 90%.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org