[jira] [Created] (SPARK-59140) Close PySpark 4.2.0 public-API parity gaps in the Rust drop-in client

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

Hyukjin Kwon created SPARK-59140:
------------------------------------

             Summary: Close PySpark 4.2.0 public-API parity gaps in the Rust drop-in client
                 Key: SPARK-59140
                 URL: https://issues.apache.org/jira/browse/SPARK-59140
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The Rust drop-in client (pyspark-client-rust) advertises "100% public-API parity with PySpark 4.2.0 / runs unchanged", but keyword-argument calls raised TypeError on ~507 SQL functions and dozens of DataFrame/Column/Catalog/Reader/Writer/Streaming/Session methods, and several parameters were missing entirely. So common PySpark code did not run unchanged.

This issue tracks bringing the public API of pyspark.sql.connect.* to keyword-argument and signature parity with PySpark 4.2.0.

Verified with an independent signature harness against the v4.2.0 tag:
- function keyword-arg mismatches: 513 -> 0
- class-method param mismatches: 70 -> 2 (both intentional: Column.alias/name expose an explicit metadata= where the reference uses **kwargs)

Also verified: all Rust golden/core tests pass, and a focused end-to-end suite runs 16/16 green against a real Spark Connect 4.2.0 server.

A few params are accepted for signature parity but not yet behaviorally wired pending connect-core/proto support (documented in-code and in PARITY_GAP_ANALYSIS.md): mapInPandas/mapInArrow(profile), localCheckpoint(storageLevel), printSchema(level), createDataFrame(verifySchema), trigger(realTime), multi-path read.parquet(*paths).




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org