You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/05/13 15:02:00 UTC

[jira] [Updated] (SPARK-52104) Validate column name eagerly in Spark Connect Scala Client

     [ https://issues.apache.org/jira/browse/SPARK-52104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-52104:
-----------------------------------
    Labels: pull-request-available  (was: )

> Validate column name eagerly in Spark Connect Scala Client
> ----------------------------------------------------------
>
>                 Key: SPARK-52104
>                 URL: https://issues.apache.org/jira/browse/SPARK-52104
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 4.0.0, 4.1.0
>            Reporter: Xi Lyu
>            Priority: Major
>              Labels: pull-request-available
>
> Currently, calling DataFrame.col(colName) with a non-existent column in the Scala Spark Connect client does not raise an error. In contrast, both PySpark (in Spark Connect and Spark Classic) and Scala in Spark Classic do raise an exception in such cases. This leads to inconsistent behavior between Spark Connect and Spark Classic in Scala.
> In this task, eager validation of column names should be implemented in the DataFrame.col(colName) method of the Scala client to ensure consistent behavior with both Spark Classic and PySpark.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org