[jira] [Created] (SPARK-37686) Migrare remaining pyspark.sql.functions to _invoke_* style

["Maciej Szymkiewicz (Jira)" <ji...@apache.org>]

Maciej Szymkiewicz created SPARK-37686:
------------------------------------------

             Summary: Migrare remaining pyspark.sql.functions to _invoke_* style
                 Key: SPARK-37686
                 URL: https://issues.apache.org/jira/browse/SPARK-37686
             Project: Spark
          Issue Type: Improvement
          Components: PySpark, SQL
    Affects Versions: 3.3.0
            Reporter: Maciej Szymkiewicz


In SPARK-32084 we converted a number of dynamically created functions to standard def using newly added {{_invoke_function}} helpers. Remaining functions stayed untouched. 

These allow us to implement these with minimum amount of boilerplate code.

With recent migration of type hints to inline variant we had to type checker hints:

{code:python}
    sc = SparkContext._active_spark_context
    assert sc is not None and sc._jvm is not Non
{code}

to the functions not covered in SPARK-32084. This added to standard conversions (i.e. ({_to_java_column}}, {{Column(jc)}}) make these pretty verbose.

However, it is possible to convert remaining functions using {{_invoke_function}} helpers to address that.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org