You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/11/13 04:33:00 UTC

[jira] [Updated] (SPARK-38912) Clean "classproperty" workaround in pyspark.sql.session once support for Python 3.8 is dropped

     [ https://issues.apache.org/jira/browse/SPARK-38912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-38912:
-----------------------------------
    Labels: pull-request-available  (was: )

> Clean "classproperty" workaround in pyspark.sql.session once support for Python 3.8 is dropped
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-38912
>                 URL: https://issues.apache.org/jira/browse/SPARK-38912
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, SQL
>    Affects Versions: 3.2.1
>            Reporter: Furcy Pin
>            Priority: Minor
>              Labels: pull-request-available
>
> *Context*
> The proper bugfix of SPARK-38870 would be to use the @classmethod + @property decorators on the SparkSession.builder attribute to make it return a new builder every time like in Scala. 
> However, @classmethod + @property decorators have only been made compatible since Python 3.9: [https://docs.python.org/3/howto/descriptor.html#class-methods]
> > The code path for {{hasattr(type(self.f), '_{_}get{_}_')}} was added in Python 3.9 and makes it possible for [{{classmethod()}}|https://docs.python.org/3/library/functions.html#classmethod] to support chained decorators.
> Moreover, @classmethod + @property waq also affected by a bug which has been been fixed in Python 3.9.6 (see also SPARK-38964) :
> [https://github.com/python/cpython/pull/28838]
> So we had to use a temporary workaround by defining a custom decorator named @classproperty to fix SPARK-38870. 
> *What should be done*
> Once support for Python 3.8 and "<=3.9.5" is officially dropped, we should remove this @classproperty workaround and replace it with @classmethod + @property (cf TODOs in the [pyspark.sql.session module|https://github.com/apache/spark/pull/36161/files]).
>  
> *Further details*
> _Python 3.8 will end-of-life is 2024/10._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org