You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "skanderboudawara (via GitHub)" <gi...@apache.org> on 2024/12/27 22:29:06 UTC

[PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]

skanderboudawara opened a new pull request, #49326:
URL: https://github.com/apache/spark/pull/49326

   # What changes were proposed in this pull request?
   
   Change all the functions parameters shadowing python built-in to a distinguished ones.
   
   Why are the changes needed?
   
   As mentionned in the Jira ticket
   ```
   # breaks:
   foo(str="x", bar="y")
   
   # okay:
   foo("x", bar="y")
   ```
   
   Does this PR introduce any user-facing change?
   
   Yes.
   
   How was this patch tested?
   
   Github actions
   
   Was this patch authored or co-authored using generative AI tooling?
   
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]

Posted by "skanderboudawara (via GitHub)" <gi...@apache.org>.
skanderboudawara commented on code in PR #49326:
URL: https://github.com/apache/spark/pull/49326#discussion_r1899106300


##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -339,10 +339,10 @@ def coalesce(*cols: "ColumnOrName") -> Column:
 coalesce.__doc__ = pysparkfuncs.coalesce.__doc__
 
 
-def expr(str: str) -> Column:
+def expr(expression: str) -> Column:

Review Comment:
   Hey,
   
   I think my decorator needs another update. Since we can deprecate more than one parameter, I’m wondering if it makes sense to use a dictionary or maybe something else.
   
   On the consistency side, all the new functions since version 3.3 use “col” instead of “str” for parameters. It feels like harmonizing everything would be a good move, but as you mentioned, it could be a breaking change for folks using keyword arguments.
   
   What do you think?
   Should we wait for more opinions on this topic ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49326:
URL: https://github.com/apache/spark/pull/49326#issuecomment-2807850427

   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #49326:
URL: https://github.com/apache/spark/pull/49326#discussion_r1898861186


##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -339,10 +339,10 @@ def coalesce(*cols: "ColumnOrName") -> Column:
 coalesce.__doc__ = pysparkfuncs.coalesce.__doc__
 
 
-def expr(str: str) -> Column:
+def expr(expression: str) -> Column:

Review Comment:
   This is actually a breaking change if users use this API via kwargs ..



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #49326:
URL: https://github.com/apache/spark/pull/49326#discussion_r1899088912


##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -339,10 +339,10 @@ def coalesce(*cols: "ColumnOrName") -> Column:
 coalesce.__doc__ = pysparkfuncs.coalesce.__doc__
 
 
-def expr(str: str) -> Column:
+def expr(expression: str) -> Column:

Review Comment:
   We can do but i feel like deprecating the parameter is too much too. Shadowing the builtin names are not nice but it won't affect end users anyway



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]

Posted by "skanderboudawara (via GitHub)" <gi...@apache.org>.
skanderboudawara commented on code in PR #49326:
URL: https://github.com/apache/spark/pull/49326#discussion_r1898882047


##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -339,10 +339,10 @@ def coalesce(*cols: "ColumnOrName") -> Column:
 coalesce.__doc__ = pysparkfuncs.coalesce.__doc__
 
 
-def expr(str: str) -> Column:
+def expr(expression: str) -> Column:

Review Comment:
   Shall I add to keep both versions? 
   ```
   @overload
   def expr(str: str) -> Column: …
   ```
   
   Or else I create a decorator for future releases like so
   
   ```
   from functools import wraps
   
   def deprecated_param(old_param: str, new_param: str):
       def decorator(func):
           @wraps(func)
           def wrapper(*args, **kwargs):
               if old_param in kwargs:
                   import warnings
                   warnings.warn(f"'{old_param}' is deprecated, use '{new_param}' instead", DeprecationWarning)
                   kwargs[new_param] = kwargs.pop(old_param)
               return func(*args, **kwargs)
           return wrapper
       return decorator
   
   @deprecated_param("str", "expression")
   def expr(expression: str): ...
   ```
   
   
   what do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #49326: [SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming
URL: https://github.com/apache/spark/pull/49326


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org