You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "skanderboudawara (via GitHub)" <gi...@apache.org> on 2024/12/27 22:29:06 UTC
[PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]
skanderboudawara opened a new pull request, #49326:
URL: https://github.com/apache/spark/pull/49326
# What changes were proposed in this pull request?
Change all the functions parameters shadowing python built-in to a distinguished ones.
Why are the changes needed?
As mentionned in the Jira ticket
```
# breaks:
foo(str="x", bar="y")
# okay:
foo("x", bar="y")
```
Does this PR introduce any user-facing change?
Yes.
How was this patch tested?
Github actions
Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]
Posted by "skanderboudawara (via GitHub)" <gi...@apache.org>.
skanderboudawara commented on code in PR #49326:
URL: https://github.com/apache/spark/pull/49326#discussion_r1899106300
##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -339,10 +339,10 @@ def coalesce(*cols: "ColumnOrName") -> Column:
coalesce.__doc__ = pysparkfuncs.coalesce.__doc__
-def expr(str: str) -> Column:
+def expr(expression: str) -> Column:
Review Comment:
Hey,
I think my decorator needs another update. Since we can deprecate more than one parameter, I’m wondering if it makes sense to use a dictionary or maybe something else.
On the consistency side, all the new functions since version 3.3 use “col” instead of “str” for parameters. It feels like harmonizing everything would be a good move, but as you mentioned, it could be a breaking change for folks using keyword arguments.
What do you think?
Should we wait for more opinions on this topic ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49326:
URL: https://github.com/apache/spark/pull/49326#issuecomment-2807850427
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #49326:
URL: https://github.com/apache/spark/pull/49326#discussion_r1898861186
##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -339,10 +339,10 @@ def coalesce(*cols: "ColumnOrName") -> Column:
coalesce.__doc__ = pysparkfuncs.coalesce.__doc__
-def expr(str: str) -> Column:
+def expr(expression: str) -> Column:
Review Comment:
This is actually a breaking change if users use this API via kwargs ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #49326:
URL: https://github.com/apache/spark/pull/49326#discussion_r1899088912
##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -339,10 +339,10 @@ def coalesce(*cols: "ColumnOrName") -> Column:
coalesce.__doc__ = pysparkfuncs.coalesce.__doc__
-def expr(str: str) -> Column:
+def expr(expression: str) -> Column:
Review Comment:
We can do but i feel like deprecating the parameter is too much too. Shadowing the builtin names are not nice but it won't affect end users anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]
Posted by "skanderboudawara (via GitHub)" <gi...@apache.org>.
skanderboudawara commented on code in PR #49326:
URL: https://github.com/apache/spark/pull/49326#discussion_r1898882047
##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -339,10 +339,10 @@ def coalesce(*cols: "ColumnOrName") -> Column:
coalesce.__doc__ = pysparkfuncs.coalesce.__doc__
-def expr(str: str) -> Column:
+def expr(expression: str) -> Column:
Review Comment:
Shall I add to keep both versions?
```
@overload
def expr(str: str) -> Column: …
```
Or else I create a decorator for future releases like so
```
from functools import wraps
def deprecated_param(old_param: str, new_param: str):
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
if old_param in kwargs:
import warnings
warnings.warn(f"'{old_param}' is deprecated, use '{new_param}' instead", DeprecationWarning)
kwargs[new_param] = kwargs.pop(old_param)
return func(*args, **kwargs)
return wrapper
return decorator
@deprecated_param("str", "expression")
def expr(expression: str): ...
```
what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #49326: [SPARK-47854][PYTHON][CONNECT] Avoid shadowing python built-ins in python function variable naming
URL: https://github.com/apache/spark/pull/49326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org