You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2023/12/29 01:04:59 UTC
[PR] [SPARK-46397][PYTHON][CONNECT] Function `sha2` should raise `PySparkValueError` for invalid `numBits` [spark]
zhengruifeng opened a new pull request, #44529:
URL: https://github.com/apache/spark/pull/44529
### What changes were proposed in this pull request?
Function `sha2` should raise `PySparkValueError` for invalid `numBits`
### Why are the changes needed?
vanilla PySpark invokes the Scala side and raise an `IllegalArgumentException`
https://github.com/apache/spark/blob/fa4096eb6aba4c66f0d9c5dcbabdfc0804064fff/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L3212-L3217
while Python client won't do this check and raise an `AnalysisException`.
They should both raise a `PySparkValueError` for this case.
### Does this PR introduce _any_ user-facing change?
yes
```
In [1]: from pyspark.sql import functions as sf
...: spark.range(1).select(sf.sha2(sf.col("id"), 1024)).collect()
---------------------------------------------------------------------------
PySparkValueError Traceback (most recent call last)
<ipython-input-1-1ae9879dcc31> in ?()
1 from pyspark.sql import functions as sf
----> 2 spark.range(1).select(sf.sha2(sf.col("id"), 1024)).collect()
~/Dev/spark/python/pyspark/sql/utils.py in ?(*args, **kwargs)
190 from pyspark.sql.connect import functions
191
192 return getattr(functions, f.__name__)(*args, **kwargs)
193 else:
--> 194 return f(*args, **kwargs)
~/Dev/spark/python/pyspark/sql/functions/builtin.py in ?(col, numBits)
9112 |Bob |cd9fb1e148ccd8442e5aa74904cc73bf6fb54d1d54d333bd596aa9bb4bb4e961|
9113 +-----+----------------------------------------------------------------+
9114 """
9115 if numBits not in [0, 224, 256, 384, 512]:
-> 9116 raise PySparkValueError(
9117 error_class="VALUE_NOT_ALLOWED",
9118 message_parameters={
9119 "arg_name": "numBits",
PySparkValueError: [VALUE_NOT_ALLOWED] Value for `numBits` has to be amongst the following values: [0, 224, 256, 384, 512].
```
### How was this patch tested?
added ut
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46397][PYTHON][CONNECT] Function `sha2` should raise `PySparkValueError` for invalid `numBits` [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #44529: [SPARK-46397][PYTHON][CONNECT] Function `sha2` should raise `PySparkValueError` for invalid `numBits`
URL: https://github.com/apache/spark/pull/44529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46397][PYTHON][CONNECT] Function `sha2` should raise `PySparkValueError` for invalid `numBits` [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #44529:
URL: https://github.com/apache/spark/pull/44529#issuecomment-1871802668
merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46397][PYTHON][CONNECT] Function `sha2` should raise `PySparkValueError` for invalid `numBits` [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #44529:
URL: https://github.com/apache/spark/pull/44529#issuecomment-1871655620
cc @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org