You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/24 19:58:47 UTC
[GitHub] [spark] khalidmammadov opened a new pull request, #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call
khalidmammadov opened a new pull request, #37650:
URL: https://github.com/apache/spark/pull/37650
### What changes were proposed in this pull request?
PySpark atan2, hypot, pow and pmod functions marked as accepting float type as argument but produce error when called together. For example:
```
>>> from pyspark.sql.functions import *
>>> df = spark.range(1)
>>> df.select(atan2(1.1, 2.1)).first()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/sql/functions.py", line 962, in atan2
return _invoke_binary_math_function("atan2", col1, col2)
File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/sql/functions.py", line 111, in _invoke_binary_math_function
return _invoke_function(
File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/sql/functions.py", line 85, in _invoke_function
return Column(jf(*args))
File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/sql/utils.py", line 190, in deco
return f(*a, **kw)
File "/home/khalid/dev/p/SparkTest/spark_venv3.10/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 330, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling z:org.apache.spark.sql.functions.atan2. Trace:
py4j.Py4JException: Method atan2([class java.lang.Double, class java.lang.Double]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:339)
at py4j.Gateway.invoke(Gateway.java:276)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:748)
```
Although, this call could be written outside the Spark and supplied as a literal it's still good to be consistent with SQL API and help user to avoid seeing error.
after the fix:
```
>>> from pyspark.sql.functions import *
>>> df = spark.range(1)
>>> df.select(atan2(1.1, 2.1)).first()
Row(ATAN2(1.1, 2.1)=0.4825132950224769)
```
### Why are the changes needed?
Improve user experience. Bug fix
### Does this PR introduce _any_ user-facing change?
Yes, no errors
### How was this patch tested?
```
./build/sbt
test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] khalidmammadov commented on pull request #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on PR #37650:
URL: https://github.com/apache/spark/pull/37650#issuecomment-1233269525
Sorry, there wasn't yet. Got distracted to other changes. Will do one today/tomorrow to see any good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #37650:
URL: https://github.com/apache/spark/pull/37650#issuecomment-1233221075
Was there a follow-up PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] khalidmammadov commented on pull request #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on PR #37650:
URL: https://github.com/apache/spark/pull/37650#issuecomment-1233378272
follow up PR: https://github.com/apache/spark/pull/37748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] khalidmammadov commented on pull request #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on PR #37650:
URL: https://github.com/apache/spark/pull/37650#issuecomment-1226948626
Will be fixed on PySpark side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #37650:
URL: https://github.com/apache/spark/pull/37650#discussion_r954487181
##########
sql/core/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -1733,6 +1733,19 @@ object functions {
*/
def atan2(yValue: Double, xName: String): Column = atan2(yValue, Column(xName))
+ /**
+ * @param yValue coordinate on y-axis
+ * @param xValue coordinate on x-axis
+ * @return the <i>theta</i> component of the point
+ * (<i>r</i>, <i>theta</i>)
+ * in polar coordinates that corresponds to the point
+ * (<i>x</i>, <i>y</i>) in Cartesian coordinates,
+ * as if computed by `java.lang.Math.atan2`
+ * @group math_funcs
+ * @since 3.4.0
+ */
+ def atan2(yValue: Double, xValue: Double): Column = atan2(lit(yValue), lit(xValue))
Review Comment:
I think we should probably fix it in PySpark side instead of fixing it in Scala side (see the comments on the top at both functions.py and functions.scala)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] khalidmammadov commented on a diff in pull request #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on code in PR #37650:
URL: https://github.com/apache/spark/pull/37650#discussion_r954664196
##########
sql/core/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -1733,6 +1733,19 @@ object functions {
*/
def atan2(yValue: Double, xName: String): Column = atan2(yValue, Column(xName))
+ /**
+ * @param yValue coordinate on y-axis
+ * @param xValue coordinate on x-axis
+ * @return the <i>theta</i> component of the point
+ * (<i>r</i>, <i>theta</i>)
+ * in polar coordinates that corresponds to the point
+ * (<i>x</i>, <i>y</i>) in Cartesian coordinates,
+ * as if computed by `java.lang.Math.atan2`
+ * @group math_funcs
+ * @since 3.4.0
+ */
+ def atan2(yValue: Double, xValue: Double): Column = atan2(lit(yValue), lit(xValue))
Review Comment:
Sure, will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] khalidmammadov closed pull request #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call
Posted by GitBox <gi...@apache.org>.
khalidmammadov closed pull request #37650: [SPARK-40210][PYTHON][CORE] Fix math atan2, hypot, pow and pmod float argument call
URL: https://github.com/apache/spark/pull/37650
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org