You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2022/07/05 11:52:51 UTC
[spark] branch master updated: [SPARK-39611][PYTHON][PS] Fix wrong aliases in __array_ufunc__
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fb48a14a679 [SPARK-39611][PYTHON][PS] Fix wrong aliases in __array_ufunc__
fb48a14a679 is described below
commit fb48a14a67940b9270390b8ce74c19ae58e2880e
Author: Yikun Jiang <yi...@gmail.com>
AuthorDate: Tue Jul 5 20:52:36 2022 +0900
[SPARK-39611][PYTHON][PS] Fix wrong aliases in __array_ufunc__
### What changes were proposed in this pull request?
This PR fix the wrong aliases in `__array_ufunc__`
### Why are the changes needed?
When running test with numpy 1.23.0 (current latest), hit a bug: `NotImplementedError: pandas-on-Spark objects currently do not support <ufunc 'divide'>.`
In `__array_ufunc__` we first call `maybe_dispatch_ufunc_to_dunder_op` to try dunder methods first, and then we try pyspark API. `maybe_dispatch_ufunc_to_dunder_op` is from pandas code.
pandas fix a bug https://github.com/pandas-dev/pandas/pull/44822#issuecomment-991166419 https://github.com/pandas-dev/pandas/pull/44822/commits/206b2496bc6f6aa025cb26cb42f52abeec227741 when upgrade to numpy 1.23.0, we need to also sync this.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Current CI passed
- The exsiting UT `test_series_datetime` already cover this, I also test it in my local env with 1.23.0
```shell
pip install "numpy==1.23.0"
python/run-tests --testnames 'pyspark.pandas.tests.test_series_datetime SeriesDateTimeTest.test_arithmetic_op_exceptions'
```
Closes #37078 from Yikun/SPARK-39611.
Authored-by: Yikun Jiang <yi...@gmail.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/pandas/numpy_compat.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/numpy_compat.py b/python/pyspark/pandas/numpy_compat.py
index ea72fa658e4..f9b7bd67a9b 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -166,7 +166,7 @@ def maybe_dispatch_ufunc_to_dunder_op(
"true_divide": "truediv",
"power": "pow",
"remainder": "mod",
- "divide": "div",
+ "divide": "truediv",
"equal": "eq",
"not_equal": "ne",
"less": "lt",
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org