You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/07/13 03:08:30 UTC
[spark] branch branch-3.2 updated: [SPARK-36104][PYTHON] Manage
InternalField in DataTypeOps.neg/abs
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new eae79dd [SPARK-36104][PYTHON] Manage InternalField in DataTypeOps.neg/abs
eae79dd is described below
commit eae79dd31ba7ae6f7a40459a5b146b538ef42d3f
Author: Xinrong Meng <xi...@databricks.com>
AuthorDate: Tue Jul 13 12:07:05 2021 +0900
[SPARK-36104][PYTHON] Manage InternalField in DataTypeOps.neg/abs
### What changes were proposed in this pull request?
Manage InternalField for DataTypeOps.neg/abs.
### Why are the changes needed?
The spark data type and nullability must be the same as the original when DataTypeOps.neg/abs.
We should manage InternalField for this case.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit tests.
Closes #33307 from xinrong-databricks/internalField.
Authored-by: Xinrong Meng <xi...@databricks.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 5afc27f899db7881b69302b94ff996c6f2305ad1)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/pandas/data_type_ops/num_ops.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/pandas/data_type_ops/num_ops.py b/python/pyspark/pandas/data_type_ops/num_ops.py
index 16cb305..94bbd7c 100644
--- a/python/pyspark/pandas/data_type_ops/num_ops.py
+++ b/python/pyspark/pandas/data_type_ops/num_ops.py
@@ -123,10 +123,12 @@ class NumericOps(DataTypeOps):
return column_op(rmod)(left, right)
def neg(self, operand: IndexOpsLike) -> IndexOpsLike:
- return cast(IndexOpsLike, column_op(Column.__neg__)(operand))
+ return operand._with_new_scol(-operand.spark.column, field=operand._internal.data_fields[0])
def abs(self, operand: IndexOpsLike) -> IndexOpsLike:
- return cast(IndexOpsLike, column_op(F.abs)(operand))
+ return operand._with_new_scol(
+ F.abs(operand.spark.column), field=operand._internal.data_fields[0]
+ )
def lt(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
return column_op(Column.__lt__)(left, right)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org