You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "xinrong-meng (via GitHub)" <gi...@apache.org> on 2025/06/17 22:42:28 UTC
[PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
xinrong-meng opened a new pull request, #51209:
URL: https://github.com/apache/spark/pull/51209
### What changes were proposed in this pull request?
Enable divide-by-zero for numeric floordiv with ANSI enabled
### Why are the changes needed?
Ensure pandas on Spark works well with ANSI mode on.
Part of https://issues.apache.org/jira/browse/SPARK-52169.
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
Unit tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #51209:
URL: https://github.com/apache/spark/pull/51209#discussion_r2153285549
##########
python/pyspark/pandas/data_type_ops/num_ops.py:
##########
@@ -369,18 +369,28 @@ def floordiv(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
if not is_valid_operand_for_numeric_arithmetic(right):
raise TypeError("Floor division can not be applied to given types.")
+ spark_session = left._internal.spark_frame.sparkSession
def floordiv(left: PySparkColumn, right: Any) -> PySparkColumn:
- return F.when(F.lit(right is np.nan), np.nan).otherwise(
- F.when(
- F.lit(right != 0) | F.lit(right).isNull(),
- F.floor(left.__div__(right)),
- ).otherwise(
- F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
- F.lit(np.inf).__div__(left)
+ if is_ansi_mode_enabled(spark_session):
+ return F.when(F.lit(right is np.nan), F.lit(np.nan)).otherwise(
+ F.when(F.lit(right != 0), F.floor(left / F.lit(right))).otherwise(
+ F.when(
+ F.lit(left == np.inf) | F.lit(left == -np.inf), F.lit(left)
+ ).otherwise(F.lit(np.inf).__div__(left))
+ )
+ )
+ else:
+ return F.when(F.lit(right is np.nan), np.nan).otherwise(
Review Comment:
Same as before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #51209:
URL: https://github.com/apache/spark/pull/51209#discussion_r2155581851
##########
python/pyspark/pandas/data_type_ops/num_ops.py:
##########
@@ -271,13 +271,16 @@ def floordiv(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
if not is_valid_operand_for_numeric_arithmetic(right):
raise TypeError("Floor division can not be applied to given types.")
+ spark_session = left._internal.spark_frame.sparkSession
+ use_try_divide = is_ansi_mode_enabled(spark_session)
+ safe_div = F.try_divide if use_try_divide else lambda x, y: x.__div__(y) # type: ignore
Review Comment:
Please don't use `type: ignore` without error code.
Also we can do:
```suggestion
safe_div: Callable[[PySparkColumn, PySparkColumn], PySparkColumn]
if use_try_divide:
safe_div = F.try_divide
else:
safe_div = lambda x, y: x.__div__(y)
```
This may be too much, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #51209:
URL: https://github.com/apache/spark/pull/51209#discussion_r2159512503
##########
python/pyspark/pandas/data_type_ops/num_ops.py:
##########
@@ -369,6 +376,9 @@ def floordiv(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
if not is_valid_operand_for_numeric_arithmetic(right):
raise TypeError("Floor division can not be applied to given types.")
+ spark_session = left._internal.spark_frame.sparkSession
+ use_try_divide = is_ansi_mode_enabled(spark_session)
+ safe_div = F.try_divide if use_try_divide else lambda x, y: x.__div__(y) # type: ignore
Review Comment:
Sounds good, btw the code is adjusted to avoid E731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on PR #51209:
URL: https://github.com/apache/spark/pull/51209#issuecomment-2992741446
Merged to master, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #51209:
URL: https://github.com/apache/spark/pull/51209#discussion_r2155414121
##########
python/pyspark/pandas/data_type_ops/num_ops.py:
##########
@@ -271,13 +271,14 @@ def floordiv(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
if not is_valid_operand_for_numeric_arithmetic(right):
raise TypeError("Floor division can not be applied to given types.")
+ spark_session = left._internal.spark_frame.sparkSession
def floordiv(left: PySparkColumn, right: Any) -> PySparkColumn:
return F.when(F.lit(right is np.nan), np.nan).otherwise(
F.when(
F.lit(right != 0) | F.lit(right).isNull(),
F.floor(left.__div__(right)),
- ).otherwise(F.lit(np.inf).__div__(left))
+ ).otherwise(F.try_divide(F.lit(np.inf), left))
Review Comment:
Forgot we need to keep the current error, updated thanks!
##########
python/pyspark/pandas/data_type_ops/num_ops.py:
##########
@@ -369,18 +369,28 @@ def floordiv(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
if not is_valid_operand_for_numeric_arithmetic(right):
raise TypeError("Floor division can not be applied to given types.")
+ spark_session = left._internal.spark_frame.sparkSession
def floordiv(left: PySparkColumn, right: Any) -> PySparkColumn:
- return F.when(F.lit(right is np.nan), np.nan).otherwise(
- F.when(
- F.lit(right != 0) | F.lit(right).isNull(),
- F.floor(left.__div__(right)),
- ).otherwise(
- F.when(F.lit(left == np.inf) | F.lit(left == -np.inf), left).otherwise(
- F.lit(np.inf).__div__(left)
+ if is_ansi_mode_enabled(spark_session):
+ return F.when(F.lit(right is np.nan), F.lit(np.nan)).otherwise(
+ F.when(F.lit(right != 0), F.floor(left / F.lit(right))).otherwise(
+ F.when(
+ F.lit(left == np.inf) | F.lit(left == -np.inf), F.lit(left)
+ ).otherwise(F.lit(np.inf).__div__(left))
+ )
+ )
+ else:
+ return F.when(F.lit(right is np.nan), np.nan).otherwise(
Review Comment:
Same as before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng closed pull request #51209: [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled
URL: https://github.com/apache/spark/pull/51209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #51209:
URL: https://github.com/apache/spark/pull/51209#discussion_r2155791221
##########
python/pyspark/pandas/data_type_ops/num_ops.py:
##########
@@ -369,6 +376,9 @@ def floordiv(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
if not is_valid_operand_for_numeric_arithmetic(right):
raise TypeError("Floor division can not be applied to given types.")
+ spark_session = left._internal.spark_frame.sparkSession
+ use_try_divide = is_ansi_mode_enabled(spark_session)
+ safe_div = F.try_divide if use_try_divide else lambda x, y: x.__div__(y) # type: ignore
Review Comment:
ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #51209:
URL: https://github.com/apache/spark/pull/51209#discussion_r2155581851
##########
python/pyspark/pandas/data_type_ops/num_ops.py:
##########
@@ -271,13 +271,16 @@ def floordiv(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
if not is_valid_operand_for_numeric_arithmetic(right):
raise TypeError("Floor division can not be applied to given types.")
+ spark_session = left._internal.spark_frame.sparkSession
+ use_try_divide = is_ansi_mode_enabled(spark_session)
+ safe_div = F.try_divide if use_try_divide else lambda x, y: x.__div__(y) # type: ignore
Review Comment:
Please don't use `type: ignore` without error code.
Also we can do:
```suggestion
safe_div: Callable[[PySparkColumn, PySparkColumn], PySparkColumn]
if use_try_divide:
safe_div = F.try_divide
else:
safe_div = lambda x, y: x.__div__(y)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #51209:
URL: https://github.com/apache/spark/pull/51209#discussion_r2155767207
##########
python/pyspark/pandas/data_type_ops/num_ops.py:
##########
@@ -271,13 +271,16 @@ def floordiv(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
if not is_valid_operand_for_numeric_arithmetic(right):
raise TypeError("Floor division can not be applied to given types.")
+ spark_session = left._internal.spark_frame.sparkSession
+ use_try_divide = is_ansi_mode_enabled(spark_session)
+ safe_div = F.try_divide if use_try_divide else lambda x, y: x.__div__(y) # type: ignore
Review Comment:
It’s much clearer now thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52519][PS] Enable divide-by-zero for numeric floordiv with ANSI enabled [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #51209:
URL: https://github.com/apache/spark/pull/51209#discussion_r2155290002
##########
python/pyspark/pandas/data_type_ops/num_ops.py:
##########
@@ -271,13 +271,14 @@ def floordiv(self, left: IndexOpsLike, right: Any) -> SeriesOrIndex:
_sanitize_list_like(right)
if not is_valid_operand_for_numeric_arithmetic(right):
raise TypeError("Floor division can not be applied to given types.")
+ spark_session = left._internal.spark_frame.sparkSession
def floordiv(left: PySparkColumn, right: Any) -> PySparkColumn:
return F.when(F.lit(right is np.nan), np.nan).otherwise(
F.when(
F.lit(right != 0) | F.lit(right).isNull(),
F.floor(left.__div__(right)),
- ).otherwise(F.lit(np.inf).__div__(left))
+ ).otherwise(F.try_divide(F.lit(np.inf), left))
Review Comment:
Don't we need to check the config here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org