You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mzabaluev (via GitHub)" <gi...@apache.org> on 2025/12/17 17:56:12 UTC
[PR] fix(spark): Use wrapping addition/subtraction in `SparkDateAdd` and `SparkDateSub` [datafusion]
mzabaluev opened a new pull request, #19377:
URL: https://github.com/apache/datafusion/pull/19377
## Which issue does this PR close?
- Addresses [#17987(comment)](https://github.com/apache/datafusion/issues/17987#issuecomment-3666497393).
## Rationale for this change
In Spark, `date_add` and `date_sub` functions do not raise an error when the addition or subtraction overflows 32-bit integer, regardless of whether the ANSI evaluation mode is in effect.
## What changes are included in this PR?
Changed the implementations to use `wrapping_add` and `wrapping_sub`.
## Are these changes tested?
Updated `date_add.slt` to expect overflowed results rather than the errors.
## Are there any user-facing changes?
The functions behave more like Spark, even if Spark is doing the wrong thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(spark): Use wrapping addition/subtraction in `SparkDateAdd` and `SparkDateSub` [datafusion]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #19377:
URL: https://github.com/apache/datafusion/pull/19377#discussion_r2634810269
##########
datafusion/sqllogictest/test_files/spark/datetime/date_add.slt:
##########
@@ -45,11 +45,15 @@ SELECT date_sub('2016-07-30'::date, 0::int);
----
2016-07-30
-query error DataFusion error: Arrow error: Arithmetic overflow: date_add
-SELECT date_add('2016-07-30'::date, 2147483647::int);
+query I
+SELECT date_add('2016-07-30'::date, 2147483647::int)::int;
+----
+-2147466637
-query error DataFusion error: Arrow error: Arithmetic overflow: date_sub
-SELECT date_sub('1969-01-01'::date, 2147483647::int);
+query I
+SELECT date_sub('1969-01-01'::date, 2147483647::int)::int;
+----
+2147483284
Review Comment:
Apache Spark 4.0.1 returns the same results:
```
❯ ./bin/spark-sql
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
25/12/19 13:27:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/12/19 13:27:41 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
25/12/19 13:27:41 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore mgrigorov@192.168.x.y
Spark Web UI available at http://localhost:4040
Spark master: local[*], Application Id: local-1766143660589
spark-sql (default)> SELECT UNIX_DATE(date_add('2016-07-30', 2147483647));
-2147466637
Time taken: 0.028 seconds, Fetched 1 row(s)
spark-sql (default)> SELECT UNIX_DATE(date_sub('1969-01-01', 2147483647));
2147483284
Time taken: 0.03 seconds, Fetched 1 row(s)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(spark): Use wrapping addition/subtraction in `SparkDateAdd` and `SparkDateSub` [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #19377:
URL: https://github.com/apache/datafusion/pull/19377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(spark): Use wrapping addition/subtraction in `SparkDateAdd` and `SparkDateSub` [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #19377:
URL: https://github.com/apache/datafusion/pull/19377#issuecomment-3706635355
cc @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(spark): Use wrapping addition/subtraction in `SparkDateAdd` and `SparkDateSub` [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #19377:
URL: https://github.com/apache/datafusion/pull/19377#issuecomment-3712704029
Thanks @mzabaluev & @martin-g
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org