You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "mihailom-db (via GitHub)" <gi...@apache.org> on 2024/11/19 10:36:52 UTC
[PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
mihailom-db opened a new pull request, #48893:
URL: https://github.com/apache/spark/pull/48893
### What changes were proposed in this pull request?
Addition of cases for DayTimeIntervalType.
### Why are the changes needed?
This needs to be supported by documentation.
### Does this PR introduce _any_ user-facing change?
Yes, we enable queries.
### How was this patch tested?
Golden file test added.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1850283662
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
I think we should support `INTERVAL DAY` (internally `INTERVAL DAY TO DAY`) as we do in other places, and fail for other day-time intervals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1850004672
##########
sql/core/src/test/resources/sql-tests/results/window.sql.out:
##########
@@ -1442,3 +1434,19 @@ a 1 2
a NULL 1
b 1 1
b 2 2
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_daterange INTERVAL '5' days PRECEDING) AS mean FROM testData
Review Comment:
what is this test supposed to check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1851000156
##########
sql/core/src/test/resources/sql-tests/results/window.sql.out:
##########
@@ -1442,3 +1442,69 @@ a 1 2
a NULL 1
b 1 1
b 2 2
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '5' DAY PRECEDING) AS mean FROM testData
+-- !query schema
+struct<val:int,val_long:bigint,val_double:double,val_date:date,val_timestamp:timestamp,cate:string,mean:double>
+-- !query output
+1 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+1 2 2.5 2017-08-02 2017-08-05 23:13:20 a 1.5
+1 NULL 1.0 2017-08-01 2017-07-31 17:00:00 b 1.0
+2 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 a 100.001
+2 3 3.3 2017-08-03 2017-08-17 13:00:00 b 3.3
+3 1 1.0 2017-08-01 2017-07-31 17:00:00 NULL 1.0
+3 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 b 100.001
+NULL 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+NULL NULL NULL NULL NULL NULL NULL
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING) AS mean FROM testData
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.RANGE_FRAME_INVALID_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "orderSpecType" : "\"DATE\"",
+ "sqlExpr" : "\"(PARTITION BY val ORDER BY val_date ASC NULLS FIRST RANGE BETWEEN INTERVAL '1 02:03:04.001' DAY TO SECOND PRECEDING AND CURRENT ROW)\"",
+ "valueBoundaryType" : "\"INTERVAL DAY TO SECOND\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 33,
+ "stopIndex" : 121,
+ "fragment" : "(partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING)"
+ } ]
+}
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE DATE '2024-01-01' PRECEDING) AS mean FROM testData
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"DATE '2024-01-01'\"",
+ "inputType" : "\"DATE\"",
+ "paramIndex" : "first",
+ "requiredType" : "(\"NUMERIC\" or \"INTERVAL DAY TO SECOND\" or \"INTERVAL YEAR TO MONTH\" or \"INTERVAL\")",
Review Comment:
It would be nice to output `INTERVAL DAY`, and don't confuse users, but this might be not related to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1849846592
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
we convert a date to a timestamp implicitly:
```sql
> select date'now'+interval '1 10:11:12.123456' day to second;
2024-11-20 10:11:12.123456
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1851248006
##########
sql/core/src/test/resources/sql-tests/inputs/window.sql:
##########
@@ -478,3 +478,7 @@ SELECT * FROM (SELECT cate, val, dense_rank() OVER(PARTITION BY cate ORDER BY va
SELECT * FROM (SELECT cate, val, dense_rank() OVER(PARTITION BY cate ORDER BY val) as r FROM testData) where r <= 2;
SELECT * FROM (SELECT cate, val, row_number() OVER(PARTITION BY cate ORDER BY val) as r FROM testData) where r = 1;
SELECT * FROM (SELECT cate, val, row_number() OVER(PARTITION BY cate ORDER BY val) as r FROM testData) where r <= 2;
+
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '5' DAY PRECEDING) AS mean FROM testData;
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING) AS mean FROM testData;
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE DATE '2024-01-01' FOLLOWING) AS mean FROM testData;
Review Comment:
just for my curiosity: why does this query have a different error condition than the query above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1851008337
##########
sql/core/src/test/resources/sql-tests/results/window.sql.out:
##########
@@ -1442,3 +1442,69 @@ a 1 2
a NULL 1
b 1 1
b 2 2
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '5' DAY PRECEDING) AS mean FROM testData
+-- !query schema
+struct<val:int,val_long:bigint,val_double:double,val_date:date,val_timestamp:timestamp,cate:string,mean:double>
+-- !query output
+1 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+1 2 2.5 2017-08-02 2017-08-05 23:13:20 a 1.5
+1 NULL 1.0 2017-08-01 2017-07-31 17:00:00 b 1.0
+2 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 a 100.001
+2 3 3.3 2017-08-03 2017-08-17 13:00:00 b 3.3
+3 1 1.0 2017-08-01 2017-07-31 17:00:00 NULL 1.0
+3 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 b 100.001
+NULL 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+NULL NULL NULL NULL NULL NULL NULL
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING) AS mean FROM testData
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.RANGE_FRAME_INVALID_TYPE",
Review Comment:
The full error message after substitution confuses slightly:
```
The data type "DATE" used in the order specification does not match the data type "INTERVAL DAY TO SECOND" which is used in the range frame.
```
I think we could improve the error message. @mihailom-db Could you open an JIRA for this, please..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on PR #48893:
URL: https://github.com/apache/spark/pull/48893#issuecomment-2486577919
@cloud-fan @MaxGekk could you review this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1850278379
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
should we fail at runtime if the day-time interval contains non-day fields? We can relax it later (using truncation) if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50379][SQL] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #48893:
URL: https://github.com/apache/spark/pull/48893#issuecomment-2490613168
I believe the failed test is not related to the changes:
```
[info] OracleIntegrationSuite:
[info] org.apache.spark.sql.jdbc.v2.OracleIntegrationSuite *** ABORTED *** (10 minutes, 19 seconds)
[info] The code passed to eventually never returned normally. Attempted 590 times over 10.014576434266667 minutes. Last failure message: ORA-12541: Cannot connect. No listener at host 10.1.0.4 port 40219. (CONNECTION_ID=j6h8+/l4RAezQ6fJKgL7/Q==)
```
+1, LGTM. Merging to master.
Thank you, @mihailom-db and @cloud-fan @srielau for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1848957795
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
I am not sure that discarding the time part of interval is appropriate approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50379][SQL] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1851493591
##########
sql/core/src/test/resources/sql-tests/results/window.sql.out:
##########
@@ -1442,3 +1442,69 @@ a 1 2
a NULL 1
b 1 1
b 2 2
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '5' DAY PRECEDING) AS mean FROM testData
+-- !query schema
+struct<val:int,val_long:bigint,val_double:double,val_date:date,val_timestamp:timestamp,cate:string,mean:double>
+-- !query output
+1 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+1 2 2.5 2017-08-02 2017-08-05 23:13:20 a 1.5
+1 NULL 1.0 2017-08-01 2017-07-31 17:00:00 b 1.0
+2 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 a 100.001
+2 3 3.3 2017-08-03 2017-08-17 13:00:00 b 3.3
+3 1 1.0 2017-08-01 2017-07-31 17:00:00 NULL 1.0
+3 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 b 100.001
+NULL 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+NULL NULL NULL NULL NULL NULL NULL
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING) AS mean FROM testData
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.RANGE_FRAME_INVALID_TYPE",
Review Comment:
Changed to does not support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1849135272
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
I'm missing context what is this about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1849137640
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
I need to fix tests, but basically in windowFrame if we have DayTimeIntervalType for start_frame, and date as current column, we will fail instead of doing the window function properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1848961394
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
@mihailom-db BTW, what does the ANSI SQL standard says about that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1850303737
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
ah good point, we have this information in the data type, we should only support INTERVAL DAY TO DAY
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1850002935
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
Here @mihailom-db proposes to discard the time part at all, and get only days, for example above:
```sql
> select date'now'+interval '1 10:11:12.123456' day to second;
2024-11-20
```
Probably, it is ok in this case. @mihailom-db Could you add more tests at least for the cases:
```
| frameType=RANGE BETWEEN start=frameBound AND end=frameBound
| frameType=ROWS BETWEEN start=frameBound AND end=frameBound
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50379][SQL] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #48893: [SPARK-50379][SQL] Fix DayTimeIntevalType handling in WindowExecBase
URL: https://github.com/apache/spark/pull/48893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1848992964
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
@MaxGekk just checked. This is the right approach I would say. We are not trying to change addition for Date + Interval here. We have analyser rule that converts add(date, interval) to this same expression as in my PR. We are just adding a case to properly handle case when we need to add interval to a date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1848994030
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
Also, date does not have hours, mins and seconds fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1848971372
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
Hm, true, I missed that this is might be start-end interval and not only start. Will rework it now. As for ANSI SQL I am not sure. @srielau do you have info on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1849530894
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
are you saying that Spark already supports `date + day_time_interval`? what's the behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1849859430
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala:
##########
@@ -101,6 +101,8 @@ trait WindowEvaluatorFactoryBase {
val boundExpr = (expr.dataType, boundOffset.dataType) match {
case (DateType, IntegerType) => DateAdd(expr, boundOffset)
case (DateType, _: YearMonthIntervalType) => DateAddYMInterval(expr, boundOffset)
+ case (DateType, _: DayTimeIntervalType) =>
+ DateAdd(expr, ExtractANSIIntervalDays(boundOffset))
Review Comment:
which is different from what we do here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1851047679
##########
sql/core/src/test/resources/sql-tests/results/window.sql.out:
##########
@@ -1442,3 +1442,69 @@ a 1 2
a NULL 1
b 1 1
b 2 2
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '5' DAY PRECEDING) AS mean FROM testData
+-- !query schema
+struct<val:int,val_long:bigint,val_double:double,val_date:date,val_timestamp:timestamp,cate:string,mean:double>
+-- !query output
+1 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+1 2 2.5 2017-08-02 2017-08-05 23:13:20 a 1.5
+1 NULL 1.0 2017-08-01 2017-07-31 17:00:00 b 1.0
+2 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 a 100.001
+2 3 3.3 2017-08-03 2017-08-17 13:00:00 b 3.3
+3 1 1.0 2017-08-01 2017-07-31 17:00:00 NULL 1.0
+3 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 b 100.001
+NULL 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+NULL NULL NULL NULL NULL NULL NULL
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING) AS mean FROM testData
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.RANGE_FRAME_INVALID_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "orderSpecType" : "\"DATE\"",
+ "sqlExpr" : "\"(PARTITION BY val ORDER BY val_date ASC NULLS FIRST RANGE BETWEEN INTERVAL '1 02:03:04.001' DAY TO SECOND PRECEDING AND CURRENT ROW)\"",
+ "valueBoundaryType" : "\"INTERVAL DAY TO SECOND\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 33,
+ "stopIndex" : 121,
+ "fragment" : "(partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING)"
+ } ]
+}
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE DATE '2024-01-01' PRECEDING) AS mean FROM testData
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"DATE '2024-01-01'\"",
+ "inputType" : "\"DATE\"",
+ "paramIndex" : "first",
+ "requiredType" : "(\"NUMERIC\" or \"INTERVAL DAY TO SECOND\" or \"INTERVAL YEAR TO MONTH\" or \"INTERVAL\")",
Review Comment:
In this case this is failing because we cannot do UnaryMinus(DATE), I changed the error to reflect problems with DATE+DATE. The problem with printing is that it is not an easy change. The requiredType is acquired from input types we have in the expression, and DayTimeIntervalType uses INTERVAL DAY TO SECOND.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1851048420
##########
sql/core/src/test/resources/sql-tests/results/window.sql.out:
##########
@@ -1442,3 +1442,69 @@ a 1 2
a NULL 1
b 1 1
b 2 2
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '5' DAY PRECEDING) AS mean FROM testData
+-- !query schema
+struct<val:int,val_long:bigint,val_double:double,val_date:date,val_timestamp:timestamp,cate:string,mean:double>
+-- !query output
+1 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+1 2 2.5 2017-08-02 2017-08-05 23:13:20 a 1.5
+1 NULL 1.0 2017-08-01 2017-07-31 17:00:00 b 1.0
+2 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 a 100.001
+2 3 3.3 2017-08-03 2017-08-17 13:00:00 b 3.3
+3 1 1.0 2017-08-01 2017-07-31 17:00:00 NULL 1.0
+3 2147483650 100.001 2020-12-31 2020-12-30 16:00:00 b 100.001
+NULL 1 1.0 2017-08-01 2017-07-31 17:00:00 a 1.0
+NULL NULL NULL NULL NULL NULL NULL
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING) AS mean FROM testData
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.RANGE_FRAME_INVALID_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "orderSpecType" : "\"DATE\"",
+ "sqlExpr" : "\"(PARTITION BY val ORDER BY val_date ASC NULLS FIRST RANGE BETWEEN INTERVAL '1 02:03:04.001' DAY TO SECOND PRECEDING AND CURRENT ROW)\"",
+ "valueBoundaryType" : "\"INTERVAL DAY TO SECOND\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 33,
+ "stopIndex" : 121,
+ "fragment" : "(partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING)"
+ } ]
+}
+
+
+-- !query
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE DATE '2024-01-01' PRECEDING) AS mean FROM testData
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"DATE '2024-01-01'\"",
+ "inputType" : "\"DATE\"",
+ "paramIndex" : "first",
+ "requiredType" : "(\"NUMERIC\" or \"INTERVAL DAY TO SECOND\" or \"INTERVAL YEAR TO MONTH\" or \"INTERVAL\")",
Review Comment:
We might need to update what we print for DayTimeIntervalType, as it is not always INTERVAL DAY TO SECOND, but it always will include INTERVAL DAY, so we might need to scope it down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1851473484
##########
sql/core/src/test/resources/sql-tests/inputs/window.sql:
##########
@@ -478,3 +478,7 @@ SELECT * FROM (SELECT cate, val, dense_rank() OVER(PARTITION BY cate ORDER BY va
SELECT * FROM (SELECT cate, val, dense_rank() OVER(PARTITION BY cate ORDER BY val) as r FROM testData) where r <= 2;
SELECT * FROM (SELECT cate, val, row_number() OVER(PARTITION BY cate ORDER BY val) as r FROM testData) where r = 1;
SELECT * FROM (SELECT cate, val, row_number() OVER(PARTITION BY cate ORDER BY val) as r FROM testData) where r <= 2;
+
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '5' DAY PRECEDING) AS mean FROM testData;
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING) AS mean FROM testData;
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE DATE '2024-01-01' FOLLOWING) AS mean FROM testData;
Review Comment:
This is because we cannot do DATE+DATE. We need to pass a valid rangeType, which is numeric or interval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-35110][SQL][FOLLOW-UP] Fix DayTimeIntevalType handling in WindowExecBase [spark]
Posted by "mihailom-db (via GitHub)" <gi...@apache.org>.
mihailom-db commented on code in PR #48893:
URL: https://github.com/apache/spark/pull/48893#discussion_r1851474155
##########
sql/core/src/test/resources/sql-tests/inputs/window.sql:
##########
@@ -478,3 +478,7 @@ SELECT * FROM (SELECT cate, val, dense_rank() OVER(PARTITION BY cate ORDER BY va
SELECT * FROM (SELECT cate, val, dense_rank() OVER(PARTITION BY cate ORDER BY val) as r FROM testData) where r <= 2;
SELECT * FROM (SELECT cate, val, row_number() OVER(PARTITION BY cate ORDER BY val) as r FROM testData) where r = 1;
SELECT * FROM (SELECT cate, val, row_number() OVER(PARTITION BY cate ORDER BY val) as r FROM testData) where r <= 2;
+
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '5' DAY PRECEDING) AS mean FROM testData;
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE INTERVAL '1 2:3:4.001' DAY TO SECOND PRECEDING) AS mean FROM testData;
+SELECT *, mean(val_double) over (partition BY val ORDER BY val_date RANGE DATE '2024-01-01' FOLLOWING) AS mean FROM testData;
Review Comment:
Valid rangeType is something that can be added to Date type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org