You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/03 10:27:37 UTC
[GitHub] [spark] beliefer opened a new pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
beliefer opened a new pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440
### What changes were proposed in this pull request?
The `NTH_VALUE` function is an ANSI SQL.
There are some mainstream database support the syntax.
**PostgreSQL:**
https://www.postgresql.org/docs/8.4/functions-window.html
**Vertica:**
https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/SQLReferenceManual/Functions/Analytic/NTH_VALUEAnalytic.htm?tocpath=SQL%20Reference%20Manual%7CSQL%20Functions%7CAnalytic%20Functions%7C_____23
**Oracle:**
https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/NTH_VALUE.html#GUID-F8A0E88C-67E5-4AA6-9515-95D03A7F9EA0
### Why are the changes needed?
The `NTH_VALUE` function is an ANSI SQL.
The `NTH_VALUE` function is very useful.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists and new UT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-583996156
**[Test build #118128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118128/testReport)** for PR 27440 at commit [`5b3caa9`](https://github.com/apache/spark/commit/5b3caa9bcfc3fa20ea09e6ab470cd153a49cd0f2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584018801
**[Test build #118146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118146/testReport)** for PR 27440 at commit [`5b3caa9`](https://github.com/apache/spark/commit/5b3caa9bcfc3fa20ea09e6ab470cd153a49cd0f2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376885203
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
+ case i: Int if i <= 0 => TypeCheckFailure(
+ s"The 'offset' argument of nth_value must be greater than zero but it is $i.")
Review comment:
'offset' must greater or equal to zero.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376885203
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
+ case i: Int if i <= 0 => TypeCheckFailure(
+ s"The 'offset' argument of nth_value must be greater than zero but it is $i.")
Review comment:
'offset' must greater than zero.
The description of Redshift
```
offset
Determines the row number relative to the first row in the window for which to return the expression. The offset can be a constant or an expression and must be a positive integer that is greater than 0.
```
The description of Vertica
`row‑number | Specifies the row to evaluate, where row‑number evaluates to an integer ≥ 1.`
The description of Presto
` It is an error for the offset to be zero or negative.`
The description of Oracle
`n determines the nth row for which the measure value is to be returned. n can be a constant, bind variable, column, or an expression involving them, as long as it resolves to a positive integer. The function returns NULL if the data source window has fewer than n rows. If n is null, then the function returns an error.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer edited a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer edited a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-587980220
@hvanhovell Could you take a look again? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r377543013
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -363,6 +363,11 @@ abstract class OffsetWindowFunction
*/
val direction: SortDirection
+ /**
+ * Whether the offset is based on the entire frame.
+ */
+ val isWholeBased: Boolean = false
Review comment:
`LEAD/LAG` not support `IGNORE NULLS` now. Maybe the current implement only considered `postgreSQL`. So I think as long as `NTH_VALUE/FIRST_VALUE/LAST_VALUE` use the same way as `LEAD/LAG`, once the time is right, we can uniformly provide them with support for `IGNORE NULLS`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584000828
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118128/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581698710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22556/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585232852
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581405180
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585119793
**[Test build #118290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118290/testReport)** for PR 27440 at commit [`714b6d3`](https://github.com/apache/spark/commit/714b6d373b80c6ca85429f3e7d02b7381baefef8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585120297
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585212315
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585232852
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581405194
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117768/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585120297
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584555242
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22989/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581698341
**[Test build #117793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117793/testReport)** for PR 27440 at commit [`c87069f`](https://github.com/apache/spark/commit/c87069f42e54f93bf2a8d2c5abe819681af516d7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581698341
**[Test build #117793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117793/testReport)** for PR 27440 at commit [`c87069f`](https://github.com/apache/spark/commit/c87069f42e54f93bf2a8d2c5abe819681af516d7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376786736
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -363,6 +363,11 @@ abstract class OffsetWindowFunction
*/
val direction: SortDirection
+ /**
+ * Whether the offset is based on the entire frame.
+ */
+ val isWholeBased: Boolean = false
Review comment:
Just special case the NTH_VALUE aggregate function instead of doing this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376787047
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -157,6 +157,38 @@ final class OffsetWindowFunctionFrame(
override def currentUpperBound(): Int = throw new UnsupportedOperationException()
}
+class FixedOffsetWindowFunctionFrame(
+ target: InternalRow,
+ ordinal: Int,
+ expressions: Array[OffsetWindowFunction],
+ inputSchema: Seq[Attribute],
+ newMutableProjection: (Seq[Expression], Seq[Attribute]) => MutableProjection,
+ offset: Int)
+ extends OffsetWindowFunctionFrame(
Review comment:
See my comment below. I don't think you should extend this, if you need some functionality just put in in a common base class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584554698
**[Test build #118229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118229/testReport)** for PR 27440 at commit [`4b3d80b`](https://github.com/apache/spark/commit/4b3d80bac17ba2b1b7a225ea27117d3f008cdaa5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585212321
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118301/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581748503
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584661417
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584019285
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r377531726
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -363,6 +363,11 @@ abstract class OffsetWindowFunction
*/
val direction: SortDirection
+ /**
+ * Whether the offset is based on the entire frame.
+ */
+ val isWholeBased: Boolean = false
Review comment:
Why would we use this for `FIRST` and `LAST`? How would you deal with `IGNORE NULLS`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584019295
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22908/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585212321
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118301/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376786807
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
Review comment:
add a check to make sure it is foldable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-583994237
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584148277
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118146/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585193279
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584555242
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22989/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584660131
**[Test build #118229 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118229/testReport)** for PR 27440 at commit [`4b3d80b`](https://github.com/apache/spark/commit/4b3d80bac17ba2b1b7a225ea27117d3f008cdaa5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376879241
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -363,6 +363,11 @@ abstract class OffsetWindowFunction
*/
val direction: SortDirection
+ /**
+ * Whether the offset is based on the entire frame.
+ */
+ val isWholeBased: Boolean = false
Review comment:
I want `NTH_VALUE/FIRST_VALUE/LAST_VALUE` use this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376885203
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
+ case i: Int if i <= 0 => TypeCheckFailure(
+ s"The 'offset' argument of nth_value must be greater than zero but it is $i.")
Review comment:
'offset' must greater than zero.
The description of Redshift
```
offset
Determines the row number relative to the first row in the window for which to return the expression. The offset can be a constant or an expression and must be a positive integer that is greater than 0.
```
The description of Vertica
`row‑number | Specifies the row to evaluate, where row‑number evaluates to an integer ≥ 1.`
The description of Presto
` It is an error for the offset to be zero or negative.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585119793
**[Test build #118290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118290/testReport)** for PR 27440 at commit [`714b6d3`](https://github.com/apache/spark/commit/714b6d373b80c6ca85429f3e7d02b7381baefef8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585192841
**[Test build #118301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118301/testReport)** for PR 27440 at commit [`591c146`](https://github.com/apache/spark/commit/591c146cdbb71b7bcf771243bcc2b2e8c7792fa1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585212234
**[Test build #118301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118301/testReport)** for PR 27440 at commit [`591c146`](https://github.com/apache/spark/commit/591c146cdbb71b7bcf771243bcc2b2e8c7792fa1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585231598
**[Test build #118290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118290/testReport)** for PR 27440 at commit [`714b6d3`](https://github.com/apache/spark/commit/714b6d373b80c6ca85429f3e7d02b7381baefef8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class OffsetWindowFunctionFrameBase(`
* `class OffsetWindowFunctionFrame(`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581698704
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581344791
**[Test build #117768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117768/testReport)** for PR 27440 at commit [`2ee523b`](https://github.com/apache/spark/commit/2ee523be2b6c41ac480bab90652cd326fd782d7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581748509
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117793/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376786642
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -157,6 +157,38 @@ final class OffsetWindowFunctionFrame(
override def currentUpperBound(): Int = throw new UnsupportedOperationException()
}
+class FixedOffsetWindowFunctionFrame(
+ target: InternalRow,
+ ordinal: Int,
+ expressions: Array[OffsetWindowFunction],
+ inputSchema: Seq[Attribute],
+ newMutableProjection: (Seq[Expression], Seq[Attribute]) => MutableProjection,
+ offset: Int)
+ extends OffsetWindowFunctionFrame(
+ target, ordinal, expressions, inputSchema, newMutableProjection, offset) {
+
+ override def prepare(rows: ExternalAppendOnlyUnsafeRowArray): Unit = {
+ input = rows
+ inputIndex = 0
+ }
+
+ override def write(index: Int, current: InternalRow): Unit = {
Review comment:
NTH_VALUE should return the same value for all rows in the window partition right? So why are you doing so much heavy lifting here? Everything can be computed in prepare.
If you think about it, then this could also be treated as an unbounded window frame. You could even move this into the `UnboundedWindowFunctionFrame` if you add the update logic to the NTH_VALUE aggregate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584000768
**[Test build #118128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118128/testReport)** for PR 27440 at commit [`5b3caa9`](https://github.com/apache/spark/commit/5b3caa9bcfc3fa20ea09e6ab470cd153a49cd0f2).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581748126
**[Test build #117793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117793/testReport)** for PR 27440 at commit [`c87069f`](https://github.com/apache/spark/commit/c87069f42e54f93bf2a8d2c5abe819681af516d7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584148259
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581698710
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22556/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376889490
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -157,6 +157,38 @@ final class OffsetWindowFunctionFrame(
override def currentUpperBound(): Int = throw new UnsupportedOperationException()
}
+class FixedOffsetWindowFunctionFrame(
+ target: InternalRow,
+ ordinal: Int,
+ expressions: Array[OffsetWindowFunction],
+ inputSchema: Seq[Attribute],
+ newMutableProjection: (Seq[Expression], Seq[Attribute]) => MutableProjection,
+ offset: Int)
+ extends OffsetWindowFunctionFrame(
+ target, ordinal, expressions, inputSchema, newMutableProjection, offset) {
+
+ override def prepare(rows: ExternalAppendOnlyUnsafeRowArray): Unit = {
+ input = rows
+ inputIndex = 0
+ }
+
+ override def write(index: Int, current: InternalRow): Unit = {
Review comment:
Because the behavior of `NTH_VALUE/FIRST_VALUE/LAST_VALUE` more similar to `LEAD/LAG`, so I make `FixedOffsetWindowFunctionFrame` extends `OffsetWindowFunctionFrame`. `OffsetWindowFunctionFrame` only to fetch projects but `UnboundedWindowFunctionFrame` will `update` and `execute` expressions.
If you think so, I can abstract a base class for `FixedOffsetWindowFunctionFrame` and `OffsetWindowFunctionFrame`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376887001
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -157,6 +157,38 @@ final class OffsetWindowFunctionFrame(
override def currentUpperBound(): Int = throw new UnsupportedOperationException()
}
+class FixedOffsetWindowFunctionFrame(
Review comment:
I want reference the unit test for `OffsetWindowFunctionFrame`, can you tell me where they are?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584554698
**[Test build #118229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118229/testReport)** for PR 27440 at commit [`4b3d80b`](https://github.com/apache/spark/commit/4b3d80bac17ba2b1b7a225ea27117d3f008cdaa5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r377539183
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -363,6 +363,11 @@ abstract class OffsetWindowFunction
*/
val direction: SortDirection
+ /**
+ * Whether the offset is based on the entire frame.
+ */
+ val isWholeBased: Boolean = false
Review comment:
`FIRST` and `LAST` is aggregate function, there were PRs who used them as `FIRST_VALUE/LAST_VALUE`.
You can reference https://github.com/apache/spark/pull/25082.
The above mentioned PR has been reverted.
All of LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE need `IGNORE NULLS`, I think we should handle in a consistent way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581344791
**[Test build #117768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117768/testReport)** for PR 27440 at commit [`2ee523b`](https://github.com/apache/spark/commit/2ee523be2b6c41ac480bab90652cd326fd782d7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581345453
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22530/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585193281
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23059/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584000811
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584000828
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118128/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581345439
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581748503
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581405180
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584148259
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585232864
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118290/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581748509
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117793/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r377539183
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -363,6 +363,11 @@ abstract class OffsetWindowFunction
*/
val direction: SortDirection
+ /**
+ * Whether the offset is based on the entire frame.
+ */
+ val isWholeBased: Boolean = false
Review comment:
`FIRST` and `LAST` is aggregate function, there were PRs who used them as `FIRST_VALUE/LAST_VALUE`.
You can reference https://github.com/apache/spark/pull/25082.
The above mentioned PR has been reverted.
All of `LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE` need `IGNORE NULLS`, I think we should handle in a consistent way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376884949
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
Review comment:
Ok. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r377539183
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -363,6 +363,11 @@ abstract class OffsetWindowFunction
*/
val direction: SortDirection
+ /**
+ * Whether the offset is based on the entire frame.
+ */
+ val isWholeBased: Boolean = false
Review comment:
`FIRST` and `LAST` is aggregate function, there were PRs who used them as `FIRST_VALUE/LAST_VALUE`.
You can reference https://github.com/apache/spark/pull/25082.
All of LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE need `IGNORE NULLS`, I think we should handle in a consistent way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584019285
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584018801
**[Test build #118146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118146/testReport)** for PR 27440 at commit [`5b3caa9`](https://github.com/apache/spark/commit/5b3caa9bcfc3fa20ea09e6ab470cd153a49cd0f2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585193281
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23059/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585120302
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23048/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584019295
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22908/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376885203
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
+ case i: Int if i <= 0 => TypeCheckFailure(
+ s"The 'offset' argument of nth_value must be greater than zero but it is $i.")
Review comment:
No, 'offset' must greater than zero.
The description of Redshift
```
offset
Determines the row number relative to the first row in the window for which to return the expression. The offset can be a constant or an expression and must be a positive integer that is greater than 0.
```
The description of Vertica
`row‑number | Specifies the row to evaluate, where row‑number evaluates to an integer ≥ 1.`
The description of Presto
` It is an error for the offset to be zero or negative.`
The description of Oracle
`n determines the nth row for which the measure value is to be returned. n can be a constant, bind variable, column, or an expression involving them, as long as it resolves to a positive integer. The function returns NULL if the data source window has fewer than n rows. If n is null, then the function returns an error.`
The description of Postgresql
`returns value evaluated at the row that is the nth row of the window frame (counting from 1); null if no such row`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584661424
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118229/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584148277
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118146/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376885203
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
+ case i: Int if i <= 0 => TypeCheckFailure(
+ s"The 'offset' argument of nth_value must be greater than zero but it is $i.")
Review comment:
'offset' must greater than zero.
The description of Redshift
```
offset
Determines the row number relative to the first row in the window for which to return the expression. The offset can be a constant or an expression and must be a positive integer that is greater than 0.
```
The description of Vertica
`row‑number | Specifies the row to evaluate, where row‑number evaluates to an integer ≥ 1.`
row‑number | Specifies the row to evaluate, where row‑number evaluates to an integer ≥ 1.
-- | --
row‑number | Specifies the row to evaluate, where row‑number evaluates to an integer ≥ 1.
-- | --
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581698704
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584661417
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584147385
**[Test build #118146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118146/testReport)** for PR 27440 at commit [`5b3caa9`](https://github.com/apache/spark/commit/5b3caa9bcfc3fa20ea09e6ab470cd153a49cd0f2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584555231
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376889551
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -157,6 +157,38 @@ final class OffsetWindowFunctionFrame(
override def currentUpperBound(): Int = throw new UnsupportedOperationException()
}
+class FixedOffsetWindowFunctionFrame(
+ target: InternalRow,
+ ordinal: Int,
+ expressions: Array[OffsetWindowFunction],
+ inputSchema: Seq[Attribute],
+ newMutableProjection: (Seq[Expression], Seq[Attribute]) => MutableProjection,
+ offset: Int)
+ extends OffsetWindowFunctionFrame(
Review comment:
I see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376787181
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -157,6 +157,38 @@ final class OffsetWindowFunctionFrame(
override def currentUpperBound(): Int = throw new UnsupportedOperationException()
}
+class FixedOffsetWindowFunctionFrame(
Review comment:
Add documentation.
I would like to see a unit test for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585120302
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23048/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-587980220
@hvanhovell Could you take a look again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584017347
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584661424
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118229/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585193279
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581405194
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117768/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376885203
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
+ case i: Int if i <= 0 => TypeCheckFailure(
+ s"The 'offset' argument of nth_value must be greater than zero but it is $i.")
Review comment:
'offset' must greater than zero.
The description of Redshift
```
offset
Determines the row number relative to the first row in the window for which to return the expression. The offset can be a constant or an expression and must be a positive integer that is greater than 0.
```
The description of Vertica
`row‑number | Specifies the row to evaluate, where row‑number evaluates to an integer ≥ 1.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581872977
cc @gatorsmile @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-583994237
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r377551709
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -157,6 +157,38 @@ final class OffsetWindowFunctionFrame(
override def currentUpperBound(): Int = throw new UnsupportedOperationException()
}
+class FixedOffsetWindowFunctionFrame(
+ target: InternalRow,
+ ordinal: Int,
+ expressions: Array[OffsetWindowFunction],
+ inputSchema: Seq[Attribute],
+ newMutableProjection: (Seq[Expression], Seq[Attribute]) => MutableProjection,
+ offset: Int)
+ extends OffsetWindowFunctionFrame(
+ target, ordinal, expressions, inputSchema, newMutableProjection, offset) {
+
+ override def prepare(rows: ExternalAppendOnlyUnsafeRowArray): Unit = {
+ input = rows
+ inputIndex = 0
+ }
+
+ override def write(index: Int, current: InternalRow): Unit = {
Review comment:
I have optimized the code so that the skip operator runs once in `prepare`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376887001
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -157,6 +157,38 @@ final class OffsetWindowFunctionFrame(
override def currentUpperBound(): Int = throw new UnsupportedOperationException()
}
+class FixedOffsetWindowFunctionFrame(
Review comment:
I want reference the unit test for `OffsetWindowFunctionFrame`, can you tell me where they are? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585232864
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118290/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-583996156
**[Test build #118128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118128/testReport)** for PR 27440 at commit [`5b3caa9`](https://github.com/apache/spark/commit/5b3caa9bcfc3fa20ea09e6ab470cd153a49cd0f2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-583994242
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22889/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584000811
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585212315
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376786764
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
+ case i: Int if i <= 0 => TypeCheckFailure(
+ s"The 'offset' argument of nth_value must be greater than zero but it is $i.")
Review comment:
greater or equal to?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#discussion_r376885203
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala
##########
@@ -465,6 +470,47 @@ case class Lag(input: Expression, offset: Expression, default: Expression)
override val direction = Descending
}
+/**
+ * The NthValue function returns the value of `input` at the row that is the `offset`th row of
+ * the window frame (counting from 1). Offsets start at 0, which is the current row. When the
+ * value of `input` is null at the `offset`th row or there is no such an `offset`th row, null
+ * is returned.
+ *
+ * @param input expression to evaluate `offset`th row of the window frame.
+ * @param offset rows to jump ahead in the partition.
+ */
+@ExpressionDescription(
+ usage = """
+ _FUNC_(input[, offset]) - Returns the value of `input` at the row that is the`offset`th row
+ of the window frame (counting from 1). If the value of `input` at the `offset`th row is
+ null, null is returned. If there is no such an offset row (e.g., when the offset is 10,
+ size of the window frame less than 10), null is returned.
+ """)
+case class NthValue(input: Expression, offset: Expression)
+ extends OffsetWindowFunction {
+
+ override val default = Literal(null)
+
+ override val isWholeBased = true
+
+ override val direction = Ascending
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val check = super.checkInputDataTypes()
+ if (check.isFailure) {
+ check
+ } else {
+ offset.eval() match {
+ case i: Int if i <= 0 => TypeCheckFailure(
+ s"The 'offset' argument of nth_value must be greater than zero but it is $i.")
Review comment:
'offset' must greater than zero.
The description of Redshift
```
offset
Determines the row number relative to the first row in the window for which to return the expression. The offset can be a constant or an expression and must be a positive integer that is greater than 0.
```
The description of Vertica
`row‑number | Specifies the row to evaluate, where row‑number evaluates to an integer ≥ 1.`
The description of Presto
` It is an error for the offset to be zero or negative.`
The description of Oracle
`n determines the nth row for which the measure value is to be returned. n can be a constant, bind variable, column, or an expression involving them, as long as it resolves to a positive integer. The function returns NULL if the data source window has fewer than n rows. If n is null, then the function returns an error.`
The description of Postgresql
`returns value evaluated at the row that is the nth row of the window frame (counting from 1); null if no such row`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-583994242
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22889/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581345439
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-585192841
**[Test build #118301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118301/testReport)** for PR 27440 at commit [`591c146`](https://github.com/apache/spark/commit/591c146cdbb71b7bcf771243bcc2b2e8c7792fa1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581345453
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22530/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27440:
[SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-584555231
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27440: [SPARK-27951][SQL]
Support ANSI SQL NTH_VALUE window function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27440: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function
URL: https://github.com/apache/spark/pull/27440#issuecomment-581404917
**[Test build #117768 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117768/testReport)** for PR 27440 at commit [`2ee523b`](https://github.com/apache/spark/commit/2ee523be2b6c41ac480bab90652cd326fd782d7c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class NthValue(input: Expression, offset: Expression)`
* `class OffsetWindowFunctionFrame(`
* `class FixedOffsetWindowFunctionFrame(`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org