You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/28 06:36:44 UTC
[GitHub] [spark] beliefer opened a new pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
beliefer opened a new pull request #30943:
URL: https://github.com/apache/spark/pull/30943
### What changes were proposed in this pull request?
All of `LEAD`/`LAG`/`NTH_VALUE`/`FIRST_VALUE`/`LAST_VALUE` should support IGNORE NULLS | RESPECT NULLS. For example:
```
LEAD (value_expr [, offset ])
[ IGNORE NULLS | RESPECT NULLS ]
OVER ( [ PARTITION BY window_partition ] ORDER BY window_ordering )
```
```
LAG (value_expr [, offset ])
[ IGNORE NULLS | RESPECT NULLS ]
OVER ( [ PARTITION BY window_partition ] ORDER BY window_ordering )
```
```
NTH_VALUE (expr, offset)
[ IGNORE NULLS | RESPECT NULLS ]
OVER
( [ PARTITION BY window_partition ]
[ ORDER BY window_ordering
frame_clause ] )
```
The mainstream database or engine supports this syntax contains:
**Oracle**
https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/NTH_VALUE.html#GUID-F8A0E88C-67E5-4AA6-9515-95D03A7F9EA0
**Redshift**
https://docs.aws.amazon.com/redshift/latest/dg/r_WF_NTH.html
**Presto**
https://prestodb.io/docs/current/functions/window.html
**DB2**
https://www.ibm.com/support/knowledgecenter/SSGU8G_14.1.0/com.ibm.sqls.doc/ids_sqs_1513.htm
**Teradata**
https://docs.teradata.com/r/756LNiPSFdY~4JcCCcR5Cw/GjCT6l7trjkIEjt~7Dhx4w
**Snowflake**
https://docs.snowflake.com/en/sql-reference/functions/lead.html
https://docs.snowflake.com/en/sql-reference/functions/lag.html
### Why are the changes needed?
Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE is very useful.
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
Jenkins test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752353572
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133518/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752382292
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38116/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752389677
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38114/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549937828
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2123,19 +2123,58 @@ class Analyzer(override val catalogManager: CatalogManager)
if (isDistinct || filter.isDefined) {
throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
wf.prettyName)
+ } else if (ignoreNulls) {
+ wf match {
+ case nthValue: NthValue =>
+ nthValue.copy(ignoreNulls = ignoreNulls)
+ case _ =>
+ throw QueryCompilationErrors.ignoreNullsWithUnsupportedFunctionError(
+ wf.prettyName)
+ }
} else {
wf
}
+ case owf: FrameLessOffsetWindowFunction =>
+ if (isDistinct || filter.isDefined) {
+ throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
+ owf.prettyName)
+ } else if (ignoreNulls) {
+ owf match {
+ case lead: Lead =>
+ lead.copy(ignoreNulls = ignoreNulls)
+ case lag: Lag =>
+ lag.copy(ignoreNulls = ignoreNulls)
+ }
+ } else {
+ owf
+ }
// We get an aggregate function, we need to wrap it in an AggregateExpression.
case agg: AggregateFunction =>
if (filter.isDefined && !filter.get.deterministic) {
throw QueryCompilationErrors.nonDeterministicFilterInAggregateError
}
- AggregateExpression(agg, Complete, isDistinct, filter)
+ if (ignoreNulls) {
+ agg match {
Review comment:
nit:
```
val aggFunc = agg match {
case first: First => first.copy(ignoreNulls = ignoreNulls)
...
}
AggregateExpression(aggFunc, Complete, isDistinct, filter)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751738289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133444/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751675813
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38033/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751675802
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38033/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549943234
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/QueryCompilationErrors.scala
##########
@@ -268,6 +268,11 @@ object QueryCompilationErrors {
s"but $prettyName is not an aggregate function")
}
+ def ignoreNullsWithUnsupportedFunctionError(prettyName: String): Throwable = {
+ new AnalysisException("IGNORE NULLS specified, " +
+ s"but $prettyName is not first, last or an offset window function")
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752374853
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38112/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549947505
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala
##########
@@ -212,6 +212,28 @@ class AnalysisErrorSuite extends AnalysisTest {
CatalystSqlParser.parsePlan("SELECT count(a) FILTER (WHERE rand(int(c)) > 1) FROM TaBlE2"),
"FILTER expression is non-deterministic, it cannot be used in aggregate functions" :: Nil)
+ errorTest(
+ "function don't support ignore nulls",
+ CatalystSqlParser.parsePlan("SELECT hex(a) IGNORE NULLS FROM TaBlE2"),
+ "IGNORE NULLS specified, but hex is not first, last or an offset window function" :: Nil)
Review comment:
we need to update tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752368958
**[Test build #133526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133526/testReport)** for PR 30943 at commit [`4cc5eb1`](https://github.com/apache/spark/commit/4cc5eb1c6c4ee31186dca059154621753b0ee62f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752393991
All newly added test cases in `sql/core/src/test/resources/sql-tests/inputs/window.sql` are completely consistent with the comparison results of Oracle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752361429
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133521/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752361408
**[Test build #133521 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133521/testReport)** for PR 30943 at commit [`4cc5eb1`](https://github.com/apache/spark/commit/4cc5eb1c6c4ee31186dca059154621753b0ee62f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751737160
**[Test build #133444 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133444/testReport)** for PR 30943 at commit [`564aae8`](https://github.com/apache/spark/commit/564aae8de635c8ccf207b43292d15a39a68a8485).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751657966
**[Test build #133444 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133444/testReport)** for PR 30943 at commit [`564aae8`](https://github.com/apache/spark/commit/564aae8de635c8ccf207b43292d15a39a68a8485).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752374703
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38112/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30943:
URL: https://github.com/apache/spark/pull/30943
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751621034
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38020/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549946660
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2123,19 +2123,58 @@ class Analyzer(override val catalogManager: CatalogManager)
if (isDistinct || filter.isDefined) {
throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
wf.prettyName)
+ } else if (ignoreNulls) {
+ wf match {
+ case nthValue: NthValue =>
+ nthValue.copy(ignoreNulls = ignoreNulls)
+ case _ =>
+ throw QueryCompilationErrors.ignoreNullsWithUnsupportedFunctionError(
+ wf.prettyName)
+ }
} else {
wf
}
+ case owf: FrameLessOffsetWindowFunction =>
+ if (isDistinct || filter.isDefined) {
+ throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
+ owf.prettyName)
+ } else if (ignoreNulls) {
+ owf match {
+ case lead: Lead =>
+ lead.copy(ignoreNulls = ignoreNulls)
+ case lag: Lag =>
+ lag.copy(ignoreNulls = ignoreNulls)
+ case _ =>
+ throw QueryCompilationErrors.ignoreNullsWithUnsupportedFunctionError(
+ owf.prettyName)
+ }
+ } else {
+ owf
+ }
// We get an aggregate function, we need to wrap it in an AggregateExpression.
case agg: AggregateFunction =>
if (filter.isDefined && !filter.get.deterministic) {
throw QueryCompilationErrors.nonDeterministicFilterInAggregateError
}
- AggregateExpression(agg, Complete, isDistinct, filter)
+ if (ignoreNulls) {
+ val aggFunc = agg match {
+ case first: First => first.copy(ignoreNulls = ignoreNulls)
+ case last: Last => last.copy(ignoreNulls = ignoreNulls)
+ case _ =>
+ throw QueryCompilationErrors.ignoreNullsWithUnsupportedFunctionError(
+ agg.prettyName)
+ }
+ AggregateExpression(aggFunc, Complete, isDistinct, filter)
+ } else {
+ AggregateExpression(agg, Complete, isDistinct, filter)
+ }
// This function is not an aggregate function, just return the resolved one.
case other if (isDistinct || filter.isDefined) =>
Review comment:
not related to this PR, but seems better to report error separately for DISTINCT and FILTER, like IGNORE NULLS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549937413
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2123,19 +2123,58 @@ class Analyzer(override val catalogManager: CatalogManager)
if (isDistinct || filter.isDefined) {
throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
wf.prettyName)
+ } else if (ignoreNulls) {
+ wf match {
+ case nthValue: NthValue =>
+ nthValue.copy(ignoreNulls = ignoreNulls)
+ case _ =>
+ throw QueryCompilationErrors.ignoreNullsWithUnsupportedFunctionError(
+ wf.prettyName)
+ }
} else {
wf
}
+ case owf: FrameLessOffsetWindowFunction =>
+ if (isDistinct || filter.isDefined) {
+ throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
+ owf.prettyName)
+ } else if (ignoreNulls) {
+ owf match {
+ case lead: Lead =>
+ lead.copy(ignoreNulls = ignoreNulls)
+ case lag: Lag =>
+ lag.copy(ignoreNulls = ignoreNulls)
Review comment:
we need to add a catch-all case at the end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751605734
**[Test build #133430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133430/testReport)** for PR 30943 at commit [`c82bb15`](https://github.com/apache/spark/commit/c82bb15172847ed39e3b9b59ae1642c32529e300).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752382292
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38116/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752368958
**[Test build #133526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133526/testReport)** for PR 30943 at commit [`4cc5eb1`](https://github.com/apache/spark/commit/4cc5eb1c6c4ee31186dca059154621753b0ee62f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751738289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133444/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752364661
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752353572
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133518/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752358779
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38107/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752351713
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38107/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752565698
**[Test build #133526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133526/testReport)** for PR 30943 at commit [`4cc5eb1`](https://github.com/apache/spark/commit/4cc5eb1c6c4ee31186dca059154621753b0ee62f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752374853
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38112/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752570398
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133526/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751657966
**[Test build #133444 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133444/testReport)** for PR 30943 at commit [`564aae8`](https://github.com/apache/spark/commit/564aae8de635c8ccf207b43292d15a39a68a8485).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751605734
**[Test build #133430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133430/testReport)** for PR 30943 at commit [`c82bb15`](https://github.com/apache/spark/commit/c82bb15172847ed39e3b9b59ae1642c32529e300).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752379362
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38114/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752341726
**[Test build #133518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133518/testReport)** for PR 30943 at commit [`9793eb1`](https://github.com/apache/spark/commit/9793eb1349141b32272b08891f4cbd845087851d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751656048
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133430/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752353482
**[Test build #133518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133518/testReport)** for PR 30943 at commit [`9793eb1`](https://github.com/apache/spark/commit/9793eb1349141b32272b08891f4cbd845087851d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752358779
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38107/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549938102
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -1697,8 +1697,13 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
expressions
}
val filter = Option(ctx.where).map(expression(_))
+ val ignoreNulls =
Review comment:
`val ignoreNulls = Option(ctx.nullsOption).map(_.getType == SqlBaseParser.IGNORE).getOrElse(false)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752302382
cc @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751621034
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38020/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752341726
**[Test build #133518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133518/testReport)** for PR 30943 at commit [`9793eb1`](https://github.com/apache/spark/commit/9793eb1349141b32272b08891f4cbd845087851d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549936864
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/QueryCompilationErrors.scala
##########
@@ -268,6 +268,11 @@ object QueryCompilationErrors {
s"but $prettyName is not an aggregate function")
}
+ def ignoreNullsWithUnsupportedFunctionError(prettyName: String): Throwable = {
+ new AnalysisException("IGNORE NULLS specified, " +
+ s"but $prettyName is not first, last or an offset window function")
Review comment:
we can simply say `Function $prettyName does not support IGNORE NULLS`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752353905
**[Test build #133521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133521/testReport)** for PR 30943 at commit [`4cc5eb1`](https://github.com/apache/spark/commit/4cc5eb1c6c4ee31186dca059154621753b0ee62f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751645526
**[Test build #133430 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133430/testReport)** for PR 30943 at commit [`c82bb15`](https://github.com/apache/spark/commit/c82bb15172847ed39e3b9b59ae1642c32529e300).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752365163
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38112/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752389677
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38114/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752381502
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38116/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549945063
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -1697,8 +1697,13 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
expressions
}
val filter = Option(ctx.where).map(expression(_))
+ val ignoreNulls =
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751668028
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38033/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752361429
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133521/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752570398
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133526/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751656048
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133430/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549944681
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2123,19 +2123,58 @@ class Analyzer(override val catalogManager: CatalogManager)
if (isDistinct || filter.isDefined) {
throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
wf.prettyName)
+ } else if (ignoreNulls) {
+ wf match {
+ case nthValue: NthValue =>
+ nthValue.copy(ignoreNulls = ignoreNulls)
+ case _ =>
+ throw QueryCompilationErrors.ignoreNullsWithUnsupportedFunctionError(
+ wf.prettyName)
+ }
} else {
wf
}
+ case owf: FrameLessOffsetWindowFunction =>
+ if (isDistinct || filter.isDefined) {
+ throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
+ owf.prettyName)
+ } else if (ignoreNulls) {
+ owf match {
+ case lead: Lead =>
+ lead.copy(ignoreNulls = ignoreNulls)
+ case lag: Lag =>
+ lag.copy(ignoreNulls = ignoreNulls)
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752353905
**[Test build #133521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133521/testReport)** for PR 30943 at commit [`4cc5eb1`](https://github.com/apache/spark/commit/4cc5eb1c6c4ee31186dca059154621753b0ee62f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-751675813
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38033/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752581945
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752358759
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38107/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30943:
URL: https://github.com/apache/spark/pull/30943#discussion_r549955100
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -2123,19 +2123,58 @@ class Analyzer(override val catalogManager: CatalogManager)
if (isDistinct || filter.isDefined) {
throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
wf.prettyName)
+ } else if (ignoreNulls) {
+ wf match {
+ case nthValue: NthValue =>
+ nthValue.copy(ignoreNulls = ignoreNulls)
+ case _ =>
+ throw QueryCompilationErrors.ignoreNullsWithUnsupportedFunctionError(
+ wf.prettyName)
+ }
} else {
wf
}
+ case owf: FrameLessOffsetWindowFunction =>
+ if (isDistinct || filter.isDefined) {
+ throw QueryCompilationErrors.distinctOrFilterOnlyWithAggregateFunctionError(
+ owf.prettyName)
+ } else if (ignoreNulls) {
+ owf match {
+ case lead: Lead =>
+ lead.copy(ignoreNulls = ignoreNulls)
+ case lag: Lag =>
+ lag.copy(ignoreNulls = ignoreNulls)
+ case _ =>
+ throw QueryCompilationErrors.ignoreNullsWithUnsupportedFunctionError(
+ owf.prettyName)
+ }
+ } else {
+ owf
+ }
// We get an aggregate function, we need to wrap it in an AggregateExpression.
case agg: AggregateFunction =>
if (filter.isDefined && !filter.get.deterministic) {
throw QueryCompilationErrors.nonDeterministicFilterInAggregateError
}
- AggregateExpression(agg, Complete, isDistinct, filter)
+ if (ignoreNulls) {
+ val aggFunc = agg match {
+ case first: First => first.copy(ignoreNulls = ignoreNulls)
+ case last: Last => last.copy(ignoreNulls = ignoreNulls)
+ case _ =>
+ throw QueryCompilationErrors.ignoreNullsWithUnsupportedFunctionError(
+ agg.prettyName)
+ }
+ AggregateExpression(aggFunc, Complete, isDistinct, filter)
+ } else {
+ AggregateExpression(agg, Complete, isDistinct, filter)
+ }
// This function is not an aggregate function, just return the resolved one.
case other if (isDistinct || filter.isDefined) =>
Review comment:
I got it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752380211
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38116/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30943: [SPARK-30789][SQL] Support (IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30943:
URL: https://github.com/apache/spark/pull/30943#issuecomment-752389666
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38114/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org