You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/26 03:01:14 UTC
[GitHub] [spark] beliefer opened a new pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
beliefer opened a new pull request #30512:
URL: https://github.com/apache/spark/pull/30512
### What changes were proposed in this pull request?
Currently in Spark one could redefine a window. For instance:
`select count(*) OVER w FROM tenk1 WINDOW w AS (ORDER BY unique1), w AS (ORDER BY unique1);`
The window `w` is defined two times. In PgSQL, on the other hand, a thrown will happen:
`ERROR: window "w" is already defined`
### Why are the changes needed?
The current implement gives the following window definitions a higher priority. But it wasn't Spark's intention and users can't know from any document of Spark.
This PR fixes the bug.
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
Jenkins test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30512: [SPARK-28645][SQL] ParseException is thrown when the window is redefined
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30512:
URL: https://github.com/apache/spark/pull/30512#discussion_r531439870
##########
File path: sql/core/src/test/resources/sql-tests/inputs/window.sql
##########
@@ -250,4 +250,16 @@ WINDOW w AS (
ORDER BY salary DESC
RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
)
-ORDER BY department;
\ No newline at end of file
+ORDER BY department;
+
+SELECT
+ employee_name,
+ salary,
+ first_value(employee_name) OVER w highest_salary,
+ nth_value(employee_name, 2) OVER w second_highest_salary
+FROM
+ basic_pays
+WINDOW
+ w AS (ORDER BY salary DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING),
Review comment:
is the window spec name case insensitive?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #30512: [SPARK-28645][SQL] ParseException is thrown when the window is redefined
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734633191
The fix looks fine. In the PR description, could you add an example query output with/without this fix?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #30512: [SPARK-28645][SQL] ParseException is thrown when the window is redefined
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30512:
URL: https://github.com/apache/spark/pull/30512
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734261557
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734076829
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30512: [SPARK-28645][SQL] ParseException is thrown when the window is redefined
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #30512:
URL: https://github.com/apache/spark/pull/30512#discussion_r531462006
##########
File path: sql/core/src/test/resources/sql-tests/inputs/window.sql
##########
@@ -250,4 +250,16 @@ WINDOW w AS (
ORDER BY salary DESC
RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
)
-ORDER BY department;
\ No newline at end of file
+ORDER BY department;
+
+SELECT
+ employee_name,
+ salary,
+ first_value(employee_name) OVER w highest_salary,
+ nth_value(employee_name, 2) OVER w second_highest_salary
+FROM
+ basic_pays
+WINDOW
+ w AS (ORDER BY salary DESC ROWS BETWEEN UNBOUNDED PRECEDING AND 1 FOLLOWING),
Review comment:
I checked the PgSQL and find the alias of window frame is case insensitive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734046549
**[Test build #131824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131824/testReport)** for PR 30512 at commit [`43e8542`](https://github.com/apache/spark/commit/43e85422e28ec23ed057d41755d265a65a532dae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734261557
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734150420
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734150420
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734046549
**[Test build #131824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131824/testReport)** for PR 30512 at commit [`43e8542`](https://github.com/apache/spark/commit/43e85422e28ec23ed057d41755d265a65a532dae).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #30512: [SPARK-28645][SQL] ParseException is thrown when the window is redefined
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734666009
> The fix looks fine. In the PR description, could you add an example query output with/without this fix?
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #30512: [SPARK-28645][SQL] ParseException is thrown when the window is redefined
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734631160
cc @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734076829
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734245900
**[Test build #131832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131832/testReport)** for PR 30512 at commit [`9f29766`](https://github.com/apache/spark/commit/9f29766511aed5728707c5cfc3d46176826d4148).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734118818
**[Test build #131832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131832/testReport)** for PR 30512 at commit [`9f29766`](https://github.com/apache/spark/commit/9f29766511aed5728707c5cfc3d46176826d4148).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734118818
**[Test build #131832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131832/testReport)** for PR 30512 at commit [`9f29766`](https://github.com/apache/spark/commit/9f29766511aed5728707c5cfc3d46176826d4148).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #30512: [SPARK-28645][SQL] ParseException is thrown when the window is redefined
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734770699
@cloud-fan @maropu Thanks for your work!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30512: [SPARK-28645][SQL] Throw ParseException on window redefinition
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734070893
**[Test build #131824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131824/testReport)** for PR 30512 at commit [`43e8542`](https://github.com/apache/spark/commit/43e85422e28ec23ed057d41755d265a65a532dae).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30512: [SPARK-28645][SQL] ParseException is thrown when the window is redefined
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30512:
URL: https://github.com/apache/spark/pull/30512#issuecomment-734763582
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org