You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/14 06:53:53 UTC

[GitHub] [spark] beliefer opened a new pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

beliefer opened a new pull request #31178:
URL: https://github.com/apache/spark/pull/31178


   ### What changes were proposed in this pull request?
   The current implement of `UnboundedOffsetWindowFunctionFrame` and `UnboundedPrecedingOffsetWindowFunctionFrame` only support `nth_value` that respect nulls. So nth_value will execute with `UnboundedWindowFunctionFrame` and `UnboundedPrecedingWindowFunctionFrame`.
   `UnboundedWindowFunctionFrame` and `UnboundedPrecedingWindowFunctionFrame` will call `updateExpressions` of `nth_value` multiple times.
   
   
   ### Why are the changes needed?
   Improve performance for nth_value ignore nulls over offset window
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   Jenkins test
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763944989


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134285/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31178:
URL: https://github.com/apache/spark/pull/31178#discussion_r557843329



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExecBase.scala
##########
@@ -149,8 +150,8 @@ trait WindowExecBase extends UnaryExecNode {
             case AggregateExpression(f, _, _, _, _) => collect("AGGREGATE", frame, e, f)
             case f: FrameLessOffsetWindowFunction =>
               collect("FRAME_LESS_OFFSET", f.fakeFrame, e, f)
-            case f: OffsetWindowFunction if !f.ignoreNulls &&
-              frame.frameType == RowFrame && frame.lower == UnboundedPreceding =>
+            case f: OffsetWindowFunction if frame.frameType == RowFrame &&
+              frame.lower == UnboundedPreceding =>
               frame.upper match {

Review comment:
       if we go with the `case _ => collect("AGGREGATE", frame, e, f)` branch, do we still support ignore nulls?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31178:
URL: https://github.com/apache/spark/pull/31178


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31178:
URL: https://github.com/apache/spark/pull/31178#discussion_r561051697



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -305,12 +308,21 @@ class UnboundedOffsetWindowFunctionFrame(
       inputIterator = input.generateIterator()
       // drain the first few rows if offset is larger than one
       inputIndex = 0
-      while (inputIndex < offset - 1) {
-        if (inputIterator.hasNext) inputIterator.next()
-        inputIndex += 1
+      if (ignoreNulls) {
+        findNextRowWithNonNullInput()
+        if (nextSelectedRow == EmptyRow) {
+          // Use default values since the offset row whose input value is not null does not exist.
+          fillDefaultValue(EmptyRow)
+        } else {
+          projection(nextSelectedRow)
+        }
+      } else {
+        while (inputIndex < offset) {
+          selectedRow = WindowFunctionFrame.getNextOrNull(inputIterator)

Review comment:
       In this class, `write` is empty, so `selectedRow` can be a local variable
   ```
   var selectedRow: UnsafeRow = null
   while (inputIndex < offset) {
     selectedRow = WindowFunctionFrame.getNextOrNull(inputIterator)
     inputIndex += 1
   }
   projection(selectedRow)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763653986


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38860/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760641016


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38659/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763789727


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38864/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763733954


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38864/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on a change in pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31178:
URL: https://github.com/apache/spark/pull/31178#discussion_r557843994



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExecBase.scala
##########
@@ -149,8 +150,8 @@ trait WindowExecBase extends UnaryExecNode {
             case AggregateExpression(f, _, _, _, _) => collect("AGGREGATE", frame, e, f)
             case f: FrameLessOffsetWindowFunction =>
               collect("FRAME_LESS_OFFSET", f.fakeFrame, e, f)
-            case f: OffsetWindowFunction if !f.ignoreNulls &&
-              frame.frameType == RowFrame && frame.lower == UnboundedPreceding =>
+            case f: OffsetWindowFunction if frame.frameType == RowFrame &&
+              frame.lower == UnboundedPreceding =>
               frame.upper match {

Review comment:
       Nth_value support ignore nulls. lead and lag not.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-764442309


   thanks, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760627504


   **[Test build #134073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134073/testReport)** for PR 31178 at commit [`c5259be`](https://github.com/apache/spark/commit/c5259beea04ee20620b622f67fdb83e5bc30cd79).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760626263


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38659/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760758025


   **[Test build #134073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134073/testReport)** for PR 31178 at commit [`c5259be`](https://github.com/apache/spark/commit/c5259beea04ee20620b622f67fdb83e5bc30cd79).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-764447673


   @cloud-fan Thanks for your work!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763898643


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134280/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on a change in pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31178:
URL: https://github.com/apache/spark/pull/31178#discussion_r560988916



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -305,12 +306,22 @@ class UnboundedOffsetWindowFunctionFrame(
       inputIterator = input.generateIterator()
       // drain the first few rows if offset is larger than one
       inputIndex = 0
-      while (inputIndex < offset - 1) {
-        if (inputIterator.hasNext) inputIterator.next()
-        inputIndex += 1
+      if (ignoreNulls) {
+        findNextRowWithNonNullInput()
+        if (nextSelectedRow == EmptyRow) {
+          // Use default values since the offset row whose input value is not null does not exist.
+          fillDefaultValue(EmptyRow)
+        } else {
+          projection(nextSelectedRow)
+        }
+      } else {
+        while (inputIndex < offset - 1) {

Review comment:
       OK




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763627160


   **[Test build #134274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134274/testReport)** for PR 31178 at commit [`4ba5a6b`](https://github.com/apache/spark/commit/4ba5a6b3745f6eca0aa671ae8a9c5616520533fb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763802029


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38871/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763898643






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760758882


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134073/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763700333


   **[Test build #134280 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134280/testReport)** for PR 31178 at commit [`64642a2`](https://github.com/apache/spark/commit/64642a22aeac80087059ef5384382ab09fda26ea).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763898643






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763904422


   **[Test build #134285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134285/testReport)** for PR 31178 at commit [`bf17bae`](https://github.com/apache/spark/commit/bf17bae5e8d3f7c0eb1c17d90b75618d60553d4d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31178:
URL: https://github.com/apache/spark/pull/31178


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763742671


   **[Test build #134285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134285/testReport)** for PR 31178 at commit [`bf17bae`](https://github.com/apache/spark/commit/bf17bae5e8d3f7c0eb1c17d90b75618d60553d4d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763878043


   **[Test build #134280 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134280/testReport)** for PR 31178 at commit [`64642a2`](https://github.com/apache/spark/commit/64642a22aeac80087059ef5384382ab09fda26ea).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763742671


   **[Test build #134285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134285/testReport)** for PR 31178 at commit [`bf17bae`](https://github.com/apache/spark/commit/bf17bae5e8d3f7c0eb1c17d90b75618d60553d4d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on a change in pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31178:
URL: https://github.com/apache/spark/pull/31178#discussion_r557843994



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowExecBase.scala
##########
@@ -149,8 +150,8 @@ trait WindowExecBase extends UnaryExecNode {
             case AggregateExpression(f, _, _, _, _) => collect("AGGREGATE", frame, e, f)
             case f: FrameLessOffsetWindowFunction =>
               collect("FRAME_LESS_OFFSET", f.fakeFrame, e, f)
-            case f: OffsetWindowFunction if !f.ignoreNulls &&
-              frame.frameType == RowFrame && frame.lower == UnboundedPreceding =>
+            case f: OffsetWindowFunction if frame.frameType == RowFrame &&
+              frame.lower == UnboundedPreceding =>
               frame.upper match {

Review comment:
       Nth_value also supports ignore nulls




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763828225


   **[Test build #134274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134274/testReport)** for PR 31178 at commit [`4ba5a6b`](https://github.com/apache/spark/commit/4ba5a6b3745f6eca0aa671ae8a9c5616520533fb).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760078487






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763789727


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38864/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760078487


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38630/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760170401


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134044/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763627160


   **[Test build #134274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134274/testReport)** for PR 31178 at commit [`4ba5a6b`](https://github.com/apache/spark/commit/4ba5a6b3745f6eca0aa671ae8a9c5616520533fb).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on a change in pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31178:
URL: https://github.com/apache/spark/pull/31178#discussion_r561078618



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -305,12 +308,21 @@ class UnboundedOffsetWindowFunctionFrame(
       inputIterator = input.generateIterator()
       // drain the first few rows if offset is larger than one
       inputIndex = 0
-      while (inputIndex < offset - 1) {
-        if (inputIterator.hasNext) inputIterator.next()
-        inputIndex += 1
+      if (ignoreNulls) {
+        findNextRowWithNonNullInput()
+        if (nextSelectedRow == EmptyRow) {
+          // Use default values since the offset row whose input value is not null does not exist.
+          fillDefaultValue(EmptyRow)
+        } else {
+          projection(nextSelectedRow)
+        }
+      } else {
+        while (inputIndex < offset) {
+          selectedRow = WindowFunctionFrame.getNextOrNull(inputIterator)

Review comment:
       OK




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-764442309


   thanks, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760627504


   **[Test build #134073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134073/testReport)** for PR 31178 at commit [`c5259be`](https://github.com/apache/spark/commit/c5259beea04ee20620b622f67fdb83e5bc30cd79).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760758882


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134073/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31178:
URL: https://github.com/apache/spark/pull/31178#discussion_r560962193



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowFunctionFrame.scala
##########
@@ -305,12 +306,22 @@ class UnboundedOffsetWindowFunctionFrame(
       inputIterator = input.generateIterator()
       // drain the first few rows if offset is larger than one
       inputIndex = 0
-      while (inputIndex < offset - 1) {
-        if (inputIterator.hasNext) inputIterator.next()
-        inputIndex += 1
+      if (ignoreNulls) {
+        findNextRowWithNonNullInput()
+        if (nextSelectedRow == EmptyRow) {
+          // Use default values since the offset row whose input value is not null does not exist.
+          fillDefaultValue(EmptyRow)
+        } else {
+          projection(nextSelectedRow)
+        }
+      } else {
+        while (inputIndex < offset - 1) {

Review comment:
       Can we follow https://github.com/apache/spark/pull/31178/files#diff-09c876759e9b45e198238f7244eba48900058896287bf371acb27de9e25032b8R367 to write the loop?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763829858






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760641016


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38659/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763778989


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38871/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-764447673


   @cloud-fan Thanks for your work!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760633826


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38659/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763699786


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38860/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763699746


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38860/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-759989714


   **[Test build #134044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134044/testReport)** for PR 31178 at commit [`cdab348`](https://github.com/apache/spark/commit/cdab34831e63f8aa8c8d5d819169d0dfc1c5344c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763779295


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38864/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763904422


   **[Test build #134285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134285/testReport)** for PR 31178 at commit [`bf17bae`](https://github.com/apache/spark/commit/bf17bae5e8d3f7c0eb1c17d90b75618d60553d4d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760602651


   cc @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763898643


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134280/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763742671


   **[Test build #134285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134285/testReport)** for PR 31178 at commit [`bf17bae`](https://github.com/apache/spark/commit/bf17bae5e8d3f7c0eb1c17d90b75618d60553d4d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-759989714


   **[Test build #134044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134044/testReport)** for PR 31178 at commit [`cdab348`](https://github.com/apache/spark/commit/cdab34831e63f8aa8c8d5d819169d0dfc1c5344c).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763700333


   **[Test build #134280 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134280/testReport)** for PR 31178 at commit [`64642a2`](https://github.com/apache/spark/commit/64642a22aeac80087059ef5384382ab09fda26ea).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-760160047


   **[Test build #134044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134044/testReport)** for PR 31178 at commit [`cdab348`](https://github.com/apache/spark/commit/cdab34831e63f8aa8c8d5d819169d0dfc1c5344c).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763829858






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763944989


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134285/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #31178: [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31178:
URL: https://github.com/apache/spark/pull/31178#issuecomment-763699786


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38860/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org