You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/31 09:33:12 UTC

[GitHub] [spark] beliefer opened a new pull request #35079: [SPARK-30789][SQL] Add document for syntax `(IGNORE | RESPECT) NULLS`

beliefer opened a new pull request #35079:
URL: https://github.com/apache/spark/pull/35079


   ### What changes were proposed in this pull request?
   https://github.com/apache/spark/pull/30943 supports syntax `(IGNORE | RESPECT) NULLS for LEAD/LAG/NTH_VALUE/FIRST_VALUE/LAST_VALUE`, but update document.
   
   This PR adds document for syntax `(IGNORE | RESPECT) NULLS`
   
   
   ### Why are the changes needed?
   Add document for syntax `(IGNORE | RESPECT) NULLS`
   
   
   ### Does this PR introduce _any_ user-facing change?
   'Yes'.
   
   
   ### How was this patch tested?
   Manual check.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on a change in pull request #35079: [SPARK-30789][SQL][DOCS][FOLLOWUP] Add document for syntax `(IGNORE | RESPECT) NULLS`

Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #35079:
URL: https://github.com/apache/spark/pull/35079#discussion_r778049940



##########
File path: docs/sql-ref-syntax-qry-select-window.md
##########
@@ -39,17 +39,21 @@ window_function OVER
     * Ranking Functions
 
       **Syntax:** `RANK | DENSE_RANK | PERCENT_RANK | NTILE | ROW_NUMBER`
+      
+      Note: Cannot append `[ { IGNORE | RESPECT } NULLS ]`
 
     * Analytic Functions
 
-      **Syntax:** `CUME_DIST | LAG | LEAD`
+      **Syntax:** `[ CUME_DIST | [ LAG | LEAD | NTH_VALUE | FIRST_VALUE | LAST_VALUE ] [ { IGNORE | RESPECT } NULLS ] ]`
 
     * Aggregate Functions
 
       **Syntax:** `MAX | MIN | COUNT | SUM | AVG | ...`
 
       Please refer to the [Built-in Aggregation Functions](sql-ref-functions-builtin.html#aggregate-functions) document for a complete list of Spark aggregate functions.
 
+      Note: Cannot append `[ { IGNORE | RESPECT } NULLS ]`
+
 * **window_frame**

Review comment:
       OK




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on pull request #35079: [SPARK-30789][SQL][DOCS][FOLLOWUP] Add document for syntax `(IGNORE | RESPECT) NULLS`

Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #35079:
URL: https://github.com/apache/spark/pull/35079#issuecomment-1005398372


   @cloud-fan Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #35079: [SPARK-30789][SQL][DOCS][FOLLOWUP] Add document for syntax `(IGNORE | RESPECT) NULLS`

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #35079:
URL: https://github.com/apache/spark/pull/35079


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #35079: [SPARK-30789][SQL][DOCS][FOLLOWUP] Add document for syntax `(IGNORE | RESPECT) NULLS`

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35079:
URL: https://github.com/apache/spark/pull/35079#discussion_r778070618



##########
File path: docs/sql-ref-syntax-qry-select-window.md
##########
@@ -42,14 +42,24 @@ window_function OVER
 
     * Analytic Functions
 
-      **Syntax:** `CUME_DIST | LAG | LEAD`
+      **Syntax:** `CUME_DIST | LAG | LEAD | NTH_VALUE | FIRST_VALUE | LAST_VALUE`
 
     * Aggregate Functions
 
       **Syntax:** `MAX | MIN | COUNT | SUM | AVG | ...`
 
       Please refer to the [Built-in Aggregation Functions](sql-ref-functions-builtin.html#aggregate-functions) document for a complete list of Spark aggregate functions.
 
+* **nulls_option**
+
+    Specified to indicate whether skip null values in the determination of which row to use. `RESPECT NULLS` is supported by default if you do not specify `IGNORE NULLS`.
+
+    **Syntax:**
+
+    `{ IGNORE | RESPECT } NULLS`
+
+    **Note:** Only `LAG | LEAD | NTH_VALUE | FIRST_VALUE | LAST_VALUE` could append `{ IGNORE | RESPECT } NULLS`.

Review comment:
       ```suggestion
       **Note:** Only `LAG | LEAD | NTH_VALUE | FIRST_VALUE | LAST_VALUE` can be used with `IGNORE NULLS`.
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #35079: [SPARK-30789][SQL][DOCS][FOLLOWUP] Add document for syntax `(IGNORE | RESPECT) NULLS`

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35079:
URL: https://github.com/apache/spark/pull/35079#discussion_r777963872



##########
File path: docs/sql-ref-syntax-qry-select-window.md
##########
@@ -39,17 +39,21 @@ window_function OVER
     * Ranking Functions
 
       **Syntax:** `RANK | DENSE_RANK | PERCENT_RANK | NTILE | ROW_NUMBER`
+      
+      Note: Cannot append `[ { IGNORE | RESPECT } NULLS ]`
 
     * Analytic Functions
 
-      **Syntax:** `CUME_DIST | LAG | LEAD`
+      **Syntax:** `[ CUME_DIST | [ LAG | LEAD | NTH_VALUE | FIRST_VALUE | LAST_VALUE ] [ { IGNORE | RESPECT } NULLS ] ]`
 
     * Aggregate Functions
 
       **Syntax:** `MAX | MIN | COUNT | SUM | AVG | ...`
 
       Please refer to the [Built-in Aggregation Functions](sql-ref-functions-builtin.html#aggregate-functions) document for a complete list of Spark aggregate functions.
 
+      Note: Cannot append `[ { IGNORE | RESPECT } NULLS ]`
+
 * **window_frame**

Review comment:
       Can we add a new section for `{ IGNORE | RESPECT } NULLS`? We need to explain its semantic, the default value, and can apply to which functions.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] beliefer commented on pull request #35079: [SPARK-30789][SQL][DOCS][FOLLOWUP] Add document for syntax `(IGNORE | RESPECT) NULLS`

Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #35079:
URL: https://github.com/apache/spark/pull/35079#issuecomment-1004675343


   ping @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #35079: [SPARK-30789][SQL][DOCS][FOLLOWUP] Add document for syntax `(IGNORE | RESPECT) NULLS`

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35079:
URL: https://github.com/apache/spark/pull/35079#discussion_r778069946



##########
File path: docs/sql-ref-syntax-qry-select-window.md
##########
@@ -42,14 +42,24 @@ window_function OVER
 
     * Analytic Functions
 
-      **Syntax:** `CUME_DIST | LAG | LEAD`
+      **Syntax:** `CUME_DIST | LAG | LEAD | NTH_VALUE | FIRST_VALUE | LAST_VALUE`
 
     * Aggregate Functions
 
       **Syntax:** `MAX | MIN | COUNT | SUM | AVG | ...`
 
       Please refer to the [Built-in Aggregation Functions](sql-ref-functions-builtin.html#aggregate-functions) document for a complete list of Spark aggregate functions.
 
+* **nulls_option**
+
+    Specified to indicate whether skip null values in the determination of which row to use. `RESPECT NULLS` is supported by default if you do not specify `IGNORE NULLS`.

Review comment:
       Specifies whether or not to skip null values when evaluating the window function. `RESECT NULLS` means not skipping null values, while `IGNORE NULLS` means skipping. If not specified, the default is `RESECT NULLS`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #35079: [SPARK-30789][SQL][DOCS][FOLLOWUP] Add document for syntax `(IGNORE | RESPECT) NULLS`

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35079:
URL: https://github.com/apache/spark/pull/35079#issuecomment-1005382695


   thanks, merging to master/3.2!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org