You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/26 09:02:01 UTC
[GitHub] [spark] zhengruifeng commented on pull request #38867: [SPARK-41234][SQL][PYTHON] Add `array_insert` function
zhengruifeng commented on PR #38867:
URL: https://github.com/apache/spark/pull/38867#issuecomment-1365013704
@Daniel-Davies Sorry for the late reply.
On `Input item parameter is null:`
The issue of NULL handling was controversial, and we spend some time to discuss with some SQL experts and decided to adopt snowflake's method of NULL handling which is more reasonable compared with existing SparkSQL's method, that is:
1, if the array is NULL, returns NULL;
2, if the array is NOT NULL and the element is NULL, insert this NULL into the array;
```
+--------------------------------------------+
| ARRAY_INSERT(ARRAY_CONSTRUCT(1,2,3), 0, 4) |
|--------------------------------------------|
| [ |
| 4, |
| 1, |
| 2, |
| 3 |
| ] |
+--------------------------------------------+
1 Row(s) produced. Time Elapsed: 0.139s
+--------------------------------------------+
| ARRAY_INSERT(ARRAY_CONSTRUCT(1,2,3), 2, 4) |
|--------------------------------------------|
| [ |
| 1, |
| 2, |
| 4, |
| 3 |
| ] |
+--------------------------------------------+
1 Row(s) produced. Time Elapsed: 0.116s
+-----------------------------------------------+
| ARRAY_INSERT(ARRAY_CONSTRUCT(1,2,3), 2, NULL) |
|-----------------------------------------------|
| [ |
| 1, |
| 2, |
| undefined, |
| 3 |
| ] |
+-----------------------------------------------+
1 Row(s) produced. Time Elapsed: 0.130s
+--------------------------+
| ARRAY_INSERT(NULL, 2, 1) |
|--------------------------|
| NULL |
+--------------------------+
1 Row(s) produced. Time Elapsed: 0.106s
+-----------------------------+
| ARRAY_INSERT(NULL, 2, NULL) |
|-----------------------------|
| NULL |
+-----------------------------+
1 Row(s) produced. Time Elapsed: 0.113s
```
On `Index out of bounds`: I think we should choose option b, filling with nulls
```
+---------------------------------------------+
| ARRAY_INSERT(ARRAY_CONSTRUCT(1,2,3), 10, 1) |
|---------------------------------------------|
| [ |
| 1, |
| 2, |
| 3, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| 1 |
| ] |
+---------------------------------------------+
1 Row(s) produced. Time Elapsed: 0.116s
+----------------------------------------------+
| ARRAY_INSERT(ARRAY_CONSTRUCT(1,2,3), -10, 1) |
|----------------------------------------------|
| [ |
| 1, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| 1, |
| 2, |
| 3 |
| ] |
+----------------------------------------------+
1 Row(s) produced. Time Elapsed: 0.111s
+------------------------------------------------+
| ARRAY_INSERT(ARRAY_CONSTRUCT(1,2,3), 10, NULL) |
|------------------------------------------------|
| [ |
| 1, |
| 2, |
| 3, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined |
| ] |
+------------------------------------------------+
1 Row(s) produced. Time Elapsed: 0.109s
+-------------------------------------------------+
| ARRAY_INSERT(ARRAY_CONSTRUCT(1,2,3), -10, NULL) |
|-------------------------------------------------|
| [ |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| undefined, |
| 1, |
| 2, |
| 3 |
| ] |
+-------------------------------------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org