You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ruifeng Zheng (Jira)" <ji...@apache.org> on 2022/12/26 08:54:00 UTC
[jira] [Updated] (SPARK-41234) High-order function: array_insert
[ https://issues.apache.org/jira/browse/SPARK-41234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruifeng Zheng updated SPARK-41234:
----------------------------------
Description:
refer to https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/api/snowflake.snowpark.functions.array_insert.html
1, about the data type validation:
In Snowflake’s array_append, array_prepend and array_insert functions, the element data type does not need to match the data type of the existing elements in the array.
While in Spark, we want to leverage the same data type validation as array_remove.
2, about the NULL handling
Currently, SparkSQL, SnowSQL and PostgreSQL deal with NULL values in different ways.
Existing functions array_contains, array_position and array_remove in SparkSQL handle NULL in this way, if the input array or/and element is NULL, returns NULL. However, this behavior should be broken.
We should implement the NULL handling in array_append in this way:
2.1, if the array is NULL, returns NULL;
2.2 if the array is not NULL, the element is NULL, append the NULL value into the array
was:refer to https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/api/snowflake.snowpark.functions.array_insert.html
> High-order function: array_insert
> ---------------------------------
>
> Key: SPARK-41234
> URL: https://issues.apache.org/jira/browse/SPARK-41234
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark, SQL
> Affects Versions: 3.4.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> refer to https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/api/snowflake.snowpark.functions.array_insert.html
> 1, about the data type validation:
> In Snowflake’s array_append, array_prepend and array_insert functions, the element data type does not need to match the data type of the existing elements in the array.
> While in Spark, we want to leverage the same data type validation as array_remove.
> 2, about the NULL handling
> Currently, SparkSQL, SnowSQL and PostgreSQL deal with NULL values in different ways.
> Existing functions array_contains, array_position and array_remove in SparkSQL handle NULL in this way, if the input array or/and element is NULL, returns NULL. However, this behavior should be broken.
> We should implement the NULL handling in array_append in this way:
> 2.1, if the array is NULL, returns NULL;
> 2.2 if the array is not NULL, the element is NULL, append the NULL value into the array
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org