You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Aman Omer (Jira)" <ji...@apache.org> on 2019/10/29 08:14:00 UTC
[jira] [Comment Edited] (SPARK-29595) Insertion with named_struct
should match by name
[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961783#comment-16961783 ]
Aman Omer edited comment on SPARK-29595 at 10/29/19 8:13 AM:
-------------------------------------------------------------
{color:#172b4d}Parameters required for named_struct is Seq(name1, val1, name2, val2, ...). Validation step for named_struct only check for string at odd places. For example following query will add a row in _str_ table.{color}{color}
{code:java}
insert into str values named_struct( "ab", 1, "ba", 2);{code}
According to the discussion in [https://github.com/apache/spark/pull/26275] , which was tackling similar issue, changing fields of struct type according to names will introduce complexity.
So I think Spark should throw an exception when names does not match in named_struct.
cc [~srowen] [~maropu]
was (Author: aman_omer):
Parameters required for named_struct is _{color:#172b4d}Seq(name1, val1, name2, val2, ...){color}_{color:#808080}{color:#172b4d}. Validation step for named_struct only check for string at odd places. For example following query will add a row in _str_ table.{color} {color}
{code:java}
insert into str values named_struct( "ab", 1, "ba", 2);{code}
According to the discussion in [https://github.com/apache/spark/pull/26275] , which was tackling similar issue, changing fields of struct type according to names will introduce complexity.
So I think Spark should throw an exception when names does not match in named_struct.
cc [~srowen] [~maropu]
> Insertion with named_struct should match by name
> ------------------------------------------------
>
> Key: SPARK-29595
> URL: https://issues.apache.org/jira/browse/SPARK-29595
> Project: Spark
> Issue Type: Task
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Gengliang Wang
> Priority: Major
>
> {code:java}
> spark-sql> create table str using parquet as(select named_struct('a', 1, 'b', 2) as data);
> spark-sql> insert into str values named_struct("b", 3, "a", 1);
> spark-sql> select * from str;
> {"a":3,"b":1}
> {"a":1,"b":2}
> {code}
> The result should be
> {code:java}
> {"a":1,"b":3}
> {"a":1,"b":2}
> {code}
> Spark should match the field names of named_struct on insertion
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org