You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Manu Zhang (Jira)" <ji...@apache.org> on 2021/11/29 08:00:00 UTC
[jira] [Commented] (SPARK-34714) collect_list(struct()) fails when used with GROUP BY
[ https://issues.apache.org/jira/browse/SPARK-34714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450236#comment-17450236 ]
Manu Zhang commented on SPARK-34714:
------------------------------------
FYI, this was resolved by https://issues.apache.org/jira/browse/SPARK-34713 and https://issues.apache.org/jira/browse/SPARK-34749
> collect_list(struct()) fails when used with GROUP BY
> ----------------------------------------------------
>
> Key: SPARK-34714
> URL: https://issues.apache.org/jira/browse/SPARK-34714
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1
> Environment: Databricks Runtime 8.0
> Reporter: Lauri Koobas
> Priority: Major
> Fix For: 3.1.2
>
>
> The following is failing in DBR8.0 / Spark 3.1.1, but works in earlier DBR and Spark versions:
> {quote}with step_1 as (
> select 'E' as name, named_struct('subfield', 1) as field_1
> )
> select name, collect_list(struct(field_1.subfield))
> from step_1
> group by 1
> {quote}
> Fails with the following error message:
> {quote}AnalysisException: cannot resolve 'struct(step_1.`field_1`.`subfield`)' due to data type mismatch: Only foldable string expressions are allowed to appear at odd position, got: NamePlaceholder
> {quote}
> If you modify the query in any of the following ways then it still works::
> * if you remove the field "name" and the "group by 1" part of the query
> * if you remove the "struct()" from within the collect_list()
> * if you use "named_struct()" instead of "struct()" within the collect_list()
> Similarly collect_set() is broken and possibly more related functions, but I haven't done thorough testing.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org