You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2022/05/19 14:53:00 UTC
[jira] [Resolved] (SPARK-38529) Prevent GeneratorNestedColumnAliasing to be applied to non-Explode generators
[ https://issues.apache.org/jira/browse/SPARK-38529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-38529.
---------------------------------
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 35850
[https://github.com/apache/spark/pull/35850]
> Prevent GeneratorNestedColumnAliasing to be applied to non-Explode generators
> -----------------------------------------------------------------------------
>
> Key: SPARK-38529
> URL: https://issues.apache.org/jira/browse/SPARK-38529
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer
> Affects Versions: 3.2.1
> Reporter: Min Yang
> Priority: Minor
> Fix For: 3.3.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The Project(_, g: Generate) branch in GeneratorNestedColumnAliasing is only supposed to work for ExplodeBase generators but we do not explicitly return for other types like Inline. Currently the bug is not trigger because there is another bug in the "prune unrequired child" branch in the ColumnPruning which makes other generators like Inline always go to that branch even if it is not applicable.
>
> An easy example to show the bug:
> Input: <col1: int, col2: array<struct<field1 struct<field1: int, field2: int>, field2 int>>>
> Project(field1.field1 as ...)
> - Generate(Inline(col2), ..., field1, field2)
>
> We will try to incorrectly push the .field1 on field1 into the input of the Inline (col2).
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org