You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Zheng Shao (Jira)" <ji...@apache.org> on 2024/03/24 21:07:00 UTC
[jira] [Updated] (SPARK-47532) LATERAL VIEW EXPLODE(exp) unnecessarily calculates the exp twice due to InferFiltersFromGenerate
[ https://issues.apache.org/jira/browse/SPARK-47532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated SPARK-47532:
-------------------------------
Summary: LATERAL VIEW EXPLODE(exp) unnecessarily calculates the exp twice due to InferFiltersFromGenerate (was: LATERAL VIEW EXPLODE(exp) unnecessarily calculates the exp twice)
> LATERAL VIEW EXPLODE(exp) unnecessarily calculates the exp twice due to InferFiltersFromGenerate
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-47532
> URL: https://issues.apache.org/jira/browse/SPARK-47532
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.5.0
> Reporter: Zheng Shao
> Priority: Major
>
> We have a query like this:
> {{WITH staging AS (}}
> {{ SELECT}}
> {{ col1,}}
> {{ FROM_JSON(MY_PYTHON_UDF(col2), 'map<string, string>') AS mapping}}
> {{ FROM my_table}}
> {{)}}
> {{SELECT col1, key1, value1}}
> {{FROM staging}}
> {{{}LATERAL VIEW EXPLODE({}}}{{{}mapping{}}}{{{}) AS key1, value1{}}}
>
> *Current behavior:*
> The generated plan uses two invocations of MY_PYTHON_UDF:
> {{+- EvalExternalUDF}}
> {{ +- Project (get rid of mapping)}}
> {{ +- Filter (size(mapping) > 0 && isnotnull(mapping))}}
> {{ +- EvalExternalUDF }}
> However, if we use {{{}LATERAL VIEW OUTER EXPLODE{}}}, then there is only one {{{}EvalExternalUDF{}}}, since there is no need to do the filter in between.
>
> *Expected behavior:* Even with {{LATERAL VIEW EXPLODE, there is no need to do project and then EvalExternalUDF again. There should be just one invocation of EvalExternalUDF.}}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org