You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/12/01 00:22:00 UTC
[jira] [Updated] (SPARK-44773) Code-gen CodegenFallback expression in WholeStageCodegen if possible
[ https://issues.apache.org/jira/browse/SPARK-44773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-44773:
-----------------------------------
Labels: pull-request-available (was: )
> Code-gen CodegenFallback expression in WholeStageCodegen if possible
> --------------------------------------------------------------------
>
> Key: SPARK-44773
> URL: https://issues.apache.org/jira/browse/SPARK-44773
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.0
> Reporter: Wan Kun
> Priority: Major
> Labels: pull-request-available
>
> Now both WholeStageCodegen framework and SubExpressionElimination framework does not support CodegenFallback expression, but the CodegenFallback expression which contains nullSafeEval method could gen-code just like common expressions, now they are always be executed in a new SpecificUnsafeProjection class, and we can not eliminate the sub expressions.
> For example:
> SQL:
> {code:sql}
> SELECT from_json(regexp_replace(s, 'a', 'x'), 'x INT, b DOUBLE').x,
> from_json(regexp_replace(s, 'a', 'x'), 'x INT, b DOUBLE').b
> FROM values('{"a":1, "b":0.8}') t(s)
> {code}
> plan:
> {code:java}
> *(1) Project [from_json(StructField(x,IntegerType,true), regexp_replace(s#218, a, x, 1), Some(America/Los_Angeles)).x AS from_json(regexp_replace(s, a, x, 1)).x#219, from_json(StructField(b,DoubleType,true), regexp_replace(s#218, a, x, 1), Some(America/Los_Angeles)).b AS from_json(regexp_replace(s, a, x, 1)).b#220]
> +- *(1) LocalTableScan [s#218]
> {code}
> Due to expression org.apache.spark.sql.catalyst.expressions.JsonToStructs is CodegenFallback expression, so we can not reuse the result of {*}regexp_replace(s, 'a', 'x'){*}.
> We can support expression org.apache.spark.sql.catalyst.expressions.JsonToStructs code-gen in WholeStageCodegen framework, and then reuse the result of {*}regexp_replace(s, 'a', 'x'){*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org