You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/05/13 13:11:00 UTC

[jira] [Updated] (SPARK-52103) Fallback complex expression whole-stage codegen

     [ https://issues.apache.org/jira/browse/SPARK-52103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-52103:
-----------------------------------
    Labels: pull-request-available  (was: )

> Fallback complex expression whole-stage codegen
> -----------------------------------------------
>
>                 Key: SPARK-52103
>                 URL: https://issues.apache.org/jira/browse/SPARK-52103
>             Project: Spark
>          Issue Type: Task
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: Wan Kun
>            Priority: Major
>              Labels: pull-request-available
>
> The whole-stage code of a complex expression may too long to be JIT compiled. 
> Add a config to determines whether fallback the complex expressions codegen.
> Test query 
> {code:sql}
> SELECT  vv
> FROM
> (
>     SELECT  vv, case vv
>                 when '1' then '1'
>                 when '2' then '2'
>                 when '3' then '3'
>                 when '4' then '4'
>                 when '5' then '5'
>                 when '6' then '6'
>                 when '7' then '7'
>                 when '8' then '8'
>                 when '9' then '9'
>                 when '10' then '10'
>                 when '11' then '11'
>                 when '12' then '12'
>                 when '13' then '13'
>                 when '14' then '14'
>                 when '15' then '15'
>                 when '16' then '16'
>                 when '17' then '17'
>                 when '18' then '18'
>                 when '19' then '19'
>                 when '20' then '20'
>                 when '21' then '21'
>                 when '22' then '22'
>                 when '23' then '23'
>                 when '24' then '24'
>                 when '25' then '25'
>                 when '26' then '26'
>                 when '27' then '27'
>                 when '28' then '28'
>                 when '29' then '29'
>                 when '30' then '30'
>                 when '31' then '31'
>                 when '32' then '32'
>                 else ''
>                 end as cv
>     FROM (
>         SELECT  regexp_replace(trim(lower(
>                    get_json_object(concat(v,'}'),'$$.s'))),'\\n','') AS vv
>         FROM values('a') as t(v)
>     ) tmp
> ) t2
> WHERE length(cv) > 0
> AND cv not LIKE '%xxx%'
> {code}
> JIT error message: 
> 25/05/13 13:19:35 INFO CodeGenerator: Generated method too long to be JIT compiled: MessageWithContext(org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.generate_doConsume_0$ ,{})MessageWithContext(is 21798 bytes,{})
> 25/05/13 13:19:35 INFO CodeGenerator: Code generated in 1658.222814 ms



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org