You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/05/13 13:11:00 UTC
[jira] [Updated] (SPARK-52103) Fallback complex expression whole-stage codegen
[ https://issues.apache.org/jira/browse/SPARK-52103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-52103:
-----------------------------------
Labels: pull-request-available (was: )
> Fallback complex expression whole-stage codegen
> -----------------------------------------------
>
> Key: SPARK-52103
> URL: https://issues.apache.org/jira/browse/SPARK-52103
> Project: Spark
> Issue Type: Task
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Wan Kun
> Priority: Major
> Labels: pull-request-available
>
> The whole-stage code of a complex expression may too long to be JIT compiled.
> Add a config to determines whether fallback the complex expressions codegen.
> Test query
> {code:sql}
> SELECT vv
> FROM
> (
> SELECT vv, case vv
> when '1' then '1'
> when '2' then '2'
> when '3' then '3'
> when '4' then '4'
> when '5' then '5'
> when '6' then '6'
> when '7' then '7'
> when '8' then '8'
> when '9' then '9'
> when '10' then '10'
> when '11' then '11'
> when '12' then '12'
> when '13' then '13'
> when '14' then '14'
> when '15' then '15'
> when '16' then '16'
> when '17' then '17'
> when '18' then '18'
> when '19' then '19'
> when '20' then '20'
> when '21' then '21'
> when '22' then '22'
> when '23' then '23'
> when '24' then '24'
> when '25' then '25'
> when '26' then '26'
> when '27' then '27'
> when '28' then '28'
> when '29' then '29'
> when '30' then '30'
> when '31' then '31'
> when '32' then '32'
> else ''
> end as cv
> FROM (
> SELECT regexp_replace(trim(lower(
> get_json_object(concat(v,'}'),'$$.s'))),'\\n','') AS vv
> FROM values('a') as t(v)
> ) tmp
> ) t2
> WHERE length(cv) > 0
> AND cv not LIKE '%xxx%'
> {code}
> JIT error message:
> 25/05/13 13:19:35 INFO CodeGenerator: Generated method too long to be JIT compiled: MessageWithContext(org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.generate_doConsume_0$ ,{})MessageWithContext(is 21798 bytes,{})
> 25/05/13 13:19:35 INFO CodeGenerator: Code generated in 1658.222814 ms
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org