[jira] [Created] (SPARK-60048) Remember a whole-stage codegen split decision by a key that ignores literal values

["Max Gekk (Jira)" <ji...@apache.org>]

Max Gekk created SPARK-60048:
--------------------------------

             Summary: Remember a whole-stage codegen split decision by a key that ignores literal values
                 Key: SPARK-60048
                 URL: https://issues.apache.org/jira/browse/SPARK-60048
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Max Gekk


Whole-stage codegen can split the expressions of a stage that has a method past the JIT limit (SPARK-33301, https://github.com/apache/spark/pull/59225). It decides by generating the stage's code in one piece, compiling it as a trial, and generating it again with some expressions split. The decision is remembered by WholeStageCodegenExec.decisionKey, a digest of the generated code with the class loader, the compiler, the method limit and methodSplitThreshold, so that executing a query again skips the trial compiles.

A literal that changes on every run changes that digest on every run. ComputeCurrentTime turns current_timestamp() into a literal and Literal.doGenCode writes it into the source, so a stage holding one decides afresh each run and pays two code generations and two compiles where master pays one of each. Queries without such literals compile nothing new after the first execution.

Proposed: remember the decision by a key that ignores literal values, either the code digested with numeric literals masked, or the plan with literals normalized plus every conf that affects code generation.

Care is needed: a literal's value moves a method's bytecode size by a byte or two, which can matter for a method near the limit, and a key built from the plan has to carry all the codegen confs. Tests: two executions of a query with current_timestamp() decide once; a decision is not reused under a different conf that changes the code.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org