[jira] [Created] (SPARK-59774) Log at WARN, and name the remedy, when a generated method is too long to be JIT compiled

["Max Gekk (Jira)" <ji...@apache.org>]

Max Gekk created SPARK-59774:
--------------------------------

             Summary: Log at WARN, and name the remedy, when a generated method is too long to be JIT compiled
                 Key: SPARK-59774
                 URL: https://issues.apache.org/jira/browse/SPARK-59774
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Max Gekk


HotSpot does not JIT-compile a method whose bytecode is past 8000 bytes ({{-XX:+DontCompileHugeMethods}}, on by default), so a whole-stage codegen method past that size runs in the bytecode interpreter for the life of the executor. Spark detects this: since SPARK-25113 (2.4.0), {{CodeGenerator}} (now {{CodeCompiler.computeByteCodeStats}}) logs

{noformat}
INFO CodeGenerator: Generated method too long to be JIT compiled: <class>.<method> is N bytes
{noformat}

But it logs it at INFO. {{spark-shell}} and {{pyspark}} set the root level to WARN, so an interactive user never sees it, and in a batch job's log it is one line among thousands of INFO lines. Nothing acts on it by default either: {{spark.sql.codegen.hugeMethodLimit}}, which falls back from whole-stage codegen for such a stage, defaults to 65535, and a compiled method cannot exceed that, so at the default the fallback never fires.

The cost is large. On stock Spark 4.2.0, a projection whose consume method grows from 7945 to 8677 bytes steps from about 770 to about 4100 ns a row (JDK 17, 21 and 25 alike), and a filter of date ranges over a scan, the partition-key filter shape of TPC-DS {{modified-q3}}, steps about 135 times, because the filter is generated into {{processNext}} and so the whole scan loop runs interpreted. With {{spark.sql.codegen.hugeMethodLimit=8000}} the projection's step shrinks to about 1.3 times.

Proposal: log the line at WARN, and say what it means and what can be done: that the method runs interpreted, and that in whole-stage codegen {{spark.sql.codegen.hugeMethodLimit}} set to the JVM's limit runs such a stage without whole-stage codegen instead. The line is logged once per compiled class (compilation is cached), so the extra WARN output is bounded by the number of distinct generated classes past the limit, which is rare: in the TPC-DS and TPC-H plans only {{modified-q3}}'s stage crosses it.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org