[jira] [Created] (SPARK-59782) Make spark.sql.codegen.hugeMethodLimit public or document it as a tuning knob

["Max Gekk (Jira)" <ji...@apache.org>]

Max Gekk created SPARK-59782:
--------------------------------

             Summary: Make spark.sql.codegen.hugeMethodLimit public or document it as a tuning knob
                 Key: SPARK-59782
                 URL: https://issues.apache.org/jira/browse/SPARK-59782
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Max Gekk


spark.sql.codegen.hugeMethodLimit is the only setting that acts on the JIT cliff: HotSpot does not compile a method past 8000 bytes (-XX:+DontCompileHugeMethods, on by default), so a whole-stage codegen stage whose per-row method crosses it runs interpreted, typically several times slower. Setting the limit to 8000 makes such a stage fall back to the non-whole-stage path instead, and the config's own doc already suggests that value on HotSpot.

The config is .internal(), yet SPARK-59774 names it in a user-facing WARN as the remedy. Internal configs already appear as tuning knobs in docs/sql-performance-tuning.md (spark.sql.files.openCostInBytes, spark.sql.sources.parallelPartitionDiscovery.parallelism, spark.sql.requireAllClusterKeysForCoPartition), so there is precedent either way.

Proposal: either make spark.sql.codegen.hugeMethodLimit public, or keep it internal and list it in the configuration tables of sql-performance-tuning.md with a short explanation of the cliff and when to set it to 8000.

Context: https://github.com/apache/spark/pull/59020




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org