You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Max Gekk (Jira)" <ji...@apache.org> on 2026/09/25 04:50:00 UTC
[jira] [Resolved] (SPARK-59764) BenchmarkQueryTest.checkGeneratedCode checks nothing under adaptive query execution
[ https://issues.apache.org/jira/browse/SPARK-59764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Max Gekk resolved SPARK-59764.
------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59017
[https://github.com/apache/spark/pull/59017]
> BenchmarkQueryTest.checkGeneratedCode checks nothing under adaptive query execution
> -----------------------------------------------------------------------------------
>
> Key: SPARK-59764
> URL: https://issues.apache.org/jira/browse/SPARK-59764
> Project: Spark
> Issue Type: Test
> Components: SQL
> Affects Versions: 3.2.0, 4.1.0, 4.0.0, 4.2.0, 4.3.0, 5.0.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> {{BenchmarkQueryTest.checkGeneratedCode}} is meant to check that every whole-stage codegen subtree of a benchmark query compiles and that its largest generated method is within {{CodeGenerator.DEFAULT_JVM_HUGE_METHOD_LIMIT}} (8000 bytes, HotSpot's {{HugeMethodLimit}}). {{TPCDSQuerySuite}} (and {{TPCDSQueryWithStatsSuite}}, {{TPCDSQueryANSISuite}}), {{TPCHQuerySuite}} and {{SSBQuerySuite}} call it on {{sql(query).queryExecution.executedPlan}}.
> Since adaptive query execution became the default (SPARK-33679, 3.2.0), that plan is an {{AdaptiveSparkPlanExec}}, a leaf node whose query stages, and so its {{WholeStageCodegenExec}} nodes, are created only when the query runs. The suites never run the queries, so the {{plan foreach}} walk in {{checkGeneratedCode}} finds no {{WholeStageCodegenExec}} and the method returns without compiling or checking anything. The tests pass vacuously.
> Evidence: walking the same way as {{checkGeneratedCode}} (the plan plus its subqueries) over all TPC-DS v1.4 (103), v2.7 (32) and modified (21) queries and all 22 TPC-H queries finds 0 whole-stage stages with {{spark.sql.adaptive.enabled=true}}. With {{spark.sql.adaptive.enabled=false}} the same walk finds about 1,700 TPC-DS stages (without injected statistics) and 126 TPC-H stages, and exactly one of them is past 8000 bytes: {{modified-q3}}'s stage (12214 bytes; 12167 with injected statistics), which the suite already excludes as SPARK-29128.
> Two consequences:
> * The "can be properly compiled" check and the method size check have not run since 3.2.0, so a codegen regression in these queries would not be caught here.
> * The workarounds in {{TPCDSQuerySuite}} guard an assertion that does not run. {{spark.sql.readSideCharPadding=false}} ("so that the generated code is less than 8000") is not needed with AQE off either: padding changes the largest method of some queries by at most about 500 bytes, and none crosses 8000.
> Proposed fix: set {{spark.sql.adaptive.enabled=false}} in {{BenchmarkQueryTest}}'s {{sparkConf}}, since these suites only compile plans, and assert that at least one {{WholeStageCodegenExec}} was found, so the check cannot become vacuous again. Keep the {{modified-q3}} exclusion, and consider dropping the char padding override.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org