You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Kent Yao (Jira)" <ji...@apache.org> on 2026/05/20 04:30:00 UTC

[jira] [Assigned] (SPARK-56948) TPCDSQueryBenchmark: make hardcoded SparkConf overridable via setIfMissing

     [ https://issues.apache.org/jira/browse/SPARK-56948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kent Yao reassigned SPARK-56948:
--------------------------------

    Assignee: Kent Yao

> TPCDSQueryBenchmark: make hardcoded SparkConf overridable via setIfMissing
> --------------------------------------------------------------------------
>
>                 Key: SPARK-56948
>                 URL: https://issues.apache.org/jira/browse/SPARK-56948
>             Project: Spark
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: 5.0.0
>            Reporter: Kent Yao
>            Assignee: Kent Yao
>            Priority: Major
>              Labels: pull-request-available
>
> Currently TPCDSQueryBenchmark hardcodes spark.driver.memory=3g, spark.executor.memory=3g, and spark.sql.autoBroadcastJoinThreshold=20MB via .set(...), which takes precedence over -Dspark.* system properties.
> This prevents tuning heap/broadcast when running the benchmark at larger scale factors (SF10 / SF100 / SF1000), where the 3g default OOMs.
> spark.sql.shuffle.partitions already supports override via System.getProperty in the same file -- extend the same pattern (via setIfMissing for consistency) to the remaining three keys. Defaults unchanged, fully backward compatible.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org