You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ilya (Jira)" <ji...@apache.org> on 2026/09/03 13:12:00 UTC

[jira] [Commented] (SPARK-50992) OOMs and performance issues with AQE in large plans

    [ https://issues.apache.org/jira/browse/SPARK-50992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111097#comment-18111097 ] 

Ilya commented on SPARK-50992:
------------------------------

We can reproduce the OOM behavior described here on Apache Spark 4.1.3 in an iterative DataFrame workload using AQE and cached DataFrames.

Environment:
 * Apache Spark: 4.1.3

 * Scala: 2.13.17

 * Java: OpenJDK 17.0.19

 * CatBoost Spark: ai.catboost:catboost-spark_4.1_2.13:1.2.10

With the default value of {{{}spark.sql.maxPlanStringLength{}}}, the Spark 4.1.3 driver eventually fails with:

{{java.lang.OutOfMemoryError: Java heap space}}

Changing only:

{{spark.sql.maxPlanStringLength=8192}}

prevents the driver OOM.

However, it does not fix the underlying planning-time degradation. The workload continues running, but planning time grows very rapidly in later iterations.

For example, wall-clock time around construction of a downstream lazy transform, with no Spark action executed inside the measured section, grows as:

step 11: 0.322 s
step 12: 0.703 s
step 13: 0.989 s
step 14: 1.570 s
step 15: 2.956 s
step 16: 5.797 s
step 17: 11.336 s
step 18: 22.606 s

In another run it continued to:

step 19: 45.476 s
step 20: 94.096 s
step 21: 179.974 s
step 22: 367.955 s

This A/B appears consistent with {{spark.sql.maxPlanStringLength}} mitigating the memory impact of very large plan string representations while not addressing the underlying large-plan / AQE planning overhead.

We also observe a Spark 3.4.3 -> 4.1.3 performance regression in the same iterative cached-DataFrame workload. Disabling AQE only while materializing the iterative cached state prevents the planning-time explosion while AQE remains enabled for the rest of the application.

I am planning to report that behavior separately with a standalone synthetic PySpark reproducer.

I can provide a sanitized OOM stack trace and additional per-iteration timing data if they are useful.

> OOMs and performance issues with AQE in large plans
> ---------------------------------------------------
>
>                 Key: SPARK-50992
>                 URL: https://issues.apache.org/jira/browse/SPARK-50992
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.5.3, 3.5.4, 4.0.0
>            Reporter: Ángel Álvarez Pascua
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: Main.scala
>
>
> When AQE is enabled, Spark triggers update events to the internal listener bus whenever a plan changes. These events include a plain-text description of the plan, which is computationally expensive to generate for large plans.
> {*}Key Issues:{*}{*}{{*}}
> *1. High Cost of Plan String Calculation:*
>  * Generating the string description for large physical plans is a costly operation.
>  * This impacts performance, particularly in complex workflows with frequent plan updates (e.g. persisting DataFrames).
> *2. Out-of-Memory (OOM) Errors:*
>  * Events are stored in the listener bus as {{SQLExecutionUIData}} objects and retained until a threshold is reached.
>  * This retention behavior can lead to memory exhaustion when processing large plans, causing OOM errors.
>  
> *Current Workarounds Are Ineffective:*
>  * *Reducing Retained Executions* ({{{}spark.sql.ui.retainedExecutions{}}}): Even when set to {{1}} or {{{}0{}}}, events are still created, requiring plan string calculations.
>  * *Limiting Plan String Length* ({{{}spark.sql.maxPlanStringLength{}}}): Reducing the maximum string length (e.g., to {{{}1,000,000{}}}) may mitigate OOMs but does not eliminate the overhead of string generation.
>  * *Available Explain Modes:* All existing explain modes are verbose and computationally expensive, failing to resolve these issues.
>  
> *Proposed Solution:*
> Introduce a new explain mode, {*}{{off}}{*}, which suppresses the generation of plan string descriptions.
>  * When this mode is enabled, Spark skips the calculation of plan descriptions altogether.
>  * This resolves OOM errors and restores performance parity with non-AQE execution.
>  
> *Impact of Proposed Solution:*
>  * Eliminates OOMs in large plans with AQE enabled.
>  * Reduces the performance overhead associated with plan string generation.
>  * Ensures Spark scales better in environments with large, complex plans.
>  
> *Reproducibility:*
> A test reproducing the issue has been attached.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org