[jira] [Created] (SPARK-59855) Analyze long withColumns chains in one pass and remove the eager analysis in Spark Connect

["Wenchen Fan (Jira)" <ji...@apache.org>]

Wenchen Fan created SPARK-59855:
-----------------------------------

             Summary: Analyze long withColumns chains in one pass and remove the eager analysis in Spark Connect
                 Key: SPARK-59855
                 URL: https://issues.apache.org/jira/browse/SPARK-59855
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Wenchen Fan


Spark Connect plans each withColumns / withColumnsRenamed call lazily as Project(Seq(UnresolvedStarWithColumns(...)), child) (SPARK-50601), so a loop of df = df.withColumn(...) becomes a stack of such Projects. The fixed-point analyzer resolves at most one of them per iteration of the Resolution batch, and a chain of about 100 calls hits spark.sql.analyzer.maxIterations. SPARK-53505 worked around this by eagerly analyzing each withColumns in the Connect planner, which is quadratic and binds the plan when it's built (e.g. pipeline flows lose their dependency on datasets that already exist in the catalog).

This ticket makes the analyzer resolve a lazily planned withColumns as soon as its child is resolved, in isolation from the child plan, so the whole chain is resolved in one pass, and removes the eager analysis from the Connect planner.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org