You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Anton Lykov (Jira)" <ji...@apache.org> on 2024/12/17 14:33:00 UTC

[jira] [Created] (SPARK-50597) Refactor batch creation style in Optimizer

Anton Lykov created SPARK-50597:
-----------------------------------

             Summary: Refactor batch creation style in Optimizer
                 Key: SPARK-50597
                 URL: https://issues.apache.org/jira/browse/SPARK-50597
             Project: Spark
          Issue Type: Improvement
          Components: Optimizer
    Affects Versions: 4.0.0
            Reporter: Anton Lykov


It is a pain to reorder batches and guard behavior of certain batches / sequences of batches by a flag. This was primarily due to ample usage of `{{{}::`{}}}, `{{{}:::`{}}}, and {{`:+`}} to juggle rules and batches around which imposed syntactic limitations. For example, see here: https://github.com/apache/spark/pull/49208/files#diff-11264d807efa58054cca2d220aae8fba644ee0f0f2a4722c46d52828394846efL209.

We'd like to refactor to use unified logic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org