You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "mosche (via GitHub)" <gi...@apache.org> on 2023/01/26 11:43:12 UTC

[GitHub] [beam] mosche opened a new pull request, #25187: [Spark Dataset runner] Experiment to limit execution plan complexity

mosche opened a new pull request, #25187:
URL: https://github.com/apache/beam/pull/25187

   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093354645


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java:
##########
@@ -61,25 +61,25 @@ public class PipelineTranslatorBatch extends PipelineTranslator {
   // https://github.com/seznam/euphoria/blob/master/euphoria-spark/src/main/java/cz/seznam/euphoria/spark/SparkFlowTranslator.java#L106
 
   static {
-    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch());
-    TRANSFORM_TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslatorBatch<>());
-    TRANSFORM_TRANSLATORS.put(Combine.Globally.class, new CombineGloballyTranslatorBatch<>());
+    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch(0));

Review Comment:
   That's mostly just relative factors comparing what the translator adds to the DAG, no exact science needed here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1413359920

   Thanks @aromanenko-dev , have to look into the vr tests, not sure what's going on :(


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] aromanenko-dev commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093485127


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -80,6 +83,12 @@
 public abstract class PipelineTranslator {
   private static final Logger LOG = LoggerFactory.getLogger(PipelineTranslator.class);
 
+  // Threshold to limit query plan complexity to avoid unnecessary planning overhead. Currently this
+  // is fairly low, Catalyst won't be able to optimize beyond ParDos anyways. Until there's
+  // dedicated support for schema transforms, there's little value of allowing more complex plans at
+  // this point.
+  private static final int PLAN_COMPLEXITY_THRESHOLD = 6;

Review Comment:
   Yes but planner depends on the complexity of pipeline (like initial cause of this PR and TCPDS query 83). 
   I'm ok to keep it hardcoded but I don't see any disadvantages to keep this default value and make it configurable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093429241


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java:
##########
@@ -61,25 +61,25 @@ public class PipelineTranslatorBatch extends PipelineTranslator {
   // https://github.com/seznam/euphoria/blob/master/euphoria-spark/src/main/java/cz/seznam/euphoria/spark/SparkFlowTranslator.java#L106
 
   static {
-    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch());
-    TRANSFORM_TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslatorBatch<>());
-    TRANSFORM_TRANSLATORS.put(Combine.Globally.class, new CombineGloballyTranslatorBatch<>());
+    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch(0));

Review Comment:
   No, it doesn't  ... this is only about a single PTransform (and it's translation) looked at in isolation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093488917


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -80,6 +83,12 @@
 public abstract class PipelineTranslator {
   private static final Logger LOG = LoggerFactory.getLogger(PipelineTranslator.class);
 
+  // Threshold to limit query plan complexity to avoid unnecessary planning overhead. Currently this
+  // is fairly low, Catalyst won't be able to optimize beyond ParDos anyways. Until there's
+  // dedicated support for schema transforms, there's little value of allowing more complex plans at
+  // this point.
+  private static final int PLAN_COMPLEXITY_THRESHOLD = 6;

Review Comment:
   At this point i strongly think this should not be exposed to the user ... it's deeply internal and more confusing than helpful. Happy to reevaluate if something comes up that proves otherwise.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Experiment to limit execution plan complexity

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1406205133

   Run Spark Runner Tpcds Tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] aromanenko-dev commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1406812192

   @echauchot Sure, I'll take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1412561315

   Run Spark StructuredStreaming ValidatesRunner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] aromanenko-dev commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093489087


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java:
##########
@@ -61,25 +61,25 @@ public class PipelineTranslatorBatch extends PipelineTranslator {
   // https://github.com/seznam/euphoria/blob/master/euphoria-spark/src/main/java/cz/seznam/euphoria/spark/SparkFlowTranslator.java#L106
 
   static {
-    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch());
-    TRANSFORM_TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslatorBatch<>());
-    TRANSFORM_TRANSLATORS.put(Combine.Globally.class, new CombineGloballyTranslatorBatch<>());
+    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch(0));

Review Comment:
   Then it could be just hardcoded as a constant value in every transform translator class and I don't see a reason to have a dedicated constructor for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] aromanenko-dev commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093166881


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java:
##########
@@ -61,25 +61,25 @@ public class PipelineTranslatorBatch extends PipelineTranslator {
   // https://github.com/seznam/euphoria/blob/master/euphoria-spark/src/main/java/cz/seznam/euphoria/spark/SparkFlowTranslator.java#L106
 
   static {
-    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch());
-    TRANSFORM_TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslatorBatch<>());
-    TRANSFORM_TRANSLATORS.put(Combine.Globally.class, new CombineGloballyTranslatorBatch<>());
+    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch(0));

Review Comment:
   How all this and below complexity factors were estimated?



##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -146,13 +165,37 @@ public String name() {
     public @Nullable Dataset<WindowedValue<T>> dataset() {
       return dataset;
     }
+
+    private boolean isLeaf() {
+      return dependentTransforms.isEmpty();
+    }
+
+    private int usages() {
+      return dependentTransforms.size();
+    }
+
+    private void resetPlanComplexity() {
+      planComplexity = 1;
+    }
+
+    /** Estimate complexity of query plan by multiplying complexities of all dependencies. */
+    private float estimatePlanComplexity() {
+      if (planComplexity > 0) {
+        return planComplexity;
+      }
+      float complexity = 1 + complexityFactor;
+      for (TranslationResult<?> res : dependencies) {

Review Comment:
   nit: res -> result



##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -129,12 +138,22 @@ public EvaluationContext translate(
    */
   private static final class TranslationResult<T> implements EvaluationContext.NamedDataset<T> {
     private final String name;
+    private final float complexityFactor;

Review Comment:
   Default value?



##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -80,6 +83,12 @@
 public abstract class PipelineTranslator {
   private static final Logger LOG = LoggerFactory.getLogger(PipelineTranslator.class);
 
+  // Threshold to limit query plan complexity to avoid unnecessary planning overhead. Currently this
+  // is fairly low, Catalyst won't be able to optimize beyond ParDos anyways. Until there's
+  // dedicated support for schema transforms, there's little value of allowing more complex plans at
+  // this point.
+  private static final int PLAN_COMPLEXITY_THRESHOLD = 6;

Review Comment:
   Why it is 6? Should it be configurable? 



##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -129,12 +138,22 @@ public EvaluationContext translate(
    */
   private static final class TranslationResult<T> implements EvaluationContext.NamedDataset<T> {
     private final String name;
+    private final float complexityFactor;
+    private float planComplexity = 0;
+
     private @MonotonicNonNull Dataset<WindowedValue<T>> dataset = null;
     private @MonotonicNonNull Broadcast<SideInputValues<T>> sideInputBroadcast = null;
+
+    // dependent downstream transforms (if empty this is a leaf)
     private final Set<PTransform<?, ?>> dependentTransforms = new HashSet<>();
+    // upstream dependencies (requires inputs)

Review Comment:
   nit: downstream?



##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -247,20 +292,37 @@ public <T> Dataset<WindowedValue<T>> getDataset(PCollection<T> pCollection) {
     public <T> void putDataset(
         PCollection<T> pCollection, Dataset<WindowedValue<T>> dataset, boolean cache) {
       TranslationResult<T> result = getResult(pCollection);
-      if (cache && result.dependentTransforms.size() > 1) {
-        LOG.info("Dataset {} will be cached.", result.name);
-        result.dataset = dataset.persist(storageLevel); // use NONE to disable
-      } else {
-        result.dataset = dataset;
-        if (result.dependentTransforms.isEmpty()) {
-          leaves.add(result);
+      result.dataset = dataset;
+
+      if (!cache && isMemoryOnly) {
+        result.resetPlanComplexity(); // cached as RDD in memory which breaks linage
+      } else if (cache && result.usages() > 1) {
+        if (isMemoryOnly) {
+          // Cache as RDD in-memory only, this helps to also break linage of complex query plans.
+          LOG.info("Dataset {} will be cached in-memory as RDD for reuse.", result.name);
+          result.dataset = sparkSession.createDataset(dataset.rdd().persist(), dataset.encoder());
+          result.resetPlanComplexity();
+        } else {
+          LOG.info("Dataset {} will be cached for reuse.", result.name);
+          dataset.persist(storageLevel); // use NONE to disable
         }
       }
+
+      if (result.estimatePlanComplexity() > PLAN_COMPLEXITY_THRESHOLD) {
+        // Break linage of dataset to limit planning overhead for complex query plans.
+        LOG.debug("Breaking linage of dataset {} to limit complexity of query plan.", result.name);

Review Comment:
   nit: `info` log level



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] aromanenko-dev commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1413556951

   @mosche Are you sure that it's caused by your changes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1415726168

   Run Spark Runner Tpcds Tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Experiment to limit execution plan complexity

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1404932611

   Run Spark Runner Tpcds Tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093351777


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -247,20 +292,37 @@ public <T> Dataset<WindowedValue<T>> getDataset(PCollection<T> pCollection) {
     public <T> void putDataset(
         PCollection<T> pCollection, Dataset<WindowedValue<T>> dataset, boolean cache) {
       TranslationResult<T> result = getResult(pCollection);
-      if (cache && result.dependentTransforms.size() > 1) {
-        LOG.info("Dataset {} will be cached.", result.name);
-        result.dataset = dataset.persist(storageLevel); // use NONE to disable
-      } else {
-        result.dataset = dataset;
-        if (result.dependentTransforms.isEmpty()) {
-          leaves.add(result);
+      result.dataset = dataset;
+
+      if (!cache && isMemoryOnly) {
+        result.resetPlanComplexity(); // cached as RDD in memory which breaks linage
+      } else if (cache && result.usages() > 1) {
+        if (isMemoryOnly) {
+          // Cache as RDD in-memory only, this helps to also break linage of complex query plans.
+          LOG.info("Dataset {} will be cached in-memory as RDD for reuse.", result.name);
+          result.dataset = sparkSession.createDataset(dataset.rdd().persist(), dataset.encoder());
+          result.resetPlanComplexity();
+        } else {
+          LOG.info("Dataset {} will be cached for reuse.", result.name);
+          dataset.persist(storageLevel); // use NONE to disable
         }
       }
+
+      if (result.estimatePlanComplexity() > PLAN_COMPLEXITY_THRESHOLD) {
+        // Break linage of dataset to limit planning overhead for complex query plans.
+        LOG.debug("Breaking linage of dataset {} to limit complexity of query plan.", result.name);

Review Comment:
   Compared to other log statements here this is really just worth debug level ...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093351777


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -247,20 +292,37 @@ public <T> Dataset<WindowedValue<T>> getDataset(PCollection<T> pCollection) {
     public <T> void putDataset(
         PCollection<T> pCollection, Dataset<WindowedValue<T>> dataset, boolean cache) {
       TranslationResult<T> result = getResult(pCollection);
-      if (cache && result.dependentTransforms.size() > 1) {
-        LOG.info("Dataset {} will be cached.", result.name);
-        result.dataset = dataset.persist(storageLevel); // use NONE to disable
-      } else {
-        result.dataset = dataset;
-        if (result.dependentTransforms.isEmpty()) {
-          leaves.add(result);
+      result.dataset = dataset;
+
+      if (!cache && isMemoryOnly) {
+        result.resetPlanComplexity(); // cached as RDD in memory which breaks linage
+      } else if (cache && result.usages() > 1) {
+        if (isMemoryOnly) {
+          // Cache as RDD in-memory only, this helps to also break linage of complex query plans.
+          LOG.info("Dataset {} will be cached in-memory as RDD for reuse.", result.name);
+          result.dataset = sparkSession.createDataset(dataset.rdd().persist(), dataset.encoder());
+          result.resetPlanComplexity();
+        } else {
+          LOG.info("Dataset {} will be cached for reuse.", result.name);
+          dataset.persist(storageLevel); // use NONE to disable
         }
       }
+
+      if (result.estimatePlanComplexity() > PLAN_COMPLEXITY_THRESHOLD) {
+        // Break linage of dataset to limit planning overhead for complex query plans.
+        LOG.debug("Breaking linage of dataset {} to limit complexity of query plan.", result.name);

Review Comment:
   Compared to other log statements here this is really just worth debug level ... but i'll change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1412234299

   @aromanenko-dev pls have another look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1415121356

   Tracked this down to #25296, will fix the bug first and then get back to this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1406259240

   R: @aromanenko-dev 
   R: @echauchot 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093351777


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -247,20 +292,37 @@ public <T> Dataset<WindowedValue<T>> getDataset(PCollection<T> pCollection) {
     public <T> void putDataset(
         PCollection<T> pCollection, Dataset<WindowedValue<T>> dataset, boolean cache) {
       TranslationResult<T> result = getResult(pCollection);
-      if (cache && result.dependentTransforms.size() > 1) {
-        LOG.info("Dataset {} will be cached.", result.name);
-        result.dataset = dataset.persist(storageLevel); // use NONE to disable
-      } else {
-        result.dataset = dataset;
-        if (result.dependentTransforms.isEmpty()) {
-          leaves.add(result);
+      result.dataset = dataset;
+
+      if (!cache && isMemoryOnly) {
+        result.resetPlanComplexity(); // cached as RDD in memory which breaks linage
+      } else if (cache && result.usages() > 1) {
+        if (isMemoryOnly) {
+          // Cache as RDD in-memory only, this helps to also break linage of complex query plans.
+          LOG.info("Dataset {} will be cached in-memory as RDD for reuse.", result.name);
+          result.dataset = sparkSession.createDataset(dataset.rdd().persist(), dataset.encoder());
+          result.resetPlanComplexity();
+        } else {
+          LOG.info("Dataset {} will be cached for reuse.", result.name);
+          dataset.persist(storageLevel); // use NONE to disable
         }
       }
+
+      if (result.estimatePlanComplexity() > PLAN_COMPLEXITY_THRESHOLD) {
+        // Break linage of dataset to limit planning overhead for complex query plans.
+        LOG.debug("Breaking linage of dataset {} to limit complexity of query plan.", result.name);

Review Comment:
   Compared to other log statements here this is just worth debug level ...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093424248


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -80,6 +83,12 @@
 public abstract class PipelineTranslator {
   private static final Logger LOG = LoggerFactory.getLogger(PipelineTranslator.class);
 
+  // Threshold to limit query plan complexity to avoid unnecessary planning overhead. Currently this
+  // is fairly low, Catalyst won't be able to optimize beyond ParDos anyways. Until there's
+  // dedicated support for schema transforms, there's little value of allowing more complex plans at
+  // this point.
+  private static final int PLAN_COMPLEXITY_THRESHOLD = 6;

Review Comment:
   No, as said, it only optimizes the Calcite planner, the size of the input is irrelevant.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Experiment to limit execution plan complexity

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1406179267

   Run Spark Runner Tpcds Tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1415631736

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1412728753

   Run Spark StructuredStreaming ValidatesRunner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1413542619

   🤯 wth, below the output of the following statements, somehow `rdd.persist()` corrupts the data 💥 
   ```Java
     dataset.persist(storageLevel);
   
     System.out.println("\nContent of persisted dataset (1st eval)");
     dataset.foreach(printValue);
   
     System.out.println("\nContent of persisted dataset (2nd eval)");
     dataset.foreach(printValue);
   
     System.out.println("\nContent of rdd (1st eval)");
     dataset.rdd().foreach(printValue);
   
     System.out.println("\nContent of rdd (2nd eval)");
     dataset.rdd().foreach(printValue);
   
     System.out.println("\nContent of persisted rdd (1st eval)");
     dataset.rdd().persist().foreach(printValue);
   
     System.out.println("\nContent of persisted rdd (2nd eval)");
     dataset.rdd().persist().foreach(printValue);
   ```
   
   ```
   Content of persisted dataset (1st eval)
   key=k3 {@1797166500}, value=[0] {@621459392}
   key=k5 {@522613115}, value=[2147483647, -2147483648] {@50229760}
   key=k1 {@935655811}, value=[3, 4] {@1059339408}
   key=k2 {@1698407107}, value=[66, -33] {@1059339408}
   
   Content of persisted dataset (2nd eval)
   key=k3 {@491772294}, value=[0] {@1519705721}
   key=k1 {@1801297116}, value=[3, 4] {@929237815}
   key=k2 {@1278247606}, value=[66, -33] {@929237815}
   key=k5 {@1642777670}, value=[2147483647, -2147483648] {@822592932}
   
   Content of rdd (1st eval)
   key=k5 {@1706950017}, value=[2147483647, -2147483648] {@935984436}
   key=k1 {@830413362}, value=[3, 4] {@1431942185}
   key=k3 {@1807383422}, value=[0] {@1220036903}
   key=k2 {@1334173112}, value=[66, -33] {@1431942185}
   
   Content of rdd (2nd eval)
   key=k1 {@1977617218}, value=[3, 4] {@1966593054}
   key=k2 {@131385719}, value=[66, -33] {@1966593054}
   key=k3 {@966124643}, value=[0] {@4762028}
   key=k5 {@1477114665}, value=[2147483647, -2147483648] {@942837752}
   
   Content of persisted rdd (1st eval)
   key=k3 {@2050847337}, value=[0] {@83038769}
   key=k5 {@906982325}, value=[2147483647, -2147483648] {@714957170}
   key=k1 {@654493203}, value=[66, -33] {@1747607045}
   key=k2 {@287079803}, value=[66, -33] {@1747607045}
   
   Content of persisted rdd (2nd eval)
   key=k1 {@654493203}, value=[66, -33] {@1747607045}
   key=k2 {@287079803}, value=[66, -33] {@1747607045}
   key=k5 {@906982325}, value=[2147483647, -2147483648] {@714957170}
   key=k3 {@2050847337}, value=[0] {@83038769}
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] aromanenko-dev commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093489087


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java:
##########
@@ -61,25 +61,25 @@ public class PipelineTranslatorBatch extends PipelineTranslator {
   // https://github.com/seznam/euphoria/blob/master/euphoria-spark/src/main/java/cz/seznam/euphoria/spark/SparkFlowTranslator.java#L106
 
   static {
-    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch());
-    TRANSFORM_TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslatorBatch<>());
-    TRANSFORM_TRANSLATORS.put(Combine.Globally.class, new CombineGloballyTranslatorBatch<>());
+    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch(0));

Review Comment:
   Then it could be just hardcoded as a constant value in every translator class and I don't see a reason to have a dedicated constructor for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche merged pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche merged PR #25187:
URL: https://github.com/apache/beam/pull/25187


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093493395


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java:
##########
@@ -61,25 +61,25 @@ public class PipelineTranslatorBatch extends PipelineTranslator {
   // https://github.com/seznam/euphoria/blob/master/euphoria-spark/src/main/java/cz/seznam/euphoria/spark/SparkFlowTranslator.java#L106
 
   static {
-    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch());
-    TRANSFORM_TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslatorBatch<>());
-    TRANSFORM_TRANSLATORS.put(Combine.Globally.class, new CombineGloballyTranslatorBatch<>());
+    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch(0));

Review Comment:
   That's the way i implemented it initially... I personally find it easier to have an overview of all values in a single place, but happy to revert. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1456032767

   fixed here https://github.com/apache/beam/pull/25732


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1455711164

   Thanks @Abacn , having a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Experiment to limit execution plan complexity

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1404889956

   Run Spark Runner Tpcds Tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093364824


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -80,6 +83,12 @@
 public abstract class PipelineTranslator {
   private static final Logger LOG = LoggerFactory.getLogger(PipelineTranslator.class);
 
+  // Threshold to limit query plan complexity to avoid unnecessary planning overhead. Currently this
+  // is fairly low, Catalyst won't be able to optimize beyond ParDos anyways. Until there's
+  // dedicated support for schema transforms, there's little value of allowing more complex plans at
+  // this point.
+  private static final int PLAN_COMPLEXITY_THRESHOLD = 6;

Review Comment:
   Honestly, this is mostly optimizing for the benchmark ... the proportional overhead of the planner will decrease the more data there is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093346371


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -129,12 +138,22 @@ public EvaluationContext translate(
    */
   private static final class TranslationResult<T> implements EvaluationContext.NamedDataset<T> {
     private final String name;
+    private final float complexityFactor;
+    private float planComplexity = 0;
+
     private @MonotonicNonNull Dataset<WindowedValue<T>> dataset = null;
     private @MonotonicNonNull Broadcast<SideInputValues<T>> sideInputBroadcast = null;
+
+    // dependent downstream transforms (if empty this is a leaf)
     private final Set<PTransform<?, ?>> dependentTransforms = new HashSet<>();
+    // upstream dependencies (requires inputs)

Review Comment:
   This is the upstream dependencies in terms of data flow, so the data this depends on. Above the downstream dependencies, so transform that use this as input.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1413794166

   @aromanenko-dev It's triggered by this optional part https://github.com/apache/beam/pull/25187/files#diff-4df56f442668d45bf7269e0bc379e95298b178c2f3072c72f30c1c0c296caed9R301-R304
   
   I could simply remove that and keep caching as dataset. But there's also other places where caching is done on the RDD rather than the dataset if storage level is MEMORY_ONLY. It's a bit concerning to now know why / how this bug is triggered :(


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] aromanenko-dev commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093416009


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -80,6 +83,12 @@
 public abstract class PipelineTranslator {
   private static final Logger LOG = LoggerFactory.getLogger(PipelineTranslator.class);
 
+  // Threshold to limit query plan complexity to avoid unnecessary planning overhead. Currently this
+  // is fairly low, Catalyst won't be able to optimize beyond ParDos anyways. Until there's
+  // dedicated support for schema transforms, there's little value of allowing more complex plans at
+  // this point.
+  private static final int PLAN_COMPLEXITY_THRESHOLD = 6;

Review Comment:
   Can this value affect a pipeline performance significantly depending on user case (like a complexity of pipeline, size and type of input data, etc)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1415496535

   Run Spark StructuredStreaming ValidatesRunner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1454318593

   Hi  @mosche and @aromanenko-dev Either this PR or #25297 likely caused SparkStructuredStreaming Batch Load test failing sinve Feb 2: https://ci-beam.apache.org/view/LoadTests/job/beam_LoadTests_Java_ParDo_SparkStructuredStreaming_Batch/1016/ are there action being taken?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] echauchot commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "echauchot (via GitHub)" <gi...@apache.org>.
echauchot commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1406267956

   @aromanenko-dev I'll be in vacations tonight, can you take care of this review ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1415677762

   Run Spark Runner Tpcds Tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on PR #25187:
URL: https://github.com/apache/beam/pull/25187#issuecomment-1411679438

   @aromanenko-dev kind ping :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093349999


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -80,6 +83,12 @@
 public abstract class PipelineTranslator {
   private static final Logger LOG = LoggerFactory.getLogger(PipelineTranslator.class);
 
+  // Threshold to limit query plan complexity to avoid unnecessary planning overhead. Currently this
+  // is fairly low, Catalyst won't be able to optimize beyond ParDos anyways. Until there's
+  // dedicated support for schema transforms, there's little value of allowing more complex plans at
+  // this point.
+  private static final int PLAN_COMPLEXITY_THRESHOLD = 6;

Review Comment:
   That was a decent value when testing. There's no point in having this configurable, that way users would only worry about it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093358113


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -129,12 +138,22 @@ public EvaluationContext translate(
    */
   private static final class TranslationResult<T> implements EvaluationContext.NamedDataset<T> {
     private final String name;
+    private final float complexityFactor;

Review Comment:
   `final` and always set in constructor ...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] aromanenko-dev commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "aromanenko-dev (via GitHub)" <gi...@apache.org>.
aromanenko-dev commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093418876


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java:
##########
@@ -61,25 +61,25 @@ public class PipelineTranslatorBatch extends PipelineTranslator {
   // https://github.com/seznam/euphoria/blob/master/euphoria-spark/src/main/java/cz/seznam/euphoria/spark/SparkFlowTranslator.java#L106
 
   static {
-    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch());
-    TRANSFORM_TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslatorBatch<>());
-    TRANSFORM_TRANSLATORS.put(Combine.Globally.class, new CombineGloballyTranslatorBatch<>());
+    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch(0));

Review Comment:
   Does it depend on a translated pipeline?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093358704


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java:
##########
@@ -146,13 +165,37 @@ public String name() {
     public @Nullable Dataset<WindowedValue<T>> dataset() {
       return dataset;
     }
+
+    private boolean isLeaf() {
+      return dependentTransforms.isEmpty();
+    }
+
+    private int usages() {
+      return dependentTransforms.size();
+    }
+
+    private void resetPlanComplexity() {
+      planComplexity = 1;
+    }
+
+    /** Estimate complexity of query plan by multiplying complexities of all dependencies. */
+    private float estimatePlanComplexity() {
+      if (planComplexity > 0) {
+        return planComplexity;
+      }
+      float complexity = 1 + complexityFactor;
+      for (TranslationResult<?> res : dependencies) {

Review Comment:
   👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mosche commented on a diff in pull request #25187: [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large query plans

Posted by "mosche (via GitHub)" <gi...@apache.org>.
mosche commented on code in PR #25187:
URL: https://github.com/apache/beam/pull/25187#discussion_r1093525891


##########
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java:
##########
@@ -61,25 +61,25 @@ public class PipelineTranslatorBatch extends PipelineTranslator {
   // https://github.com/seznam/euphoria/blob/master/euphoria-spark/src/main/java/cz/seznam/euphoria/spark/SparkFlowTranslator.java#L106
 
   static {
-    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch());
-    TRANSFORM_TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslatorBatch<>());
-    TRANSFORM_TRANSLATORS.put(Combine.Globally.class, new CombineGloballyTranslatorBatch<>());
+    TRANSFORM_TRANSLATORS.put(Impulse.class, new ImpulseTranslatorBatch(0));

Review Comment:
   ok, done @aromanenko-dev 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org