[GitHub] spark pull request #21927: [SPARK-24820][SPARK-24821][Core] Fail fast when s...

[squito <gi...@git.apache.org>]

Github user squito commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21927#discussion_r206985923
  
    --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ---
    @@ -340,6 +340,22 @@ class DAGScheduler(
         }
       }
     
    +  /**
    +   * Check to make sure we don't launch a barrier stage with unsupported RDD chain pattern. The
    +   * following patterns are not supported:
    +   * 1. Ancestor RDDs that have different number of partitions from the resulting RDD (eg.
    +   * union()/coalesce()/first()/PartitionPruningRDD);
    --- End diff --
    
    but coalesce should be OK, right?  Is it just too fragile to allow coalesce while excluding the others?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org