You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2018/07/11 14:43:52 UTC

[GitHub] spark pull request #21526: [SPARK-24515][CORE] No need to warning when outpu...

Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21526#discussion_r201717626
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala ---
    @@ -1053,7 +1053,10 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
         // users that they may loss data if they are using a direct output committer.
         val speculationEnabled = self.conf.getBoolean("spark.speculation", false)
         val outputCommitterClass = hadoopConf.get("mapred.output.committer.class", "")
    -    if (speculationEnabled && outputCommitterClass.contains("Direct")) {
    +    val outputCommitCoordinationEnabled = self.conf.getBoolean(
    +      "spark.hadoop.outputCommitCoordination.enabled", true)
    --- End diff --
    
    since we are touching it, can we define this config in `org.apache.spark.internal.config`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org