You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2015/02/10 05:59:01 UTC
spark git commit: Add a config option to print DAG.
Repository: spark
Updated Branches:
refs/heads/master 08488c175 -> 31d435ecf
Add a config option to print DAG.
Add a config option "spark.rddDebug.enable" to check whether to print DAG info. When "spark.rddDebug.enable" is true, it will print information about DAG in the log.
Author: KaiXinXiaoLei <hu...@huawei.com>
Closes #4257 from KaiXinXiaoLei/DAGprint and squashes the following commits:
d9fe42e [KaiXinXiaoLei] change log info
c27ee76 [KaiXinXiaoLei] change log info
83c2b32 [KaiXinXiaoLei] change config option
adcb14f [KaiXinXiaoLei] change the file.
f4e7b9e [KaiXinXiaoLei] add a option to print DAG
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/31d435ec
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/31d435ec
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/31d435ec
Branch: refs/heads/master
Commit: 31d435ecfdc24a788a6e38f4e82767bc275a3283
Parents: 08488c1
Author: KaiXinXiaoLei <hu...@huawei.com>
Authored: Mon Feb 9 20:58:58 2015 -0800
Committer: Reynold Xin <rx...@databricks.com>
Committed: Mon Feb 9 20:58:58 2015 -0800
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/SparkContext.scala | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/31d435ec/core/src/main/scala/org/apache/spark/SparkContext.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 71bdbc9..8d3c3d0 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -1420,6 +1420,9 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
val callSite = getCallSite
val cleanedFunc = clean(func)
logInfo("Starting job: " + callSite.shortForm)
+ if (conf.getBoolean("spark.logLineage", false)) {
+ logInfo("RDD's recursive dependencies:\n" + rdd.toDebugString)
+ }
dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, allowLocal,
resultHandler, localProperties.get)
progressBar.foreach(_.finishAll())
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org