You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dong Wang (Jira)" <ji...@apache.org> on 2019/11/09 09:37:00 UTC
[jira] [Created] (SPARK-29810) Missing persist on retaggedInput in
RandomForest.run()
Dong Wang created SPARK-29810:
---------------------------------
Summary: Missing persist on retaggedInput in RandomForest.run()
Key: SPARK-29810
URL: https://issues.apache.org/jira/browse/SPARK-29810
Project: Spark
Issue Type: Improvement
Components: ML
Affects Versions: 2.4.3
Reporter: Dong Wang
The rdd retaggedInput should be persisted in ml.tree.impl.RandomForest.run(), because it will be used more than one actions.
{code:scala}
def run(
input: RDD[LabeledPoint],
strategy: OldStrategy,
numTrees: Int,
featureSubsetStrategy: String,
seed: Long,
instr: Option[Instrumentation],
prune: Boolean = true, // exposed for testing only, real trees are always pruned
parentUID: Option[String] = None): Array[DecisionTreeModel] = {
val timer = new TimeTracker()
timer.start("total")
timer.start("init")
val retaggedInput = input.retag(classOf[LabeledPoint]) // it needs to be persisted
{code}
This issue is reported by our tool CacheCheck, which is used to dynamically detecting persist()/unpersist() api misuses.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org