[jira] [Comment Edited] (SPARK-6885) Decision trees: predict class probabilities

["Yanbo Liang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644196#comment-14644196 ] 

Yanbo Liang edited comment on SPARK-6885 at 7/28/15 10:33 AM:
--------------------------------------------------------------

[~josephkb] 
Thanks for your comments.
After survey I found that we have two candidate plan:
#1 We record the raw counts for each label in an Array[Double] at every LearningNode. That is we need to implement a new class PredictionStats which stores the "counts" array.
{code:theme=FadeToGrey|linenumbers=true|language=html/xml|firstline=0001|collapse=true}
class PredictionStats(
    val predict: Double,
    val counts: Array[Double]) extends Serializable {
}
{code}
Compared with the old Predict class, we just add more prediction statistic information.
class Predict(
    val predict: Double,
    val prob: Double = 0.0) extends Serializable {
}
And we need to make corresponding change to InformationGainStats and calculatePredictionStats(), maybe need  a new InformationGainStats which will not affect the old mllib code.
#2 We only record the raw counts for each label at leaf node of LearningNode. That is we need to implement two kinds of LearningNode (InternalLearningNode and LeafLearningNode).  
I prefer the #1, looking forward your comments.



was (Author: yanboliang):
[~josephkb] 
Thanks for your comments.
After survey I found that we have two candidate plan:
#1 We record the raw counts for each label in an Array[Double] at every LearningNode. That is we need to implement a new class PredictionStats which stores the "counts" array.
class PredictionStats(
    val predict: Double,
    val counts: Array[Double]) extends Serializable {
}
Compared with the old Predict class, we just add more prediction statistic information.
class Predict(
    val predict: Double,
    val prob: Double = 0.0) extends Serializable {
}
And we need to make corresponding change to InformationGainStats and calculatePredictionStats(), maybe need  a new InformationGainStats which will not affect the old mllib code.
#2 We only record the raw counts for each label at leaf node of LearningNode. That is we need to implement two kinds of LearningNode (InternalLearningNode and LeafLearningNode).  
I prefer the #1, looking forward your comments.


> Decision trees: predict class probabilities
> -------------------------------------------
>
>                 Key: SPARK-6885
>                 URL: https://issues.apache.org/jira/browse/SPARK-6885
>             Project: Spark
>          Issue Type: Sub-task
>          Components: ML
>    Affects Versions: 1.3.0
>            Reporter: Joseph K. Bradley
>            Assignee: Yanbo Liang
>
> Under spark.ml, have DecisionTreeClassifier (currently being added) extend ProbabilisticClassifier.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org