You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by MariusFS <ma...@sien.com> on 2014/12/02 17:44:01 UTC
Re: MLlib Naive Bayes classifier confidence
Are we sure that exponentiating will give us the probabilities? I did some
tests by cloning the MLLIb class and adding the required code but the
calculated probabilities do not add up to 1.
I tried something like :
def predictProbs(testData: Vector): (BDV[Double], BDV[Double]) = {
val logProbs = brzPi + brzTheta * new BDV[Double](testData.toArray)
val probs = logProbs.map(x => math.exp(x))
(logProbs, probs)
}
This was because I need the actual probs to process downstream from this...
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Naive-Bayes-classifier-confidence-tp18456p20175.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: MLlib Naive Bayes classifier confidence
Posted by MariusFS <ma...@sien.com>.
That was it, Thanks. (Posting here so people know it's the right answer in
case they have the same need :) ).
sowen wrote
> Probabilities won't sum to 1 since this expression doesn't incorporate
> the probability of the evidence, I imagine? it's constant across
> classes so is usually excluded. It would appear as a "-
> log(P(evidence))" term.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Naive-Bayes-classifier-confidence-tp18456p20361.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: MLlib Naive Bayes classifier confidence
Posted by Sean Owen <so...@cloudera.com>.
Probabilities won't sum to 1 since this expression doesn't incorporate
the probability of the evidence, I imagine? it's constant across
classes so is usually excluded. It would appear as a "-
log(P(evidence))" term.
On Tue, Dec 2, 2014 at 10:44 AM, MariusFS <ma...@sien.com> wrote:
> Are we sure that exponentiating will give us the probabilities? I did some
> tests by cloning the MLLIb class and adding the required code but the
> calculated probabilities do not add up to 1.
>
> I tried something like :
>
> def predictProbs(testData: Vector): (BDV[Double], BDV[Double]) = {
> val logProbs = brzPi + brzTheta * new BDV[Double](testData.toArray)
> val probs = logProbs.map(x => math.exp(x))
> (logProbs, probs)
> }
>
> This was because I need the actual probs to process downstream from this...
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-Naive-Bayes-classifier-confidence-tp18456p20175.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org