You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Hector Yee <he...@gmail.com> on 2011/05/20 20:21:55 UTC
Classifier Interface
Hi,
I noticed that classifier has three functions to call to get the score.
classify - returns probabilities
classifyNoLink - returns the raw score (optional)
classifyScalar - returns the binary probability
I'm working on a few classifiers for which it doesn't make sense to return
probability. In fact, the probability is just the raw score exponentiated.
This would distort the scores a bit, rather than if the user just used the
raw score directly. Also, if they assume that the scores are really
probabilities they may be tempted to use it to compare between two
classifiers without previously calibrating on a test set.
I wonder if we can add classifiyScalarNoLink and make the NoLinks
non-optional. They just return probabilities if you're using a classifier
that returns in the 0-1 range.
This way people can choose to use either interface primarily, rather than
calling classify and assume all classifiers support probabilities.
Finally, there's some algorithms that can return regression / ranking or
classification scores depending on the training data. I was just planning to
return the same value via classifiyScalarNoLink but it seems to be a poorly
named proposed function. I could just name the function 'score' but it would
break the naming convention already set down.
Thoughts?
--
Yee Yang Li Hector
http://hectorgon.blogspot.com/ (tech + travel)
http://hectorgon.com (book reviews)
Re: Classifier Interface
Posted by Ted Dunning <te...@gmail.com>.
The argument for the no-link versions is that it allows partial evaluation
of models which in turn allows
substantial amounts of computation to be done off-line. This makes >10x
difference in speed in some
cases. See chapters 16 and 17 of MiA. The contract here should be that if
your classifier can be factored
into a linear combination of sub-models, said linear combination that is
normally transformed, *NoLink
should return that value.
Yes, it should be NoInverseLink, but I got tired fo typing.
The argument for the *Scalar forms is that in the binomial case we can avoid
allocation or copying of a result
entirely. This makes a noticeable difference in performance in a few cases.
The argument for the *Full forms is that the partial forms are cheaper to
compute for many classifiers.
On Fri, May 20, 2011 at 1:04 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Probably better ClassifierResult.classify(ClassifyResult holder), for
> the memory sake..
>
> I like the idea a lot. Except multinomial case may encounter a lot of
> unnecessary copying really so again, tragically critical for iteration
> time here.
>
> On Fri, May 20, 2011 at 1:01 PM, Grant Ingersoll <gs...@apache.org>
> wrote:
> > Perhaps we should future proof here a little bit and simply have a
> classify method that returns a typed object that contains the necessary info
> depending on the implementation? Something like:
> > ClassifierResult classify()
> >
> > and then ClassifierResult has an enum or something that indicates whether
> one should grab the Vector, Matrix or double. Just brainstorming...
> >
> >
> > On May 20, 2011, at 2:21 PM, Hector Yee wrote:
> >
> >> Hi,
> >>
> >> I noticed that classifier has three functions to call to get the score.
> >> classify - returns probabilities
> >> classifyNoLink - returns the raw score (optional)
> >> classifyScalar - returns the binary probability
> >>
> >> I'm working on a few classifiers for which it doesn't make sense to
> return
> >> probability. In fact, the probability is just the raw score
> exponentiated.
> >> This would distort the scores a bit, rather than if the user just used
> the
> >> raw score directly. Also, if they assume that the scores are really
> >> probabilities they may be tempted to use it to compare between two
> >> classifiers without previously calibrating on a test set.
> >>
> >> I wonder if we can add classifiyScalarNoLink and make the NoLinks
> >> non-optional. They just return probabilities if you're using a
> classifier
> >> that returns in the 0-1 range.
> >> This way people can choose to use either interface primarily, rather
> than
> >> calling classify and assume all classifiers support probabilities.
> >>
> >> Finally, there's some algorithms that can return regression / ranking or
> >> classification scores depending on the training data. I was just
> planning to
> >> return the same value via classifiyScalarNoLink but it seems to be a
> poorly
> >> named proposed function. I could just name the function 'score' but it
> would
> >> break the naming convention already set down.
> >>
> >> Thoughts?
> >>
> >> --
> >> Yee Yang Li Hector
> >> http://hectorgon.blogspot.com/ (tech + travel)
> >> http://hectorgon.com (book reviews)
> >
> >
> >
>
Re: Classifier Interface
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Probably better ClassifierResult.classify(ClassifyResult holder), for
the memory sake..
I like the idea a lot. Except multinomial case may encounter a lot of
unnecessary copying really so again, tragically critical for iteration
time here.
On Fri, May 20, 2011 at 1:01 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Perhaps we should future proof here a little bit and simply have a classify method that returns a typed object that contains the necessary info depending on the implementation? Something like:
> ClassifierResult classify()
>
> and then ClassifierResult has an enum or something that indicates whether one should grab the Vector, Matrix or double. Just brainstorming...
>
>
> On May 20, 2011, at 2:21 PM, Hector Yee wrote:
>
>> Hi,
>>
>> I noticed that classifier has three functions to call to get the score.
>> classify - returns probabilities
>> classifyNoLink - returns the raw score (optional)
>> classifyScalar - returns the binary probability
>>
>> I'm working on a few classifiers for which it doesn't make sense to return
>> probability. In fact, the probability is just the raw score exponentiated.
>> This would distort the scores a bit, rather than if the user just used the
>> raw score directly. Also, if they assume that the scores are really
>> probabilities they may be tempted to use it to compare between two
>> classifiers without previously calibrating on a test set.
>>
>> I wonder if we can add classifiyScalarNoLink and make the NoLinks
>> non-optional. They just return probabilities if you're using a classifier
>> that returns in the 0-1 range.
>> This way people can choose to use either interface primarily, rather than
>> calling classify and assume all classifiers support probabilities.
>>
>> Finally, there's some algorithms that can return regression / ranking or
>> classification scores depending on the training data. I was just planning to
>> return the same value via classifiyScalarNoLink but it seems to be a poorly
>> named proposed function. I could just name the function 'score' but it would
>> break the naming convention already set down.
>>
>> Thoughts?
>>
>> --
>> Yee Yang Li Hector
>> http://hectorgon.blogspot.com/ (tech + travel)
>> http://hectorgon.com (book reviews)
>
>
>
Re: Classifier Interface
Posted by Grant Ingersoll <gs...@apache.org>.
Perhaps we should future proof here a little bit and simply have a classify method that returns a typed object that contains the necessary info depending on the implementation? Something like:
ClassifierResult classify()
and then ClassifierResult has an enum or something that indicates whether one should grab the Vector, Matrix or double. Just brainstorming...
On May 20, 2011, at 2:21 PM, Hector Yee wrote:
> Hi,
>
> I noticed that classifier has three functions to call to get the score.
> classify - returns probabilities
> classifyNoLink - returns the raw score (optional)
> classifyScalar - returns the binary probability
>
> I'm working on a few classifiers for which it doesn't make sense to return
> probability. In fact, the probability is just the raw score exponentiated.
> This would distort the scores a bit, rather than if the user just used the
> raw score directly. Also, if they assume that the scores are really
> probabilities they may be tempted to use it to compare between two
> classifiers without previously calibrating on a test set.
>
> I wonder if we can add classifiyScalarNoLink and make the NoLinks
> non-optional. They just return probabilities if you're using a classifier
> that returns in the 0-1 range.
> This way people can choose to use either interface primarily, rather than
> calling classify and assume all classifiers support probabilities.
>
> Finally, there's some algorithms that can return regression / ranking or
> classification scores depending on the training data. I was just planning to
> return the same value via classifiyScalarNoLink but it seems to be a poorly
> named proposed function. I could just name the function 'score' but it would
> break the naming convention already set down.
>
> Thoughts?
>
> --
> Yee Yang Li Hector
> http://hectorgon.blogspot.com/ (tech + travel)
> http://hectorgon.com (book reviews)
Re: Classifier Interface
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
i guess I am confused what you are proposing.
This api (i think) is about GLM, so noLink cannot return "probability"
(whenever it is actually a probability), the intent here AFAICT is to
return raw linear score before appling G^-1 (binomial case implied).
AFAICT "noLink" really means "noInverseLink" applied
It struck me as a little bit inconsistent between binomial and
multinomial cases, too, but i think i failed to follow your
argumntation so far.
I was thinking of using raw scores to build model hierarchies (ideas
similar to least angle regression and incremental factorizers except
you deal with predictors in groups instead of one-by-one) in which
case what we have _almost_ fits all the cases.
On Fri, May 20, 2011 at 11:21 AM, Hector Yee <he...@gmail.com> wrote:
> Hi,
>
> I noticed that classifier has three functions to call to get the score.
> classify - returns probabilities
> classifyNoLink - returns the raw score (optional)
> classifyScalar - returns the binary probability
>
> I'm working on a few classifiers for which it doesn't make sense to return
> probability. In fact, the probability is just the raw score exponentiated.
> This would distort the scores a bit, rather than if the user just used the
> raw score directly. Also, if they assume that the scores are really
> probabilities they may be tempted to use it to compare between two
> classifiers without previously calibrating on a test set.
>
> I wonder if we can add classifiyScalarNoLink and make the NoLinks
> non-optional. They just return probabilities if you're using a classifier
> that returns in the 0-1 range.
> This way people can choose to use either interface primarily, rather than
> calling classify and assume all classifiers support probabilities.
>
> Finally, there's some algorithms that can return regression / ranking or
> classification scores depending on the training data. I was just planning to
> return the same value via classifiyScalarNoLink but it seems to be a poorly
> named proposed function. I could just name the function 'score' but it would
> break the naming convention already set down.
>
> Thoughts?
>
> --
> Yee Yang Li Hector
> http://hectorgon.blogspot.com/ (tech + travel)
> http://hectorgon.com (book reviews)
>
Re: Classifier Interface
Posted by Ted Dunning <te...@gmail.com>.
I agree that enforcing probabilities doesn't always make sense. You don't
have to invoke 0-1 loss for this.
Much of my own experience is with fraud models that have grotesquely
uncalibrated score outputs due to strange sampling, cascaded models and so
on. These models may look like probabilities on the surface, but it is
silly to take that too seriously.
Even so, I find it very useful to use evaluation metrics that are relatively
independent of a particular threshold because that lets me separate the
threshold question from the model quality question. Somewhat, anyway.
On Fri, May 20, 2011 at 6:49 PM, Hector Yee <he...@gmail.com> wrote:
> Hope that helps. By the way, it seems I'm approaching machine learning from
> a rather different point of view (empirical loss minimization that
> indirectly try to minimize 0-1 loss rather than the probabilistic
> approach),
> which is why enforcing probabilities on them don't make much sense.
>
Re: Classifier Interface
Posted by Ted Dunning <te...@gmail.com>.
Nearest neighbor techniques generally average over k nearest neighbors.
This average makes a fine score. If k = 1, then score can only take on
values 0 and 1, but that isn't a problem.
So I think that the interface makes sense. There isn't a link function, but
it is traditional to take that as meaning that the link function is the
identity function rather than that it doesn't exist.
Or did I miss your point?
On Fri, May 20, 2011 at 7:33 PM, Daniel McEnnis <dm...@gmail.com> wrote:
> All of this assumes classifiers supply a score. Nearest Neighbor
> classification can not provide a meaningful score, only the result.
> So there is at least one algorithm where whole sets of the classify
> interface makes absolutely no sense.
>
Re: Classifier Interface
Posted by Lance Norskog <go...@gmail.com>.
This is a general design problem with what Mahout is trying to do. All
of the algorithms in use do some things and not other things, but it
is really worthwhile to park them under a common API. The result is
that your code may call something doesn't work, or really mean
anything, in algorithm X. Our general approach is to throw
UnsupportedOperationException.
On Fri, May 20, 2011 at 7:50 PM, Hector Yee <he...@gmail.com> wrote:
> Ok, so the conclusion is let's add classifyScalarNoLink and make it optional
> as well by throwing an exception.
>
> On Sat, May 21, 2011 at 10:33 AM, Daniel McEnnis <dm...@gmail.com> wrote:
>
>> All of this assumes classifiers supply a score. Nearest Neighbor
>> classification can not provide a meaningful score, only the result.
>> So there is at least one algorithm where whole sets of the classify
>> interface makes absolutely no sense.
>>
>> Daniel.
>>
>> On Fri, May 20, 2011 at 10:09 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>> > On Fri, May 20, 2011 at 6:49 PM, Hector Yee <he...@gmail.com>
>> wrote:
>> >
>> >> I was concerned about classifyScalar because it enforces the contract
>> that
>> >> the scores be in the 0..1 range. There doesn't seem to be a function
>> that
>> >> returns the raw score for the scalar case.
>> >>
>> >
>> > Yes. There should be. The only reason that there isn't is that there
>> had
>> > not been any users of this yet.
>> >
>> >
>> >> ...
>> >> I was proposing that we have all classifiers support classifyNoLink as
>> >> well,
>> >> especially for the case of non-probabilistic based ones where reducing
>> >> scores to probabilities would be bad.
>> >>
>> >
>> > Why?
>> >
>> > This is the part that I don't understand. Why force this implementation?
>> >
>>
>
>
>
> --
> Yee Yang Li Hector
> http://hectorgon.blogspot.com/ (tech + travel)
> http://hectorgon.com (book reviews)
>
--
Lance Norskog
goksron@gmail.com
Re: Classifier Interface
Posted by Ted Dunning <te...@gmail.com>.
+1
On Fri, May 20, 2011 at 7:50 PM, Hector Yee <he...@gmail.com> wrote:
> Ok, so the conclusion is let's add classifyScalarNoLink and make it
> optional
> as well by throwing an exception.
>
> On Sat, May 21, 2011 at 10:33 AM, Daniel McEnnis <dm...@gmail.com>
> wrote:
>
> > All of this assumes classifiers supply a score. Nearest Neighbor
> > classification can not provide a meaningful score, only the result.
> > So there is at least one algorithm where whole sets of the classify
> > interface makes absolutely no sense.
> >
> > Daniel.
> >
> > On Fri, May 20, 2011 at 10:09 PM, Ted Dunning <te...@gmail.com>
> > wrote:
> > > On Fri, May 20, 2011 at 6:49 PM, Hector Yee <he...@gmail.com>
> > wrote:
> > >
> > >> I was concerned about classifyScalar because it enforces the contract
> > that
> > >> the scores be in the 0..1 range. There doesn't seem to be a function
> > that
> > >> returns the raw score for the scalar case.
> > >>
> > >
> > > Yes. There should be. The only reason that there isn't is that there
> > had
> > > not been any users of this yet.
> > >
> > >
> > >> ...
> > >> I was proposing that we have all classifiers support classifyNoLink as
> > >> well,
> > >> especially for the case of non-probabilistic based ones where reducing
> > >> scores to probabilities would be bad.
> > >>
> > >
> > > Why?
> > >
> > > This is the part that I don't understand. Why force this
> implementation?
> > >
> >
>
>
>
> --
> Yee Yang Li Hector
> http://hectorgon.blogspot.com/ (tech + travel)
> http://hectorgon.com (book reviews)
>
Re: Classifier Interface
Posted by Hector Yee <he...@gmail.com>.
Ok, so the conclusion is let's add classifyScalarNoLink and make it optional
as well by throwing an exception.
On Sat, May 21, 2011 at 10:33 AM, Daniel McEnnis <dm...@gmail.com> wrote:
> All of this assumes classifiers supply a score. Nearest Neighbor
> classification can not provide a meaningful score, only the result.
> So there is at least one algorithm where whole sets of the classify
> interface makes absolutely no sense.
>
> Daniel.
>
> On Fri, May 20, 2011 at 10:09 PM, Ted Dunning <te...@gmail.com>
> wrote:
> > On Fri, May 20, 2011 at 6:49 PM, Hector Yee <he...@gmail.com>
> wrote:
> >
> >> I was concerned about classifyScalar because it enforces the contract
> that
> >> the scores be in the 0..1 range. There doesn't seem to be a function
> that
> >> returns the raw score for the scalar case.
> >>
> >
> > Yes. There should be. The only reason that there isn't is that there
> had
> > not been any users of this yet.
> >
> >
> >> ...
> >> I was proposing that we have all classifiers support classifyNoLink as
> >> well,
> >> especially for the case of non-probabilistic based ones where reducing
> >> scores to probabilities would be bad.
> >>
> >
> > Why?
> >
> > This is the part that I don't understand. Why force this implementation?
> >
>
--
Yee Yang Li Hector
http://hectorgon.blogspot.com/ (tech + travel)
http://hectorgon.com (book reviews)
Re: Classifier Interface
Posted by Daniel McEnnis <dm...@gmail.com>.
All of this assumes classifiers supply a score. Nearest Neighbor
classification can not provide a meaningful score, only the result.
So there is at least one algorithm where whole sets of the classify
interface makes absolutely no sense.
Daniel.
On Fri, May 20, 2011 at 10:09 PM, Ted Dunning <te...@gmail.com> wrote:
> On Fri, May 20, 2011 at 6:49 PM, Hector Yee <he...@gmail.com> wrote:
>
>> I was concerned about classifyScalar because it enforces the contract that
>> the scores be in the 0..1 range. There doesn't seem to be a function that
>> returns the raw score for the scalar case.
>>
>
> Yes. There should be. The only reason that there isn't is that there had
> not been any users of this yet.
>
>
>> ...
>> I was proposing that we have all classifiers support classifyNoLink as
>> well,
>> especially for the case of non-probabilistic based ones where reducing
>> scores to probabilities would be bad.
>>
>
> Why?
>
> This is the part that I don't understand. Why force this implementation?
>
Re: Classifier Interface
Posted by Ted Dunning <te...@gmail.com>.
On Fri, May 20, 2011 at 6:49 PM, Hector Yee <he...@gmail.com> wrote:
> I was concerned about classifyScalar because it enforces the contract that
> the scores be in the 0..1 range. There doesn't seem to be a function that
> returns the raw score for the scalar case.
>
Yes. There should be. The only reason that there isn't is that there had
not been any users of this yet.
> ...
> I was proposing that we have all classifiers support classifyNoLink as
> well,
> especially for the case of non-probabilistic based ones where reducing
> scores to probabilities would be bad.
>
Why?
This is the part that I don't understand. Why force this implementation?
Re: Classifier Interface
Posted by Ted Dunning <te...@gmail.com>.
Well, the PR curves won't be distorted if the scores are transformed this
way (certainly not for the binary case where PR makes sense). Nor will the
ROC curve be changed.
That said, I still think it is reasonable for any diagnostic to be allowed
to call classifyScalarNoLink.
But I don't understand how that makes it good to require the noLink option.
Defining a default version that throws USOE seems less invasive.
On Fri, May 20, 2011 at 6:49 PM, Hector Yee <he...@gmail.com> wrote:
> Here's a concrete example. Suppose we make a generic program to load any
> abstract vector classifier to plot the precision recall curves for the
> binary case. If we use classifyScalar exclusively some classifiers would
> have distorted curves due to pushing them through an exponent first.
> However
> if they all support classifyScalarNoLink we could give the users an option
> of picking a threshold for their desired recall in the raw score space,
> without the added distortion introduced by forcing the scores in the 0..1
> range.
>
Re: Classifier Interface
Posted by Hector Yee <he...@gmail.com>.
classifyScalarNoLink doesn't exist yet, it was a proposed addition to mirror
the classifyNoLink (vector form), so it won't hurt any existing classifiers.
'score' was a proposed name for it.
I was concerned about classifyScalar because it enforces the contract that
the scores be in the 0..1 range. There doesn't seem to be a function that
returns the raw score for the scalar case.
@Ted, there is a throw new UnsupportedOperationException in the
AbstractVectorClassifier.java's implementation of classifyNoLink, which
implies that all classifiers must support classify but do not have to
support classifyNoLink.
I was proposing that we have all classifiers support classifyNoLink as well,
especially for the case of non-probabilistic based ones where reducing
scores to probabilities would be bad.
Here's a concrete example. Suppose we make a generic program to load any
abstract vector classifier to plot the precision recall curves for the
binary case. If we use classifyScalar exclusively some classifiers would
have distorted curves due to pushing them through an exponent first. However
if they all support classifyScalarNoLink we could give the users an option
of picking a threshold for their desired recall in the raw score space,
without the added distortion introduced by forcing the scores in the 0..1
range.
Hope that helps. By the way, it seems I'm approaching machine learning from
a rather different point of view (empirical loss minimization that
indirectly try to minimize 0-1 loss rather than the probabilistic approach),
which is why enforcing probabilities on them don't make much sense.
On Sat, May 21, 2011 at 9:31 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> On Fri, May 20, 2011 at 2:44 PM, Ted Dunning <te...@gmail.com>
> wrote:
> > score is a reasonable name. classifiyScalarNoLink is fairly descriptive
> if
> > you
> > know the jargon, but score may be better. One problem I have is that
> people
> > are already using this code in production so name changes are a bit
> painful.
> >
>
> +1 yes please don't change if no good reason
>
--
Yee Yang Li Hector
http://hectorgon.blogspot.com/ (tech + travel)
http://hectorgon.com (book reviews)
Re: Classifier Interface
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Fri, May 20, 2011 at 2:44 PM, Ted Dunning <te...@gmail.com> wrote:
> score is a reasonable name. classifiyScalarNoLink is fairly descriptive if
> you
> know the jargon, but score may be better. One problem I have is that people
> are already using this code in production so name changes are a bit painful.
>
+1 yes please don't change if no good reason
Re: Classifier Interface
Posted by Ted Dunning <te...@gmail.com>.
On Fri, May 20, 2011 at 11:21 AM, Hector Yee <he...@gmail.com> wrote:
> I'm working on a few classifiers for which it doesn't make sense to return
> probability. In fact, the probability is just the raw score exponentiated.
> This would distort the scores a bit, rather than if the user just used the
> raw score directly. Also, if they assume that the scores are really
> probabilities they may be tempted to use it to compare between two
> classifiers without previously calibrating on a test set.
>
I am not too worried about the calibration issue since it is reasonable to
handle that with documentation.
Returning raw scores without the exponentiation is a natural think to do
with the noLink form.
>
> I wonder if we can add classifiyScalarNoLink and make the NoLinks
> non-optional. They just return probabilities if you're using a classifier
> that returns in the 0-1 range.
> This way people can choose to use either interface primarily, rather than
> calling classify and assume all classifiers support probabilities.
>
I can't tell quiet what you are suggesting here. I think you have the tail
of a good
idea, but I can't see the spots on it yet.
Can you be more concrete about what you are proposing?
>
> Finally, there's some algorithms that can return regression / ranking or
> classification scores depending on the training data. I was just planning
> to
> return the same value via classifiyScalarNoLink but it seems to be a poorly
> named proposed function. I could just name the function 'score' but it
> would
> break the naming convention already set down.
>
score is a reasonable name. classifiyScalarNoLink is fairly descriptive if
you
know the jargon, but score may be better. One problem I have is that people
are already using this code in production so name changes are a bit painful.
I do think that returning scores without reducing to the 0..1 range is an
important
operation.