Seeking advice on a classification problem (needle-in-haystack situation)

[mahout-newbie <ra...@gmail.com>]

I am trying to classify a set of short text descriptions (1 - 15 words long)
into a handful of classes.  I am following the approach in the 20 newsgroup
example using Adaptive Logistic Regression. 

There are a couple of twists to the problem I am solving:
1) Only a small set of descriptions result in useful classifications - bit
like finding a needle in a haystack. My training data has a grab bag
classification called "DON'T CARE" into which 90-95% of the descriptions end
up. The remaining 5-10% are classified into roughly 10 classes.
2) There are certain words (features) in the description that immediately
imply its classification unambiguously. However, they do not occur very
frequently in the data set.

When I train and test with this data set, the overall classification
accuracy is very high (98%) except a high proportion of the incorrect
classifications occur for the descriptions I am most interested in. I guess
one should expect this intuitively! 

What's the best way to model this scenario? Is it better to exclude the
"DON'T CARE" descriptions from the training set for SGD? Because when the
proportions accurately reflect the real data set, the classification error
rate for the interesting subset is too high!

Appreciate any ideas...



--
View this message in context: http://lucene.472066.n3.nabble.com/Seeking-advice-on-a-classification-problem-needle-in-haystack-situation-tp4024362.html
Sent from the Mahout User List mailing list archive at Nabble.com.

RE: Seeking advice on a classification problem (needle-in-haystack situation)

[Jason Xin <Ja...@sas.com>]

Dear Raman Srinivasan,

In the best interest of quality analytics or ‘do it right’,

You almost certainly need a two stage classification model. In the first one, target=1 if “Don’t Care” and target=0 if otherwise. In the second model, you leave out “Don’t Care”, and conduct your regular classification. When combining the two, 95% of the chance you will not run into need for Heckman adjustment. Just watch out a bit about potential correlation among the classes in your definition phase.

In building the second model, you almost certainly do NOT want to build models on equal frequency of the input classes. Practically, when you deploy your solution, the chance that your roll-out audience will have equal frequency of classes is next to zero. If you are adamant that they are of equal frequencies, or roughly thereof, why need a model? You target them randomly, you will end up with roughly equal, and you pocket your energy and resources you are investing in this model building and do something more valuable

Technically, in logistic regression that milks on maximum likelihood, solutions tend to look elegant on training data sets with equal frequencies, but poor in resisting noise in much bigger deployment. This is the case with Mahout, R and SAS, if not others as well. Also, avoid using lest frequent class (especially if the count is really low) as your reference denomination. Use the biggest one and all other classes as it percentages

Always decide what you think the ‘true’ classes (relationships) should be in the deployment universe. Then come back to look at your training data set to make any necessary adjustment. 

Happy New Year and Happy Mahouting

Jason Xin

-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com] 
Sent: Tuesday, December 04, 2012 6:51 PM
To: user@mahout.apache.org
Subject: Re: Seeking advice on a classification problem (needle-in-haystack situation)

What Kate says is good advice.  You can have considerable amounts of bias, but you may be telling the model something about the relative cost of errors and that can result in things happening that you don't like.

As you noted, your model could have gotten 95% correct by simply saying DON'T CARE to all documents.  This is a "stopped-clock" model which is only accidentally and uninterestingly correct.

If you down-sample the DON'T CARE you can have better results because the model cannot cheat so easily.

Another thing to try is to build a binary model of CARE/DON'T CARE.  Then only on documents that are CARE, build a model of topic.  This sort of cascaded model can sometimes be much more accurate.  Downsampling DON'T CARE in the first step may still be a good thing.  In the second step, it is irrelevant.

On Wed, Dec 5, 2012 at 12:44 AM, Kate Ericson <er...@cs.colostate.edu>wrote:

> So just from an a ML approach, you want to have roughly equal amounts 
> of your target classifications in your training data - keep some 
> "DON'T CARE"s, but try not to have significantly more of them than any 
> of your 10 target classes.
>
> I hope this helps!
>
> -Kate
>
>
> On Tue, Dec 4, 2012 at 4:26 PM, mahout-newbie 
> <raman.srinivasan@gmail.com
> >wrote:
>
> > I am trying to classify a set of short text descriptions (1 - 15 
> > words
> > long)
> > into a handful of classes.  I am following the approach in the 20
> newsgroup
> > example using Adaptive Logistic Regression.
> >
> > There are a couple of twists to the problem I am solving:
> > 1) Only a small set of descriptions result in useful classifications 
> > -
> bit
> > like finding a needle in a haystack. My training data has a grab bag 
> > classification called "DON'T CARE" into which 90-95% of the 
> > descriptions end up. The remaining 5-10% are classified into roughly 
> > 10 classes.
> > 2) There are certain words (features) in the description that 
> > immediately imply its classification unambiguously. However, they do 
> > not occur very frequently in the data set.
> >
> > When I train and test with this data set, the overall classification 
> > accuracy is very high (98%) except a high proportion of the 
> > incorrect classifications occur for the descriptions I am most 
> > interested in. I
> guess
> > one should expect this intuitively!
> >
> > What's the best way to model this scenario? Is it better to exclude 
> > the "DON'T CARE" descriptions from the training set for SGD? Because 
> > when the proportions accurately reflect the real data set, the 
> > classification
> error
> > rate for the interesting subset is too high!
> >
> > Appreciate any ideas...
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Seeking-advice-on-a-classification-
> problem-needle-in-haystack-situation-tp4024362.html
> > Sent from the Mahout User List mailing list archive at Nabble.com.
> >
>

Re: Seeking advice on a classification problem (needle-in-haystack situation)

[Ted Dunning <te...@gmail.com>]

What Kate says is good advice.  You can have considerable amounts of bias,
but you may be telling the model something about the relative cost of
errors and that can result in things happening that you don't like.

As you noted, your model could have gotten 95% correct by simply saying
DON'T CARE to all documents.  This is a "stopped-clock" model which is only
accidentally and uninterestingly correct.

If you down-sample the DON'T CARE you can have better results because the
model cannot cheat so easily.

Another thing to try is to build a binary model of CARE/DON'T CARE.  Then
only on documents that are CARE, build a model of topic.  This sort of
cascaded model can sometimes be much more accurate.  Downsampling DON'T
CARE in the first step may still be a good thing.  In the second step, it
is irrelevant.

On Wed, Dec 5, 2012 at 12:44 AM, Kate Ericson <er...@cs.colostate.edu>wrote:

> So just from an a ML approach, you want to have roughly equal amounts of
> your target classifications in your training data - keep some "DON'T
> CARE"s, but try not to have significantly more of them than any of your 10
> target classes.
>
> I hope this helps!
>
> -Kate
>
>
> On Tue, Dec 4, 2012 at 4:26 PM, mahout-newbie <raman.srinivasan@gmail.com
> >wrote:
>
> > I am trying to classify a set of short text descriptions (1 - 15 words
> > long)
> > into a handful of classes.  I am following the approach in the 20
> newsgroup
> > example using Adaptive Logistic Regression.
> >
> > There are a couple of twists to the problem I am solving:
> > 1) Only a small set of descriptions result in useful classifications -
> bit
> > like finding a needle in a haystack. My training data has a grab bag
> > classification called "DON'T CARE" into which 90-95% of the descriptions
> > end
> > up. The remaining 5-10% are classified into roughly 10 classes.
> > 2) There are certain words (features) in the description that immediately
> > imply its classification unambiguously. However, they do not occur very
> > frequently in the data set.
> >
> > When I train and test with this data set, the overall classification
> > accuracy is very high (98%) except a high proportion of the incorrect
> > classifications occur for the descriptions I am most interested in. I
> guess
> > one should expect this intuitively!
> >
> > What's the best way to model this scenario? Is it better to exclude the
> > "DON'T CARE" descriptions from the training set for SGD? Because when the
> > proportions accurately reflect the real data set, the classification
> error
> > rate for the interesting subset is too high!
> >
> > Appreciate any ideas...
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Seeking-advice-on-a-classification-problem-needle-in-haystack-situation-tp4024362.html
> > Sent from the Mahout User List mailing list archive at Nabble.com.
> >
>

Re: Seeking advice on a classification problem (needle-in-haystack situation)

[Kate Ericson <er...@cs.colostate.edu>]

So just from an a ML approach, you want to have roughly equal amounts of
your target classifications in your training data - keep some "DON'T
CARE"s, but try not to have significantly more of them than any of your 10
target classes.

I hope this helps!

-Kate


On Tue, Dec 4, 2012 at 4:26 PM, mahout-newbie <ra...@gmail.com>wrote:

> I am trying to classify a set of short text descriptions (1 - 15 words
> long)
> into a handful of classes.  I am following the approach in the 20 newsgroup
> example using Adaptive Logistic Regression.
>
> There are a couple of twists to the problem I am solving:
> 1) Only a small set of descriptions result in useful classifications - bit
> like finding a needle in a haystack. My training data has a grab bag
> classification called "DON'T CARE" into which 90-95% of the descriptions
> end
> up. The remaining 5-10% are classified into roughly 10 classes.
> 2) There are certain words (features) in the description that immediately
> imply its classification unambiguously. However, they do not occur very
> frequently in the data set.
>
> When I train and test with this data set, the overall classification
> accuracy is very high (98%) except a high proportion of the incorrect
> classifications occur for the descriptions I am most interested in. I guess
> one should expect this intuitively!
>
> What's the best way to model this scenario? Is it better to exclude the
> "DON'T CARE" descriptions from the training set for SGD? Because when the
> proportions accurately reflect the real data set, the classification error
> rate for the interesting subset is too high!
>
> Appreciate any ideas...
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Seeking-advice-on-a-classification-problem-needle-in-haystack-situation-tp4024362.html
> Sent from the Mahout User List mailing list archive at Nabble.com.
>