You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Loic Descotte <lo...@kelkoo.com> on 2011/09/21 17:22:24 UTC

Classification + glossary usage

Hi,

I'm currently working on a text classification problem.
As my learning datasets are rather small (dozens of entries), I'm 
looking for a solution to use a glossary in addition to the learning 
phase, to tune the model.

I know that some text entries containing some keywords have great chance 
to be in a specific category. With SGD, I've tried to create a new 
Vector with this keywords and I added this to the current category's 
vector during the learning.

It works pretty good if I put a big weight on it (700).

My "hack" code looks like this :


       List<String> keywords = ... //keywords for the current category

       for (String keyword : keywords) {
          predictorEncoders.get(99).addToVector(keyword, 700, 
featureVector);
       }

The 99 predictor is a new predictor I've created just for this keywords :

FeatureVectorEncoder keyWordEncoder = 
TYPE_DICTIONARY.get("text").getConstructor(String.class).newInstance("keywords");
predictorEncoders.put(99, keyWordEncoder);


It works pretty well, my confusion matrix is better with this hack, but 
maybe it's not optimal because this attribute does not exists in the 
train/test data.

Did someone experienced this kind of things? Do you have advices? Or is 
it just a wrong idea?

Thanks!

Loic

Re: Classification + glossary usage

Posted by Loic Descotte <lo...@kelkoo.com>.
Yes I was guessing this kind of problem. My problem is that I will not 
always have hundred of learnig data for new categories.
This is why we wanted to improve our model with manually selected keywords.

Maybe it would be better to split this into 2 phases and use the keyword 
extraction phase only for categories with poor historical values.

Thanks for your help it's becoming clearer to me now :)

Le 22.09.2011 21:04, Ted Dunning a écrit :
> If these keywords already appear in these other fields, I think that you
> should just let the algorithm find them.  I think that your problem is
> insufficient training data or insufficient number of passes on the data.
>
> In general, you should not do anything to your training data that you will
> not later do to your production data.
>
> On Thu, Sep 22, 2011 at 10:04 AM, Loic Descotte<lo...@gmail.com>wrote:
>
>>>   Have you seen the Mahout book?
>>
>> Yes I've bought your (very good) book in early access preview. It helps me
>> a lot in my investigations.
>>
>>> ?! If a feature is not found in the production data, then you should not
>> give it to the model as a predictor during training. Otherwise, you have a
>> form of target leak.
>>
>> I think I did'nt explain myself very well, sorry.
>>
>> I mean that here is no attribute named 'keyword' in my test or train data.
>> But all the keywords I put when I create my new vector appear in the other
>> attributes of my datas (body and title)
>>
>> I've selected them because I know they will occur very often in body and
>> title.
>>
>> I was just worried about creating a "fake" attribute name (keyword), like
>> this :
>>
>> for (String keyword : keywords) {
>>   predictorEncoders.get(99).addToVector(keyword, 700, featureVector);
>>
>>
>> (The 99 predictor is a new predictor I've created just for this keywords)
>>
>> But it seems to work (with big weights), keywords seems to be found in
>> other attributes because when I do this my results are getting better in the
>> confusion matrix.
>>
>> So is it ok to do like this or it is still a dirty hack?
>>
>> Loic
>>>
>>>
>>
>>


Re: Classification + glossary usage

Posted by Ted Dunning <te...@gmail.com>.
If these keywords already appear in these other fields, I think that you
should just let the algorithm find them.  I think that your problem is
insufficient training data or insufficient number of passes on the data.

In general, you should not do anything to your training data that you will
not later do to your production data.

On Thu, Sep 22, 2011 at 10:04 AM, Loic Descotte <lo...@gmail.com>wrote:

> >  Have you seen the Mahout book?
>
>
> Yes I've bought your (very good) book in early access preview. It helps me
> a lot in my investigations.
>
> > ?! If a feature is not found in the production data, then you should not
> give it to the model as a predictor during training. Otherwise, you have a
> form of target leak.
>
> I think I did'nt explain myself very well, sorry.
>
> I mean that here is no attribute named 'keyword' in my test or train data.
> But all the keywords I put when I create my new vector appear in the other
> attributes of my datas (body and title)
>
> I've selected them because I know they will occur very often in body and
> title.
>
> I was just worried about creating a "fake" attribute name (keyword), like
> this :
>
> for (String keyword : keywords) {
>  predictorEncoders.get(99).addToVector(keyword, 700, featureVector);
>
>
> (The 99 predictor is a new predictor I've created just for this keywords)
>
> But it seems to work (with big weights), keywords seems to be found in
> other attributes because when I do this my results are getting better in the
> confusion matrix.
>
> So is it ok to do like this or it is still a dirty hack?
>
> Loic
> >
> >
> >
>
>
>

Re: Classification + glossary usage

Posted by Loic Descotte <lo...@gmail.com>.
>  Have you seen the Mahout book? 


Yes I've bought your (very good) book in early access preview. It helps me a lot in my investigations.

> ?! If a feature is not found in the production data, then you should not give it to the model as a predictor during training. Otherwise, you have a form of target leak.

I think I did'nt explain myself very well, sorry.

I mean that here is no attribute named 'keyword' in my test or train data.
But all the keywords I put when I create my new vector appear in the other attributes of my datas (body and title)

I've selected them because I know they will occur very often in body and title.

I was just worried about creating a "fake" attribute name (keyword), like this :

for (String keyword : keywords) {
 predictorEncoders.get(99).addToVector(keyword, 700, featureVector);


(The 99 predictor is a new predictor I've created just for this keywords)

But it seems to work (with big weights), keywords seems to be found in other attributes because when I do this my results are getting better in the confusion matrix.

So is it ok to do like this or it is still a dirty hack?

Loic
> 
> 
> 



Re: Classification + glossary usage

Posted by Ted Dunning <te...@gmail.com>.
On Thu, Sep 22, 2011 at 12:36 AM, Loic Descotte <lo...@kelkoo.com>wrote:

> ...
>
>  One thing that you said that worried me is your comment about putting a
>> really high weight on the feature.  I am surprised that this was required.
>>
>
> Maybe it is because my training data are not good enough...
> What would be a "reasonable" order of height for such a weight?
>

Normally features do not need special weighting.  If one feature is good and
others are noisy, then it may take a little time before the model figures
out which is which, but it will figure it out even with reasonably equal
weighting.


> Another thing that made me feel it could be a dirty hack is that the
> 'keyword' feature added manually during learning cannot be found in test or
> production data because it does not exist :
>
> My train data are text files. For example, news articles.
> They have one class, the category : politics, sport, finance...
> They have 2 attributes : title and body.
>
> So the keywords I add with the code I put in the previous mail is
> completely "virtual". It will never be found in title or body. Despite that,
> I guess that the SGD alorithm looks for the keywords into the other
> attributes (title and body) to match the right category.
>

?!

If a feature is not found in the production data, then you should not give
it to the model as a predictor during training. Otherwise, you have a form
of target leak.

Have you seen the Mahout book?  I went into extensive detail about this and
provided examples of how high quality features available at training only
could lead to poor training.

Re: Classification + glossary usage

Posted by Loic Descotte <lo...@kelkoo.com>.
Hi,

Thanks for your help,

> If you have a really good, cheap feature, it is criminal not
> to expose that feature to the classifier

Yes i's cheap because it's a human process that has already been done 
for most categories. For each one, we have a set of representative keywords.

> One thing that you said that worried me is your comment about putting a
> really high weight on the feature.  I am surprised that this was required.

Maybe it is because my training data are not good enough...
What would be a "reasonable" order of height for such a weight?

Another thing that made me feel it could be a dirty hack is that the 
'keyword' feature added manually during learning cannot be found in test 
or production data because it does not exist :

My train data are text files. For example, news articles.
They have one class, the category : politics, sport, finance...
They have 2 attributes : title and body.

So the keywords I add with the code I put in the previous mail is 
completely "virtual". It will never be found in title or body. Despite 
that, I guess that the SGD alorithm looks for the keywords into the 
other attributes (title and body) to match the right category.

But maybe it's the reason why it works only with very big weight?

Loic

Le 21.09.2011 20:32, Ted Dunning a écrit :
> This comes under the heading of "inventing good features".
>
> In real data mining projects, easily 90% of the effort goes into this sort
> of activity.  If you have a really good, cheap feature, it is criminal not
> to expose that feature to the classifier.
>
> If you don't really have the feature because it is expensive or requires a
> time machine to derive, then obviously you have to do something different.
>
> One thing that you said that worried me is your comment about putting a
> really high weight on the feature.  I am surprised that this was required.
>
> On Wed, Sep 21, 2011 at 8:22 AM, Loic Descotte<lo...@kelkoo.com>wrote:
>
>> Did someone experienced this kind of things? Do you have advices? Or is it
>> just a wrong idea?
>>


Re: Classification + glossary usage

Posted by Ted Dunning <te...@gmail.com>.
This comes under the heading of "inventing good features".

In real data mining projects, easily 90% of the effort goes into this sort
of activity.  If you have a really good, cheap feature, it is criminal not
to expose that feature to the classifier.

If you don't really have the feature because it is expensive or requires a
time machine to derive, then obviously you have to do something different.

One thing that you said that worried me is your comment about putting a
really high weight on the feature.  I am surprised that this was required.

On Wed, Sep 21, 2011 at 8:22 AM, Loic Descotte <lo...@kelkoo.com>wrote:

> Did someone experienced this kind of things? Do you have advices? Or is it
> just a wrong idea?
>