You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Claudia Grieco <gr...@crmpa.unisa.it> on 2011/04/13 11:12:21 UTC
Identify "less similar" documents
Hi guys,
I'm using SGD to classify a set of documents but I have a problem: there are
some documents that are not related to any of the categories and I want to
be able to identify them and exclude them from the classification. My idea
is to read the documents of the training set (that are currently in a Lucene
index) and identify the docs that have less terms in common with them. Any
idea on how to do it?
Thanks a lot
Claudia
Re: Identify "less similar" documents
Posted by Daniel McEnnis <dm...@gmail.com>.
Claudia,
The term to look up is 'one class classifier'. Its built on this
problem with a set of solutions pre-made. I don't know if anyone has
put it in a general classifier before, but the theory is there.
Daniel.
On Wed, Apr 13, 2011 at 11:56 AM, Claudia Grieco <gr...@crmpa.unisa.it> wrote:
> Thanks for the help :)
>> Why not just train with those documents and put a category tag of "other" on
>>them and run normal categorization? If you can distinguish these documents
>>by word frequencies, then this should do the trick.
> I don't know if this will help
> 1)I'm still not sure where to put the threshold (if a document has word frequency less than X...how to choose X?)
> 2)The classifier is built incrementally: a document who would be classified as "other" today may be classified as "new category the user has just added" tomorrow. New docs in the training set and new categories are added from time to time.
>
> -----Messaggio originale-----
> Da: Ted Dunning [mailto:ted.dunning@gmail.com]
> Inviato: mercoledì 13 aprile 2011 17.34
> A: user@mahout.apache.org
> Cc: Claudia Grieco
> Oggetto: Re: Identify "less similar" documents
>
> I think that what you are doing is inventing an "other" category and
> building a classifier for that category.
>
> Why not just train with those documents and put a category tag of "other" on
> them and run normal categorization? If you can distinguish these documents
> by word frequencies, then this should do the trick.
>
> On Wed, Apr 13, 2011 at 7:49 AM, Claudia Grieco <gr...@crmpa.unisa.it>wrote:
>
>> Let's see if this approach makes sense:
>> I have the documents to classify on a Lucene index (Index A) and the
>> training set in another Lucene index (Index B).
>> With a VectorMapper I map Term-Frequency Vectors of Index A to
>> Term-Frequency Vectors of Index B. In this way the transformed vectors have
>> only the frequency of the terms of the training set.
>> By computing vector.zSum() I should get the frequency of the terms in the
>> training set for the document, right?
>> I compute vector.zSum() for all the docs to classify and exclude from the
>> classification the ones who have a sum value of less than 10% the max
>> vector.zSum()=>they mostly contain words never seen before and could be
>> classified wrongly.
>>
>> What do you think?
>>
>> -----Messaggio originale-----
>> Da: Claudia Grieco [mailto:grieco@crmpa.unisa.it]
>> Inviato: mercoledì 13 aprile 2011 11.12
>> A: user@mahout.apache.org
>> Oggetto: Identify "less similar" documents
>>
>> Hi guys,
>>
>> I'm using SGD to classify a set of documents but I have a problem: there
>> are
>> some documents that are not related to any of the categories and I want to
>> be able to identify them and exclude them from the classification. My idea
>> is to read the documents of the training set (that are currently in a
>> Lucene
>> index) and identify the docs that have less terms in common with them. Any
>> idea on how to do it?
>>
>> Thanks a lot
>>
>> Claudia
>>
>>
>>
>
>
Re: Identify "less similar" documents
Posted by Daniel McEnnis <dm...@gmail.com>.
Claudia,
This gets into 'Goodness metrics' - measures of how good a cluster is.
This metric is effectively the max distance metric - the maximum
distance from a vector to its cluster mean. It is a less common but
still useful metric. The most commonly used is average distance to the
cluster mean.
Daniel.
On Wed, Apr 20, 2011 at 10:35 AM, Claudia Grieco <gr...@crmpa.unisa.it> wrote:
> Thanks again.
>
> Does the radius of the cluster give information on the tightness of the cluster?
>
>
>
>
>
> Da: Ted Dunning [mailto:ted.dunning@gmail.com]
> Inviato: martedì 19 aprile 2011 18.57
> A: user@mahout.apache.org
> Cc: Claudia Grieco
> Oggetto: Re: Identify "less similar" documents
>
>
>
> Yes. This makes sense.
>
>
>
> I think you might want to qualify X according to which cluster is closest. Define a function that estimates the percentile distance for members of each cluster. There will be one function per cluster.
>
>
>
> Then define a function for each new point that is the percentile score based on the distance to the nearest cluster. The issue with what you suggest is that some clusters are very tight and others very loose.
>
> On Tue, Apr 19, 2011 at 2:55 AM, Claudia Grieco <gr...@crmpa.unisa.it> wrote:
>
> Thanks for the suggestion, I'm currently trying this hack:
> I take the documents of the training set and put in each cluster all the docs of a certain category.
> I compute the centroid for each category cluster
> I compute the distance of each new document to all centroids (I'm using CosineDistanceMeasure) and I identify as "outlier" the ones who have distance more than X
>
> Do you think this makes sense?
>
>
>
>
Re: Identify "less similar" documents
Posted by Ted Dunning <te...@gmail.com>.
Yes. The distribution of radii of a cluster tells you how tight it is. As
Daniel mentions, that is different from how good the cluster is.
On Wed, Apr 20, 2011 at 7:35 AM, Claudia Grieco <gr...@crmpa.unisa.it>wrote:
> Does the radius of the cluster give information on the tightness of the
> cluster?
>
R: Identify "less similar" documents
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
Thanks again.
Does the radius of the cluster give information on the tightness of the cluster?
Da: Ted Dunning [mailto:ted.dunning@gmail.com]
Inviato: martedì 19 aprile 2011 18.57
A: user@mahout.apache.org
Cc: Claudia Grieco
Oggetto: Re: Identify "less similar" documents
Yes. This makes sense.
I think you might want to qualify X according to which cluster is closest. Define a function that estimates the percentile distance for members of each cluster. There will be one function per cluster.
Then define a function for each new point that is the percentile score based on the distance to the nearest cluster. The issue with what you suggest is that some clusters are very tight and others very loose.
On Tue, Apr 19, 2011 at 2:55 AM, Claudia Grieco <gr...@crmpa.unisa.it> wrote:
Thanks for the suggestion, I'm currently trying this hack:
I take the documents of the training set and put in each cluster all the docs of a certain category.
I compute the centroid for each category cluster
I compute the distance of each new document to all centroids (I'm using CosineDistanceMeasure) and I identify as "outlier" the ones who have distance more than X
Do you think this makes sense?
Re: Identify "less similar" documents
Posted by Ted Dunning <te...@gmail.com>.
Yes. This makes sense.
I think you might want to qualify X according to which cluster is closest.
Define a function that estimates the percentile distance for members of
each cluster. There will be one function per cluster.
Then define a function for each new point that is the percentile score based
on the distance to the nearest cluster. The issue with what you suggest is
that some clusters are very tight and others very loose.
On Tue, Apr 19, 2011 at 2:55 AM, Claudia Grieco <gr...@crmpa.unisa.it>wrote:
> Thanks for the suggestion, I'm currently trying this hack:
> I take the documents of the training set and put in each cluster all the
> docs of a certain category.
> I compute the centroid for each category cluster
> I compute the distance of each new document to all centroids (I'm using
> CosineDistanceMeasure) and I identify as "outlier" the ones who have
> distance more than X
>
> Do you think this makes sense?
>
R: Identify "less similar" documents
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
Thanks for the suggestion, I'm currently trying this hack:
I take the documents of the training set and put in each cluster all the docs of a certain category.
I compute the centroid for each category cluster
I compute the distance of each new document to all centroids (I'm using CosineDistanceMeasure) and I identify as "outlier" the ones who have distance more than X
Do you think this makes sense?
Thanks
Claudia
-----Messaggio originale-----
Da: Lance Norskog [mailto:goksron@gmail.com]
Inviato: venerdì 15 aprile 2011 4.27
A: user@mahout.apache.org
Oggetto: Re: Identify "less similar" documents
Thinking of this in terms of clustering, outliers/misfits are
one-vector clusters, items that are far away from all others.
Clustering would be a slow system for finding these outliers, but an
interesting way to check them:
Cluster a sampled set of your items. Save the centroid and radius of
each cluster. To verify an outlier, look at its distance to all
centroids, and whether it is in the radius of the closest (few).
Given a clustering algorithm you like, and a different distance method
than the categorization measure, this gives a good cross-check of the
outliers.
On 4/14/11, Ted Dunning <te...@gmail.com> wrote:
> Hand classify all the documents that you can into the categories that you
> know.
>
> Classify the ones that don't fit into "other".
>
> On Thu, Apr 14, 2011 at 12:51 AM, Claudia Grieco
> <gr...@crmpa.unisa.it>wrote:
>
>> Thanks to everyone :)
>> So I should train the category "other" with some documents...but what
>> documents?
>> I should identify them first...that's a bit of a "chicken and egg" problem
>> Maybe I should do this way:
>> -each day X new documents arrive to be classified
>> -I find 10-11 docs with a low word freq in respect to the training set(but
>> what is a "low" value?) and train them as other
>> -classify everything with the updated classifier
>>
>> -----Messaggio originale-----
>> Da: Ted Dunning [mailto:ted.dunning@gmail.com]
>> Inviato: mercoledì 13 aprile 2011 19.29
>> A: user@mahout.apache.org
>> Cc: Claudia Grieco
>> Oggetto: Re: Identify "less similar" documents
>>
>> On Wed, Apr 13, 2011 at 8:56 AM, Claudia Grieco <grieco@crmpa.unisa.it
>> >wrote:
>>
>> > Thanks for the help :)
>> > > Why not just train with those documents and put a category tag of
>> "other"
>> > on
>> > >them and run normal categorization? If you can distinguish these
>> > documents
>> > >by word frequencies, then this should do the trick.
>> > I don't know if this will help
>> >
>>
>> Only an experiment will tell you.
>>
>>
>> > 1)I'm still not sure where to put the threshold (if a document has word
>> > frequency less than X...how to choose X?)
>> >
>>
>> The classifier should handle that for you for the most part. Again,
>> experimentation is the way to go here. My first cut would be to assign to
>> the category with the highest score, possibly including the other
>> category.
>>
>>
>> > 2)The classifier is built incrementally: a document who would be
>> classified
>> > as "other" today may be classified as "new category the user has just
>> added"
>> > tomorrow. New docs in the training set and new categories are added from
>> > time to time.
>> >
>>
>> That is pretty easy. Just retrain with the new category assignments.
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Identify "less similar" documents
Posted by Lance Norskog <go...@gmail.com>.
Thinking of this in terms of clustering, outliers/misfits are
one-vector clusters, items that are far away from all others.
Clustering would be a slow system for finding these outliers, but an
interesting way to check them:
Cluster a sampled set of your items. Save the centroid and radius of
each cluster. To verify an outlier, look at its distance to all
centroids, and whether it is in the radius of the closest (few).
Given a clustering algorithm you like, and a different distance method
than the categorization measure, this gives a good cross-check of the
outliers.
On 4/14/11, Ted Dunning <te...@gmail.com> wrote:
> Hand classify all the documents that you can into the categories that you
> know.
>
> Classify the ones that don't fit into "other".
>
> On Thu, Apr 14, 2011 at 12:51 AM, Claudia Grieco
> <gr...@crmpa.unisa.it>wrote:
>
>> Thanks to everyone :)
>> So I should train the category "other" with some documents...but what
>> documents?
>> I should identify them first...that's a bit of a "chicken and egg" problem
>> Maybe I should do this way:
>> -each day X new documents arrive to be classified
>> -I find 10-11 docs with a low word freq in respect to the training set(but
>> what is a "low" value?) and train them as other
>> -classify everything with the updated classifier
>>
>> -----Messaggio originale-----
>> Da: Ted Dunning [mailto:ted.dunning@gmail.com]
>> Inviato: mercoledì 13 aprile 2011 19.29
>> A: user@mahout.apache.org
>> Cc: Claudia Grieco
>> Oggetto: Re: Identify "less similar" documents
>>
>> On Wed, Apr 13, 2011 at 8:56 AM, Claudia Grieco <grieco@crmpa.unisa.it
>> >wrote:
>>
>> > Thanks for the help :)
>> > > Why not just train with those documents and put a category tag of
>> "other"
>> > on
>> > >them and run normal categorization? If you can distinguish these
>> > documents
>> > >by word frequencies, then this should do the trick.
>> > I don't know if this will help
>> >
>>
>> Only an experiment will tell you.
>>
>>
>> > 1)I'm still not sure where to put the threshold (if a document has word
>> > frequency less than X...how to choose X?)
>> >
>>
>> The classifier should handle that for you for the most part. Again,
>> experimentation is the way to go here. My first cut would be to assign to
>> the category with the highest score, possibly including the other
>> category.
>>
>>
>> > 2)The classifier is built incrementally: a document who would be
>> classified
>> > as "other" today may be classified as "new category the user has just
>> added"
>> > tomorrow. New docs in the training set and new categories are added from
>> > time to time.
>> >
>>
>> That is pretty easy. Just retrain with the new category assignments.
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Identify "less similar" documents
Posted by Ted Dunning <te...@gmail.com>.
Hand classify all the documents that you can into the categories that you
know.
Classify the ones that don't fit into "other".
On Thu, Apr 14, 2011 at 12:51 AM, Claudia Grieco <gr...@crmpa.unisa.it>wrote:
> Thanks to everyone :)
> So I should train the category "other" with some documents...but what
> documents?
> I should identify them first...that's a bit of a "chicken and egg" problem
> Maybe I should do this way:
> -each day X new documents arrive to be classified
> -I find 10-11 docs with a low word freq in respect to the training set(but
> what is a "low" value?) and train them as other
> -classify everything with the updated classifier
>
> -----Messaggio originale-----
> Da: Ted Dunning [mailto:ted.dunning@gmail.com]
> Inviato: mercoledì 13 aprile 2011 19.29
> A: user@mahout.apache.org
> Cc: Claudia Grieco
> Oggetto: Re: Identify "less similar" documents
>
> On Wed, Apr 13, 2011 at 8:56 AM, Claudia Grieco <grieco@crmpa.unisa.it
> >wrote:
>
> > Thanks for the help :)
> > > Why not just train with those documents and put a category tag of
> "other"
> > on
> > >them and run normal categorization? If you can distinguish these
> > documents
> > >by word frequencies, then this should do the trick.
> > I don't know if this will help
> >
>
> Only an experiment will tell you.
>
>
> > 1)I'm still not sure where to put the threshold (if a document has word
> > frequency less than X...how to choose X?)
> >
>
> The classifier should handle that for you for the most part. Again,
> experimentation is the way to go here. My first cut would be to assign to
> the category with the highest score, possibly including the other category.
>
>
> > 2)The classifier is built incrementally: a document who would be
> classified
> > as "other" today may be classified as "new category the user has just
> added"
> > tomorrow. New docs in the training set and new categories are added from
> > time to time.
> >
>
> That is pretty easy. Just retrain with the new category assignments.
>
>
R: Identify "less similar" documents
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
Thanks to everyone :)
So I should train the category "other" with some documents...but what documents?
I should identify them first...that's a bit of a "chicken and egg" problem
Maybe I should do this way:
-each day X new documents arrive to be classified
-I find 10-11 docs with a low word freq in respect to the training set(but what is a "low" value?) and train them as other
-classify everything with the updated classifier
-----Messaggio originale-----
Da: Ted Dunning [mailto:ted.dunning@gmail.com]
Inviato: mercoledì 13 aprile 2011 19.29
A: user@mahout.apache.org
Cc: Claudia Grieco
Oggetto: Re: Identify "less similar" documents
On Wed, Apr 13, 2011 at 8:56 AM, Claudia Grieco <gr...@crmpa.unisa.it>wrote:
> Thanks for the help :)
> > Why not just train with those documents and put a category tag of "other"
> on
> >them and run normal categorization? If you can distinguish these
> documents
> >by word frequencies, then this should do the trick.
> I don't know if this will help
>
Only an experiment will tell you.
> 1)I'm still not sure where to put the threshold (if a document has word
> frequency less than X...how to choose X?)
>
The classifier should handle that for you for the most part. Again,
experimentation is the way to go here. My first cut would be to assign to
the category with the highest score, possibly including the other category.
> 2)The classifier is built incrementally: a document who would be classified
> as "other" today may be classified as "new category the user has just added"
> tomorrow. New docs in the training set and new categories are added from
> time to time.
>
That is pretty easy. Just retrain with the new category assignments.
Re: Identify "less similar" documents
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Apr 13, 2011 at 8:56 AM, Claudia Grieco <gr...@crmpa.unisa.it>wrote:
> Thanks for the help :)
> > Why not just train with those documents and put a category tag of "other"
> on
> >them and run normal categorization? If you can distinguish these
> documents
> >by word frequencies, then this should do the trick.
> I don't know if this will help
>
Only an experiment will tell you.
> 1)I'm still not sure where to put the threshold (if a document has word
> frequency less than X...how to choose X?)
>
The classifier should handle that for you for the most part. Again,
experimentation is the way to go here. My first cut would be to assign to
the category with the highest score, possibly including the other category.
> 2)The classifier is built incrementally: a document who would be classified
> as "other" today may be classified as "new category the user has just added"
> tomorrow. New docs in the training set and new categories are added from
> time to time.
>
That is pretty easy. Just retrain with the new category assignments.
R: Identify "less similar" documents
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
Thanks for the help :)
> Why not just train with those documents and put a category tag of "other" on
>them and run normal categorization? If you can distinguish these documents
>by word frequencies, then this should do the trick.
I don't know if this will help
1)I'm still not sure where to put the threshold (if a document has word frequency less than X...how to choose X?)
2)The classifier is built incrementally: a document who would be classified as "other" today may be classified as "new category the user has just added" tomorrow. New docs in the training set and new categories are added from time to time.
-----Messaggio originale-----
Da: Ted Dunning [mailto:ted.dunning@gmail.com]
Inviato: mercoledì 13 aprile 2011 17.34
A: user@mahout.apache.org
Cc: Claudia Grieco
Oggetto: Re: Identify "less similar" documents
I think that what you are doing is inventing an "other" category and
building a classifier for that category.
Why not just train with those documents and put a category tag of "other" on
them and run normal categorization? If you can distinguish these documents
by word frequencies, then this should do the trick.
On Wed, Apr 13, 2011 at 7:49 AM, Claudia Grieco <gr...@crmpa.unisa.it>wrote:
> Let's see if this approach makes sense:
> I have the documents to classify on a Lucene index (Index A) and the
> training set in another Lucene index (Index B).
> With a VectorMapper I map Term-Frequency Vectors of Index A to
> Term-Frequency Vectors of Index B. In this way the transformed vectors have
> only the frequency of the terms of the training set.
> By computing vector.zSum() I should get the frequency of the terms in the
> training set for the document, right?
> I compute vector.zSum() for all the docs to classify and exclude from the
> classification the ones who have a sum value of less than 10% the max
> vector.zSum()=>they mostly contain words never seen before and could be
> classified wrongly.
>
> What do you think?
>
> -----Messaggio originale-----
> Da: Claudia Grieco [mailto:grieco@crmpa.unisa.it]
> Inviato: mercoledì 13 aprile 2011 11.12
> A: user@mahout.apache.org
> Oggetto: Identify "less similar" documents
>
> Hi guys,
>
> I'm using SGD to classify a set of documents but I have a problem: there
> are
> some documents that are not related to any of the categories and I want to
> be able to identify them and exclude them from the classification. My idea
> is to read the documents of the training set (that are currently in a
> Lucene
> index) and identify the docs that have less terms in common with them. Any
> idea on how to do it?
>
> Thanks a lot
>
> Claudia
>
>
>
Re: Identify "less similar" documents
Posted by Ted Dunning <te...@gmail.com>.
I think that our estimation of whether this would work differs a bit. In
the very high dimensional space that we are working in, proximities can be a
bit surprising.
For one thing, the bias term provides a mechanism so that a logistic
regression can attribute score to an other category. This allows the
algorithm to implement something much like a score threshold.
For another, since the scores that come out of the SGD logistic regression
model always sum to 1, it is important to have an other category so that the
model can even express "low general relevance".
But, in this case as in many others, I would recommend you try it out.
On Wed, Apr 13, 2011 at 3:41 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> I guess the definition for the 'other' category is a 'low relevance
> for everything yet trained' but not 'high relevance to some category
> 'other' .
>
> As such, i think it is implied by definition that training for that
> stuff is not possible, but perhaps some cut-off threshold on the
> regressed posterior for all categories would help. But that's a
> surgery on the learner itself, i can't recollect if it is exposed by
> learner api?
>
Re: Identify "less similar" documents
Posted by Daniel McEnnis <dm...@gmail.com>.
The official solution is to assign outliers in the training set to
other. These are defined as high mean distance to other points. A
hack to get this to work would be to perform a knn-like distance
comparison with all trained sets and classify as other anything that
exceeds the threshold distance - a variation of the same technique and
already mentioned.
Daniel.
On Wed, Apr 13, 2011 at 6:41 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> I suspect but of the problem might be creating the training set for
> the 'other' since the documents are distinctly 'different' from
> anything else, including from each other.
> I guess the definition for the 'other' category is a 'low relevance
> for everything yet trained' but not 'high relevance to some category
> 'other' .
>
> As such, i think it is implied by definition that training for that
> stuff is not possible, but perhaps some cut-off threshold on the
> regressed posterior for all categories would help. But that's a
> surgery on the learner itself, i can't recollect if it is exposed by
> learner api?
>
>
> On Wed, Apr 13, 2011 at 8:34 AM, Ted Dunning <te...@gmail.com> wrote:
>> I think that what you are doing is inventing an "other" category and
>> building a classifier for that category.
>>
>> Why not just train with those documents and put a category tag of "other" on
>> them and run normal categorization? If you can distinguish these documents
>> by word frequencies, then this should do the trick.
>
Re: Identify "less similar" documents
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
I suspect but of the problem might be creating the training set for
the 'other' since the documents are distinctly 'different' from
anything else, including from each other.
I guess the definition for the 'other' category is a 'low relevance
for everything yet trained' but not 'high relevance to some category
'other' .
As such, i think it is implied by definition that training for that
stuff is not possible, but perhaps some cut-off threshold on the
regressed posterior for all categories would help. But that's a
surgery on the learner itself, i can't recollect if it is exposed by
learner api?
On Wed, Apr 13, 2011 at 8:34 AM, Ted Dunning <te...@gmail.com> wrote:
> I think that what you are doing is inventing an "other" category and
> building a classifier for that category.
>
> Why not just train with those documents and put a category tag of "other" on
> them and run normal categorization? If you can distinguish these documents
> by word frequencies, then this should do the trick.
Re: Identify "less similar" documents
Posted by Ted Dunning <te...@gmail.com>.
I think that what you are doing is inventing an "other" category and
building a classifier for that category.
Why not just train with those documents and put a category tag of "other" on
them and run normal categorization? If you can distinguish these documents
by word frequencies, then this should do the trick.
On Wed, Apr 13, 2011 at 7:49 AM, Claudia Grieco <gr...@crmpa.unisa.it>wrote:
> Let's see if this approach makes sense:
> I have the documents to classify on a Lucene index (Index A) and the
> training set in another Lucene index (Index B).
> With a VectorMapper I map Term-Frequency Vectors of Index A to
> Term-Frequency Vectors of Index B. In this way the transformed vectors have
> only the frequency of the terms of the training set.
> By computing vector.zSum() I should get the frequency of the terms in the
> training set for the document, right?
> I compute vector.zSum() for all the docs to classify and exclude from the
> classification the ones who have a sum value of less than 10% the max
> vector.zSum()=>they mostly contain words never seen before and could be
> classified wrongly.
>
> What do you think?
>
> -----Messaggio originale-----
> Da: Claudia Grieco [mailto:grieco@crmpa.unisa.it]
> Inviato: mercoledì 13 aprile 2011 11.12
> A: user@mahout.apache.org
> Oggetto: Identify "less similar" documents
>
> Hi guys,
>
> I'm using SGD to classify a set of documents but I have a problem: there
> are
> some documents that are not related to any of the categories and I want to
> be able to identify them and exclude them from the classification. My idea
> is to read the documents of the training set (that are currently in a
> Lucene
> index) and identify the docs that have less terms in common with them. Any
> idea on how to do it?
>
> Thanks a lot
>
> Claudia
>
>
>
R: Identify "less similar" documents
Posted by Claudia Grieco <gr...@crmpa.unisa.it>.
Let's see if this approach makes sense:
I have the documents to classify on a Lucene index (Index A) and the
training set in another Lucene index (Index B).
With a VectorMapper I map Term-Frequency Vectors of Index A to
Term-Frequency Vectors of Index B. In this way the transformed vectors have
only the frequency of the terms of the training set.
By computing vector.zSum() I should get the frequency of the terms in the
training set for the document, right?
I compute vector.zSum() for all the docs to classify and exclude from the
classification the ones who have a sum value of less than 10% the max
vector.zSum()=>they mostly contain words never seen before and could be
classified wrongly.
What do you think?
-----Messaggio originale-----
Da: Claudia Grieco [mailto:grieco@crmpa.unisa.it]
Inviato: mercoledì 13 aprile 2011 11.12
A: user@mahout.apache.org
Oggetto: Identify "less similar" documents
Hi guys,
I'm using SGD to classify a set of documents but I have a problem: there are
some documents that are not related to any of the categories and I want to
be able to identify them and exclude them from the classification. My idea
is to read the documents of the training set (that are currently in a Lucene
index) and identify the docs that have less terms in common with them. Any
idea on how to do it?
Thanks a lot
Claudia