You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Chad Hardin <ch...@topiatechnology.com> on 2006/08/07 22:22:09 UTC
More like this returning similarities that are too generic
hi all,
I'm new to lucene but I'm loving it! I'm writing a prototype that
links documents together based upon similarities. Obviously the
first thing I did was use MoreLikeThis. However, it seems to be
finding matches based upon words that are too common, in this case
the words "from" and "can" and seems to be missing matches using the
terms I would expect (in this case documents about "bikes").
I seems I need a more custom tailored Filter that only passes through
more less-common words. Does something like this already exist?
Thanks,
Chad
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: More like this returning similarities that are too generic
Posted by Chad Hardin <ch...@topiatechnology.com>.
You're soo right! I'm totally new to lucene (and text analyses,
searching etc), but now that you showed me I "get it". Thank you so
much for your reply.
Chad
On Aug 8, 2006, at 12:45 AM, Chris Hostetter wrote:
>
> I've never used MoreLikeThis myself, but based on how i know it works,
> your problem probably has more to do with the size of your test
> corpus and
> th frequency of the words in your docs then by the size of the docs
> themselves.
>
> : There's still the issue of the queries from MoreLikeThis not
> : returning results for terms I had expected ("bikes").
>
> A quick glance at the source for MoreLikeThis turns up these lines...
>
> /**
> * Ignore terms with less than this frequency in the source doc.
> * @see #getMinTermFreq
> * @see #setMinTermFreq
> */
> public static final int DEFAULT_MIN_TERM_FREQ = 2;
>
> /**
> * Ignore words which do not occur in at least this many docs.
> * @see #getMinDocFreq
> * @see #setMinDocFreq
> */
> public static final int DEFALT_MIN_DOC_FREQ = 5;
>
> ...which i'm guessing mean that unless a word appears in a doc at
> least
> twice, it's ignored for that doc, and unless a word appears in at
> least 5
> docs, it's ignored completely. that could easily explain your bike
> examples.
>
> : I then loaded some large (5K+) documents and I noticed that
> : MoreLikeThis's query started to return similar documents, but
> explain
> : () said they were similar because of words like "from" and "can"
> rather
> : than the text I expected to be used for similarity in the documents.
>
> Other then a stop words list, one other thing you might consider is
> the notion of a "maxDocFreq" option you could set to ignore words that
> appear in lots of documents -- or a maxDocFreqRatio that would take a
> percentage of the total number of docs ... it should be fairly
> straightforward to add.
>
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: More like this returning similarities that are too generic
Posted by Chris Hostetter <ho...@fucit.org>.
I've never used MoreLikeThis myself, but based on how i know it works,
your problem probably has more to do with the size of your test corpus and
th frequency of the words in your docs then by the size of the docs
themselves.
: There's still the issue of the queries from MoreLikeThis not
: returning results for terms I had expected ("bikes").
A quick glance at the source for MoreLikeThis turns up these lines...
/**
* Ignore terms with less than this frequency in the source doc.
* @see #getMinTermFreq
* @see #setMinTermFreq
*/
public static final int DEFAULT_MIN_TERM_FREQ = 2;
/**
* Ignore words which do not occur in at least this many docs.
* @see #getMinDocFreq
* @see #setMinDocFreq
*/
public static final int DEFALT_MIN_DOC_FREQ = 5;
...which i'm guessing mean that unless a word appears in a doc at least
twice, it's ignored for that doc, and unless a word appears in at least 5
docs, it's ignored completely. that could easily explain your bike
examples.
: I then loaded some large (5K+) documents and I noticed that
: MoreLikeThis's query started to return similar documents, but explain
: () said they were similar because of words like "from" and "can" rather
: than the text I expected to be used for similarity in the documents.
Other then a stop words list, one other thing you might consider is
the notion of a "maxDocFreq" option you could set to ignore words that
appear in lots of documents -- or a maxDocFreqRatio that would take a
percentage of the total number of docs ... it should be fairly
straightforward to add.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: More like this returning similarities that are too generic
Posted by Chad Hardin <ch...@topiatechnology.com>.
Thank you Erick, that was what I anticipated would be necessary.
There's still the issue of the queries from MoreLikeThis not
returning results for terms I had expected ("bikes").
For example, I have these four very short documents:
"bikes are a handy tool for getting from diffrent locations on the
earth, Ireally like bikes"
"Bikes can be really fast"
"bikes can be dangerous"
"I like bikes because they are fun"
I expected the query from MoreLikeThis to say that these short
documents are similar because they all have "Bikes", but it doesn't.
Is there some reason for this to be so? Perhaps because the
documents are so short?
I then loaded some large (5K+) documents and I noticed that
MoreLikeThis's query started to return similar documents, but explain
() said they were similar because of words like "from" and "can" rather
than the text I expected to be used for similarity in the documents.
Chad
On Aug 7, 2006, at 1:29 PM, Erick Erickson wrote:
> Well, I expect that defining "less common" is tricky and doesn't
> lend itself
> to a canned answer <G>. Would it work to create your own list of
> stop words
> (possibly very large) to use for indexing and/or searching? This would
> simply exclude the "less common" words (as you define them).
> StandardAnalyzer, for instance, can take a File of stop words in
> one of its
> constructors.......
>
> Erick
>
> On 8/7/06, Chad Hardin <ch...@topiatechnology.com> wrote:
>>
>> hi all,
>>
>> I'm new to lucene but I'm loving it! I'm writing a prototype that
>> links documents together based upon similarities. Obviously the
>> first thing I did was use MoreLikeThis. However, it seems to be
>> finding matches based upon words that are too common, in this case
>> the words "from" and "can" and seems to be missing matches using the
>> terms I would expect (in this case documents about "bikes").
>>
>> I seems I need a more custom tailored Filter that only passes through
>> more less-common words. Does something like this already exist?
>>
>> Thanks,
>>
>> Chad
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: More like this returning similarities that are too generic
Posted by Erick Erickson <er...@gmail.com>.
Well, I expect that defining "less common" is tricky and doesn't lend itself
to a canned answer <G>. Would it work to create your own list of stop words
(possibly very large) to use for indexing and/or searching? This would
simply exclude the "less common" words (as you define them).
StandardAnalyzer, for instance, can take a File of stop words in one of its
constructors.......
Erick
On 8/7/06, Chad Hardin <ch...@topiatechnology.com> wrote:
>
> hi all,
>
> I'm new to lucene but I'm loving it! I'm writing a prototype that
> links documents together based upon similarities. Obviously the
> first thing I did was use MoreLikeThis. However, it seems to be
> finding matches based upon words that are too common, in this case
> the words "from" and "can" and seems to be missing matches using the
> terms I would expect (in this case documents about "bikes").
>
> I seems I need a more custom tailored Filter that only passes through
> more less-common words. Does something like this already exist?
>
> Thanks,
>
> Chad
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>