You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Karsten Konrad <Ka...@xtramind.com> on 2003/03/31 18:30:25 UTC
Proposal: Statistical Stopword elimination
Hi,
I am experimenting with long queries (parts of documents as search query),
and I would like to filter all terms with high document frequencies when
searching. I.e., a kind of statistical, language independent stop word
elimination while searching.
For this, I have introduced a frequency limit factor into
Similarity and test for excessively high document frequencies
in the TermQuery. The code looks somewhat like this:
>>
public Scorer scorer(IndexReader reader) throws IOException {
TermDocs termDocs = reader.termDocs(term);
if (termDocs == null)
return null;
float limit = searcher.getSimilarity().getLimitFactor();
int docFreq = searcher.docFreq(term);
int max = searcher.maxDoc();
if (docFreq >= (max+1)*limit)
return null;
return new TermScorer(this, termDocs, searcher.getSimilarity(),
reader.norms(term.field()));
}
>>
A limit factor of 0.2 will then remove all terms from the search that appear
in more than (approximately) 20% of the documents. For long queries, the search
time is reduced - about factor 2 even on shorter text queries. A factor of 1.0 or
higher will give you identical results to the original version. Also,
highlighting often looks better as only less frequent terms are highlighted.
While the terms removed stay in the index and therefore can still be searched,
we can speed up more complex searches by this method.
My questions:
(1) Is there some more elegant way of doing this? E.g., access to the docFreq is
done again in the TermScorer and I would like to remove this redundancy.
(2) Is this a worthwhile contribution to Lucene's features in your opinion?
Comments appreciated,
--
Dr.-Ing. Karsten Konrad
Head of Information Agent Engineering
XtraMind Technologies GmbH
Stuhlsatzenhausweg 3
D-66123 Saarbrücken
Phone: +49 (681) 3025113
Fax: +49 (681) 3025109
konrad@xtramind.com
www.xtramind.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Proposal: Statistical Stopword elimination
Posted by Doug Cutting <cu...@lucene.com>.
Karsten Konrad wrote:
> For this, I have introduced a frequency limit factor into
> Similarity and test for excessively high document frequencies
> in the TermQuery.
>
> My questions:
>
> (1) Is there some more elegant way of doing this?
I think you could do this more simply by creating a subclass of
TermQuery and overriding createWeight, with something like:
protected Weight createWeight(Searcher searcher) {
float maxDoc = searcher.maxDoc();
float ratio = searcher.docFreq(getTerm()) / maxDoc;
float threshold =
(ThresholdSimilarity)getSimilarity()).getThreshold());
if (ratio >= threshold)
return super.createWeight(searcher);
else
return new NullWeight(); // a no-op weight implementation
}
You'd also need to define ThresholdSimilarity as a subclass of
Similarity or DefaultSimilarity that has a threshold, and define
NullWeight as a Weight implementation whose Scorer does nothing.
Note that, with a MultiSearcher, your implementation computed thresholds
independently for each index, whereas this computes them globally over
all indexes, which is probably what you want.
Note also that this is all done with public APIs and requires no changes
to the Lucene core.
> E.g., access to the docFreq is done again in the TermScorer
> and I would like to remove this redundancy.
I doubt that will substantially impact performance. If it does, it
would be easy to add a small cache into the IndexReader. However
someone tried this once and found that it didn't make much difference.
> (2) Is this a worthwhile contribution to Lucene's features in your opinion?
Please post the code. If folks use it, then it's worthwhile and we
should probably include it with Lucene. Ideally it should be simple to
do implement such things with the public APIs without having to build
more features into the core.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
RE: Proposal: Statistical Stopword elimination
Posted by Alex Murzaku <li...@lissus.com>.
I was also wondering about this... don't know well the internals of
Lucene though to give you any smart implementation feedback. In my
opinion, it would be a very useful addition. I would just add that, if
the frequent term is the only term in the query, it should not be
eliminated. I just tried Google and it behaves the same way. Very
frequent terms ARE indexed. They get removed only when they are part of
a query with more than one term.
--
Alex Murzaku
___________________________________________
alex(at)lissus.com http://www.lissus.com
-----Original Message-----
From: Karsten Konrad [mailto:Karsten.Konrad@xtramind.com]
Sent: Monday, March 31, 2003 11:30 AM
To: Lucene Developers List
Subject: Proposal: Statistical Stopword elimination
Hi,
I am experimenting with long queries (parts of documents as search
query), and I would like to filter all terms with high document
frequencies when searching. I.e., a kind of statistical, language
independent stop word
elimination while searching.
For this, I have introduced a frequency limit factor into Similarity and
test for excessively high document frequencies in the TermQuery. The
code looks somewhat like this:
>>
public Scorer scorer(IndexReader reader) throws IOException {
TermDocs termDocs = reader.termDocs(term);
if (termDocs == null)
return null;
float limit = searcher.getSimilarity().getLimitFactor();
int docFreq = searcher.docFreq(term);
int max = searcher.maxDoc();
if (docFreq >= (max+1)*limit)
return null;
return new TermScorer(this, termDocs, searcher.getSimilarity(),
reader.norms(term.field()));
}
>>
A limit factor of 0.2 will then remove all terms from the search that
appear in more than (approximately) 20% of the documents. For long
queries, the search time is reduced - about factor 2 even on shorter
text queries. A factor of 1.0 or
higher will give you identical results to the original version. Also,
highlighting often looks better as only less frequent terms are
highlighted.
While the terms removed stay in the index and therefore can still be
searched, we can speed up more complex searches by this method.
My questions:
(1) Is there some more elegant way of doing this? E.g., access to the
docFreq is done again in the TermScorer and I would like to remove this
redundancy.
(2) Is this a worthwhile contribution to Lucene's features in your
opinion?
Comments appreciated,
--
Dr.-Ing. Karsten Konrad
Head of Information Agent Engineering
XtraMind Technologies GmbH
Stuhlsatzenhausweg 3
D-66123 Saarbrücken
Phone: +49 (681) 3025113
Fax: +49 (681) 3025109
konrad@xtramind.com
www.xtramind.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org