You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Konstantyn Smirnov <in...@yahoo.com> on 2011/07/25 11:07:34 UTC
Reusing a CachingWrapperFilter
Hi all!
are there any limitations or implications on reusing a CWF?
In my app I'm doing the following:
Filter filter = new BooleanFilter(...)
// initialized with a couple of Term-, Range-, Boolean- and PrefixFilter
CachingWrapperFilter cwf = new CachingWrapperFilter( filter )
searcher.search( query, cwf ... ) // << here, the filter's docIdSet gets
cached, right?
// now I need to iterate over a list of categories and calculate the number
of goods in each of them
for( cat in categories ){
Filter catFilter = new BooleanFilter()
TermsFilter tf = new TermsFilter()
tf.addTerm new Term( 'category, cat )
catFilter.add new FilterClause( cwf, ... )
catFilter.add new FilterClause( tf, ... )
long freq = catFilter.getDocIdSet( indexReader ).cardinality()
//do stuff with freq
}
Is this a correct use? Shall the filter with a pre-cached bitSet be combined
with the new one inside the BooleanFilter like that?
Or maybe should I use searcher.search( query, filter, ... ).totalCount
instead of catFilter.getDocIdSet( indexReader ).cardinality()?
TIA
Konstantyn
--
View this message in context: http://lucene.472066.n3.nabble.com/Reusing-a-CachingWrapperFilter-tp3196928p3196928.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Reusing a CachingWrapperFilter
Posted by Ian Lea <ia...@gmail.com>.
My guess is that it would depend on the number of docs in the index,
how many match, how complex your query is, probably other things.
If q is complex the MatchAllDocs variant might be well be quicker but
if q is simple I wouldn't like to guess. I'm not sure there is any
point in passing the filter to the ConstantScore variation.
An alternative might be to use the filter directly via it's DocIdSet.
Whatever you do, searching is generally very quick. Reading doc
values, formatting, whatever, is often what takes the time.
--
Ian.
On Fri, Jul 29, 2011 at 3:08 PM, Konstantyn Smirnov <in...@yahoo.com> wrote:
> If I define a query and filter like this:
>
> Query q = new BooleanQuery()
> // populating q
>
> Filter filter = new CachingWrapperFilter( new QueryWrapperFilter( q ) )
>
> given that I don't need score and I do need a cached filter to reuse it
> immediately for other calculations, which way of searching would be the most
> performing:
>
> searcher.search( new MatchAllDocsQuery(), filter, max )
>
> or
>
> searcher.search( new ConstantScoreQuery( q ), filter, max )
>
> or
>
> something completely different?
>
> TIA
>
> --
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Reusing a CachingWrapperFilter
Posted by Konstantyn Smirnov <in...@yahoo.com>.
If I define a query and filter like this:
Query q = new BooleanQuery()
// populating q
Filter filter = new CachingWrapperFilter( new QueryWrapperFilter( q ) )
given that I don't need score and I do need a cached filter to reuse it
immediately for other calculations, which way of searching would be the most
performing:
searcher.search( new MatchAllDocsQuery(), filter, max )
or
searcher.search( new ConstantScoreQuery( q ), filter, max )
or
something completely different?
TIA
--
View this message in context: http://lucene.472066.n3.nabble.com/Reusing-a-CachingWrapperFilter-tp3196928p3209688.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Reusing a CachingWrapperFilter
Posted by Uwe Schindler <uw...@thetaphi.de>.
This collector uses no resources, I would recreate it inside the loop. Its
just a thin class in young gen heap (like a autoboxed number). If you really
want to reuse, you have to log the last count and simply measure the
difference as you cannot reset.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Konstantyn Smirnov [mailto:injecteer@yahoo.com]
> Sent: Monday, July 25, 2011 4:16 PM
> To: java-user@lucene.apache.org
> Subject: RE: Reusing a CachingWrapperFilter
>
>
> Uwe Schindler wrote:
> >
> > To just count the results use TotalHitCountCollector (since Lucene Core
> > 3.1)
> > with IndexSaercher.search().
> >
> ok, thanks for that!
>
> so the code should look like:
>
> CachingWrapperFilter cwf = new CachingWrapperFilter( filter )
>
> searcher.search( query, cwf ... ) // search 1st time, to populate cache
>
> for( cat in categories ){
> TotalHitCountCollector thcc = new TotalHitCountCollector()
> searcher.search new TermQuery( new Term( 'category', cat ) ), cwf, thcc
> long freq = thcc.totalHits
> }
>
> TotalHitCountCollector doesn't calculate the score, does it?
> Can I reuse the thcc instance in the loop?
>
> TIA
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Reusing-
> a-CachingWrapperFilter-tp3196928p3197593.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Reusing a CachingWrapperFilter
Posted by Konstantyn Smirnov <in...@yahoo.com>.
Uwe Schindler wrote:
>
> To just count the results use TotalHitCountCollector (since Lucene Core
> 3.1)
> with IndexSaercher.search().
>
ok, thanks for that!
so the code should look like:
CachingWrapperFilter cwf = new CachingWrapperFilter( filter )
searcher.search( query, cwf ... ) // search 1st time, to populate cache
for( cat in categories ){
TotalHitCountCollector thcc = new TotalHitCountCollector()
searcher.search new TermQuery( new Term( 'category', cat ) ), cwf, thcc
long freq = thcc.totalHits
}
TotalHitCountCollector doesn't calculate the score, does it?
Can I reuse the thcc instance in the loop?
TIA
--
View this message in context: http://lucene.472066.n3.nabble.com/Reusing-a-CachingWrapperFilter-tp3196928p3197593.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Reusing a CachingWrapperFilter
Posted by Uwe Schindler <uw...@thetaphi.de>.
The problem with your code is , that the IndexSearcher.search() method works
per segment, but in your cardinality code you are using top-level-readers,
so it will cache two times (once per segments and once for the top-level
MultiReader/DirectoryReader). Also not all DocIdSet implementations extend
OpenBitSet, so cardinality is not always available and for some queries you
might get a ClassCastException.
To just count the results use TotalHitCountCollector (since Lucene Core 3.1)
with IndexSaercher.search().
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Konstantyn Smirnov [mailto:injecteer@yahoo.com]
> Sent: Monday, July 25, 2011 11:08 AM
> To: java-user@lucene.apache.org
> Subject: Reusing a CachingWrapperFilter
>
> Hi all!
>
> are there any limitations or implications on reusing a CWF?
>
> In my app I'm doing the following:
>
> Filter filter = new BooleanFilter(...)
> // initialized with a couple of Term-, Range-, Boolean- and PrefixFilter
>
> CachingWrapperFilter cwf = new CachingWrapperFilter( filter )
>
> searcher.search( query, cwf ... ) // << here, the filter's docIdSet gets
cached,
> right?
>
> // now I need to iterate over a list of categories and calculate the
number of
> goods in each of them
>
> for( cat in categories ){
> Filter catFilter = new BooleanFilter()
> TermsFilter tf = new TermsFilter()
> tf.addTerm new Term( 'category, cat )
>
> catFilter.add new FilterClause( cwf, ... )
> catFilter.add new FilterClause( tf, ... )
>
> long freq = catFilter.getDocIdSet( indexReader ).cardinality()
>
> //do stuff with freq
> }
>
> Is this a correct use? Shall the filter with a pre-cached bitSet be
combined
> with the new one inside the BooleanFilter like that?
> Or maybe should I use searcher.search( query, filter, ... ).totalCount
instead
> of catFilter.getDocIdSet( indexReader ).cardinality()?
>
> TIA
>
> Konstantyn
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Reusing-
> a-CachingWrapperFilter-tp3196928p3196928.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org