You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by ya...@bloglines.com on 2004/07/11 07:43:10 UTC
Pool of IndexReaders or Pool of Searchers?
Hi,
I have multiple threads reading an index. Should they all be using
the same IndexReader and using a pool of IndexSearchers? Or should they be
using a pool of IndexReaders?
Basically, one reader or many?
Thanks.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Pool of IndexReaders or Pool of Searchers?
Posted by Doug Cutting <cu...@apache.org>.
Whether this will make a difference depends on the size of the index.
If your index is relatively small, then this patch will help more. If
your index is large, it will help less.
Aviran wrote:
> Try to compile this code changes into lucene
> http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg06116.html
> In our tests it improved performance by ~100%
>
> Aviran
>
> -----Original Message-----
> From: Anson Lau [mailto:alau@fulfil-net.com]
> Sent: Tuesday, July 13, 2004 3:40 AM
> To: 'Lucene Users List'
> Subject: RE: Pool of IndexReaders or Pool of Searchers?
>
>
> Don't have a formal report but I can give you a bit more details:
>
> What are we testing: a search app powered by lucene, search app is a web app
> built on Struts.
> Index: 1.8 million database records
> Hardware: Dual P4 2.8 HT, 4G Ram, Raid 5 SCIC HDs.
> Directory type: FSDirectory
>
> Load test app: A load test app which has 15 threads, each firing 1 search
> request (http request) per second to the search app.
>
> All the http requests are search request, but keep in mind all the overhead
> of jsp, struts, etc.
>
> We thought especially with multiple CPU systems using a small pool of index
> searcher may improve concurrency.
>
> Under lucene 1.4:
> Using 1 static index searcher: 12 request per second.
> Using a pool of 4 index search: same 12 request per second
>
> Under lucene 1.3:
> I can't remember the exact numbers, but pooling index searcher did make a
> noticeable difference.
>
>
> Hope that's useful.
>
> Anson
>
>
>
> -----Original Message-----
> From: Vince Taluskie [mailto:vince@taluskie.com]
> Sent: Tuesday, July 13, 2004 3:50 PM
> To: Lucene Users List
> Subject: Re: Pool of IndexReaders or Pool of Searchers?
>
> Can you supply details on the config tested?
>
> Vince
>
> Anson Lau wrote:
>
>
>>Hi,
>>
>>When I did some load testing on a lucene powered search app, using a
>>pool of index searchers doesn't give me any more search per second than
>>just using a singleton index searcher.
>>
>>Anson
>>
>>
>>Quoting yahootintin.1247688@bloglines.com:
>>
>>
>>
>>
>>>Hi,
>>>
>>>I have multiple threads reading an index. Should they all be using
>>>the same IndexReader and using a pool of IndexSearchers? Or
>>>should they be
>>>using a pool of IndexReaders?
>>>
>>>Basically, one reader or many?
>>>
>>>Thanks.
>>>
>>>
>>>
>>>
>>
>>---------------------------------------------------------------------
>>
>>
>>
>>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>>For additional commands, e-mail:
>>>lucene-user-help@jakarta.apache.org
>>>
>>>
>>>
>>
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Pool of IndexReaders or Pool of Searchers?
Posted by Aviran <am...@infosciences.com>.
Try to compile this code changes into lucene
http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg06116.html
In our tests it improved performance by ~100%
Aviran
-----Original Message-----
From: Anson Lau [mailto:alau@fulfil-net.com]
Sent: Tuesday, July 13, 2004 3:40 AM
To: 'Lucene Users List'
Subject: RE: Pool of IndexReaders or Pool of Searchers?
Don't have a formal report but I can give you a bit more details:
What are we testing: a search app powered by lucene, search app is a web app
built on Struts.
Index: 1.8 million database records
Hardware: Dual P4 2.8 HT, 4G Ram, Raid 5 SCIC HDs.
Directory type: FSDirectory
Load test app: A load test app which has 15 threads, each firing 1 search
request (http request) per second to the search app.
All the http requests are search request, but keep in mind all the overhead
of jsp, struts, etc.
We thought especially with multiple CPU systems using a small pool of index
searcher may improve concurrency.
Under lucene 1.4:
Using 1 static index searcher: 12 request per second.
Using a pool of 4 index search: same 12 request per second
Under lucene 1.3:
I can't remember the exact numbers, but pooling index searcher did make a
noticeable difference.
Hope that's useful.
Anson
-----Original Message-----
From: Vince Taluskie [mailto:vince@taluskie.com]
Sent: Tuesday, July 13, 2004 3:50 PM
To: Lucene Users List
Subject: Re: Pool of IndexReaders or Pool of Searchers?
Can you supply details on the config tested?
Vince
Anson Lau wrote:
>Hi,
>
>When I did some load testing on a lucene powered search app, using a
>pool of index searchers doesn't give me any more search per second than
>just using a singleton index searcher.
>
>Anson
>
>
>Quoting yahootintin.1247688@bloglines.com:
>
>
>
>>Hi,
>>
>>I have multiple threads reading an index. Should they all be using
>>the same IndexReader and using a pool of IndexSearchers? Or
>>should they be
>>using a pool of IndexReaders?
>>
>>Basically, one reader or many?
>>
>>Thanks.
>>
>>
>>
>>
>---------------------------------------------------------------------
>
>
>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail:
>>lucene-user-help@jakarta.apache.org
>>
>>
>>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
RE: Pool of IndexReaders or Pool of Searchers?
Posted by Anson Lau <al...@fulfil-net.com>.
Don't have a formal report but I can give you a bit more details:
What are we testing: a search app powered by lucene, search app is a web app
built on Struts.
Index: 1.8 million database records
Hardware: Dual P4 2.8 HT, 4G Ram, Raid 5 SCIC HDs.
Directory type: FSDirectory
Load test app: A load test app which has 15 threads, each firing 1 search
request (http request) per second to the search app.
All the http requests are search request, but keep in mind all the overhead
of jsp, struts, etc.
We thought especially with multiple CPU systems using a small pool of index
searcher may improve concurrency.
Under lucene 1.4:
Using 1 static index searcher: 12 request per second.
Using a pool of 4 index search: same 12 request per second
Under lucene 1.3:
I can't remember the exact numbers, but pooling index searcher did make a
noticeable difference.
Hope that's useful.
Anson
-----Original Message-----
From: Vince Taluskie [mailto:vince@taluskie.com]
Sent: Tuesday, July 13, 2004 3:50 PM
To: Lucene Users List
Subject: Re: Pool of IndexReaders or Pool of Searchers?
Can you supply details on the config tested?
Vince
Anson Lau wrote:
>Hi,
>
>When I did some load testing on a lucene powered search app, using a
>pool of index searchers doesn't give me any more search per second
>than just using a singleton index searcher.
>
>Anson
>
>
>Quoting yahootintin.1247688@bloglines.com:
>
>
>
>>Hi,
>>
>>I have multiple threads reading an index. Should they all be
>>using
>>the same IndexReader and using a pool of IndexSearchers? Or
>>should they be
>>using a pool of IndexReaders?
>>
>>Basically, one reader or many?
>>
>>Thanks.
>>
>>
>>
>>
>---------------------------------------------------------------------
>
>
>>To unsubscribe, e-mail:
>>lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail:
>>lucene-user-help@jakarta.apache.org
>>
>>
>>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Pool of IndexReaders or Pool of Searchers?
Posted by Vince Taluskie <vi...@taluskie.com>.
Can you supply details on the config tested?
Vince
Anson Lau wrote:
>Hi,
>
>When I did some load testing on a lucene powered search app, using a
>pool of index searchers doesn't give me any more search per second
>than just using a singleton index searcher.
>
>Anson
>
>
>Quoting yahootintin.1247688@bloglines.com:
>
>
>
>>Hi,
>>
>>I have multiple threads reading an index. Should they all be
>>using
>>the same IndexReader and using a pool of IndexSearchers? Or
>>should they be
>>using a pool of IndexReaders?
>>
>>Basically, one reader or many?
>>
>>Thanks.
>>
>>
>>
>>
>---------------------------------------------------------------------
>
>
>>To unsubscribe, e-mail:
>>lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail:
>>lucene-user-help@jakarta.apache.org
>>
>>
>>
>
>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Pool of IndexReaders or Pool of Searchers?
Posted by Anson Lau <al...@fulfil-net.com>.
Hi,
When I did some load testing on a lucene powered search app, using a
pool of index searchers doesn't give me any more search per second
than just using a singleton index searcher.
Anson
Quoting yahootintin.1247688@bloglines.com:
> Hi,
>
> I have multiple threads reading an index. Should they all be
> using
> the same IndexReader and using a pool of IndexSearchers? Or
> should they be
> using a pool of IndexReaders?
>
> Basically, one reader or many?
>
> Thanks.
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Pool of IndexReaders or Pool of Searchers?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
1 should be sufficient, as IR/IS are thread-safe, I believe.
Otis
--- yahootintin.1247688@bloglines.com wrote:
> Hi,
>
> I have multiple threads reading an index. Should they all be using
> the same IndexReader and using a pool of IndexSearchers? Or should
> they be
> using a pool of IndexReaders?
>
> Basically, one reader or many?
>
> Thanks.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org