You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Sebastin <se...@gmail.com> on 2008/06/18 16:17:22 UTC
Lucene Search Very Slow
Hi All,
I need to fetch approximately 225 GB of Index Store records in a web page
.the total time to fetch the record and display to the user takes 10
minutes.is it possible to reduce the time to milliseconds
sample code snippet:
IndexReader[] readArray =
{ indexIR1, indexIR2, indexIR3, indexIR4, indexIR5, indexIR6,
indexIR7,
indexIR8, indexIR9, indexIR10, indexIR11,indexIR12, indexIR13,
indexIR14,indexIR15, indexIR16, indexIR17, indexIR18, indexIR19, indexIR20,
indexIR21,
indexIR22, indexIR23, indexIR24, indexIR25,indexIR26, indexIR27,
indexIR28,indexIR29, indexIR29, indexIR30, indexIR31, indexIR32, indexIR33,
indexIR34,
indexIR35, indexIR36, indexIR37, indexIR38,indexIR39, indexIR40,
indexIR41,indexIR42, indexIR43, indexIR44, indexIR45, indexIR46, indexIR47,
indexIR48,
indexIR49, indexIR50, indexIR51, indexIR52,indexIR53, indexIR54,
indexIR55,indexIR56, indexIR57, indexIR58, indexIR59, indexIR60, indexIR61,
indexIR62,
indexIR63, indexIR64, indexIR65, indexIR66,indexIR67, indexIR68,
indexIR69,indexIR69, indexIR70, indexIR71, indexIR72, indexIR73, indexIR74,
indexIR75,
indexIR76, indexIR77, indexIR78, indexIR79,indexIR80, indexIR81,
indexIR82,indexIR83, indexIR84, indexIR85, indexIR86, indexIR87, indexIR88,
indexIR89,
indexIR90, indexIR91, indexIR92, indexIR93,indexIR94, indexIR95,
indexIR96,indexIR97, indexIR98, indexIR99, indexIR100, indexIR101,
indexIR102, indexIR103,
indexIR104, indexIR105, indexIR106, indexIR107,indexIR108,
indexIR109, indexIR110,indexIR111, indexIR112, indexIR113, indexIR114,
indexIR115, indexIR116, indexIR117,
indexIR118, indexIR119, indexIR120, indexIR121,indexIR122,
indexIR123, indexIR124,indexIR125, indexIR126, indexIR127, indexIR128,
indexIR129, indexIR130, indexIR131,
indexIR132, indexIR133, indexIR134, indexIR135,indexIR136,
indexIR137, indexIR138,indexIR139, indexIR140, indexIR141, indexIR142,
indexIR143, indexIR144, indexIR145,
indexIR146, indexIR147, indexIR148, indexIR149,indexIR150,
indexIR151, indexIR152, indexIR153, indexIR154};
*/
//merged reader
// IndexReader mergedReader = new MultiReader(readArray);
// IndexSearcher is = new IndexSearcher(mergedReader);
// q = search.integerToCompressedString(q);
BooleanQuery.setMaxClauseCount(100000);
QueryParser parser =
new QueryParser("indexRecords", new StandardAnalyzer());
Query query = parser.parse(searchQuery);
Hits hits = is.search(query);
);
for every search i open 154 folders of size 2 GB .
--
View this message in context: http://www.nabble.com/Lucene-Search-Very-Slow-tp17983687p17983687.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene Search Very Slow
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Wed, 2008-06-18 at 07:17 -0700, Sebastin wrote:
> I need to fetch approximately 225 GB of Index Store records in a web page
> .the total time to fetch the record and display to the user takes 10
> minutes.is it possible to reduce the time to milliseconds
Depends on your indexes and your queries, but it shouldn't be
impossible: Keep the searcher open between searches instead of starting
from scratch every time and do warm up searches before the real ones.
How many hits does a normal search return and what do you do with them?
You might also consider the viability of that many indexes, as there is
an upper limit to the number of available file handles. I guess it's
system-specific, but on my machine it's around 1000 and since an active
Lucene index uses more than one file handle, you might well hit the
ceiling not far from your current number of searchers.
All else failing, buy 4 * 64GB quality solid state drives, put them in
RAID 0 and be happy. Lucene still needs to warm up on those, but nowhere
near as much as conventional harddrives, and they are not as reliant on
free RAM for disk-cache.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org