[Lucene-java Wiki] Update of "ImproveSearchingSpeed" by MikeMcCandless

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-java Wiki" for change notification.

The following page has been changed by MikeMcCandless:
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed

New page:
= How to make searching faster =

Here are some things to try to speed up the seaching speed of your Lucene application.  Please see ImproveIndexingSpeed for how to speed up indexing.

 * '''Be sure you really need to speed things up.'''

 Many of the ideas here are simple to try, but others will necessarily add some complexity to your application.  So be sure your searching speed is indeed too slow and the slowness is indeed within Lucene.

 * '''Make sure you are using the latest version of Lucene.'''

 * '''Use a local filesystem.'''

 Remote filesystems are typically quite a bit slower for searching.  If the index must be remote, try to mount the remote filesystem as a "readonly" mount.  In some cases this could improve performance.

 * '''Get faster hardware, especially a faster IO system.'''

 * '''Add RAM to your hardware and/or increase the heap size for the JVM.'''

 For a large index, searching can use alot of RAM.  If you don't have enough RAM or your JVM is not running with a large enough HEAP size then the JVM can hit swapping and thrashing at which point everything will run slowly.

 * '''Use one instance of [http://lucene.apache.org/java/docs/api/org/apache/lucene/search/IndexSearcher.html IndexSearcher].'''

 Share a single [http://lucene.apache.org/java/docs/api/org/apache/lucene/search/IndexSearcher.html IndexSearcher] across queries and across threads in your application.

 * '''Re-open the [http://lucene.apache.org/java/docs/api/org/apache/lucene/search/IndexSearcher.html IndexSearcher] only when necessary.'''

 You must re-open the [http://lucene.apache.org/java/docs/api/org/apache/lucene/search/IndexSearcher.html IndexSearcher] in order to see make newly committed changes visible to searching.  However, doing this is costly because many caches are lost when opening a new searcher. Consider using a so called [http://wiki.apache.org/solr/SolrCaching warming] technique which allows the searcher to warm up its caches before the first query hits.

 * '''Run [http://lucene.apache.org/java/docs/api/org/apache/lucene/index/IndexWriter.html#optimize() optimize] on your index before searching.'''

 An optimized index has only 1 segment to search which can be much faster than the many segments that will normally be created, especially for a large index.  If your application does not often update the index then it pays to build the index, optimize it, and use the optimized one for searching.  If instead you are frequently updating the index and then refreshing searchers, then optimizing will likely be too costly and you should decrease mergeFactor instead.

 * '''Decrease [http://lucene.apache.org/java/docs/api/org/apache/lucene/index/IndexWriter.html#setMergeFactor(int) mergeFactor].'''

 Smaller mergeFactors mean fewer segments and searching will be faster.  However, this will slow down indexing speed, so you should test values to strike an appropriate balance for your application.

 * '''Limit usage of stored fields and term vectors.'''

 Retrieving these from the index is quite costly.  Typically you should only retrieve these for the current "page" the user will see, not for all documents in the full result set.  For each document retrieved, Lucene must seek to a different location in various files.  Try sorting the documents you need to retrieve by docID order first.

 * '''Use [http://lucene.apache.org/java/docs/api/org/apache/lucene/document/FieldSelector.html FieldSelector] to carefully pick which fields are loaded, and how they are loaded, when you retrieve a document.'''