You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jamie <ja...@mailarchiva.com> on 2014/06/02 08:51:59 UTC
search performance
Greetings
Despite following all the recommended optimizations (as described at
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed) , in some of
our installations, search performance has reached the point where is it
unacceptably slow. For instance, in one environment, the total index
size is 200GB, with 150 million documents indexed. With NRT enabled,
search speed is roughly 5 minutes on average. The server resources are:
2x6 Core Intel CPU, 128GB, 2 SSD for index and RAID 0, with Linux.
The only thing we haven't yet done, is to upgrade Lucene from 4.7.x to
4.8.x. Is this likely to make any noticeable difference in performance?
Clearly, longer term, we need to move to a distributed search model. We
thought to take advantage of the distributed search features offered in
Solr, however, our solution is very tightly integrated into Lucene
directly (since Solr didn't exist when we started out). Moving to Solr
now seems like a daunting prospect. We've also following the Katta
project with interest, but it doesn't appear support distributed
indexing, and development on it seems to have stalled. It would be nice
if there were a distributed search project on the Lucene level that we
could use.
I realize this is a rather vague question, but are there any further
suggestions on ways to improve search performance? We need cheap and
dirty ideas, as well as longer term advice on a possible path forward.
Much appreciate
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
I assume you meant 1000 documents. Yes, the page size is in fact
configurable. However, it only obtains the page size * 3. It preloads
the following and previous page too. The point is, it only obtains the
documents that are needed.
On 2014/06/02, 3:03 PM, Tincu Gabriel wrote:
> My bad, It's using the RamDirectory as a cache and a delegate directory
> that you pass in the constructor to do the disk operations, limiting the
> use of the RamDirectory to files that fit a certain size. So i guess the
> underlying Directory implementation will be whatever you choose it to be.
> I'd still try using a MMapDirectory and see if that improves performance.
> Also, regarding the pagination, you said you're retrieving 1000 documents
> at a time. Does that mean that if a query matches 10000 documents you want
> all of them retrieved ?
>
>
> On Mon, Jun 2, 2014 at 12:51 PM, Jamie <ja...@mailarchiva.com> wrote:
>
>> I was under the impression that NRTCachingDirectory will instantiate an
>> MMapDirectory if a 64 bit platform is detected? Is this not the case?
>>
>>
>> On 2014/06/02, 2:09 PM, Tincu Gabriel wrote:
>>
>>> MMapDirectory will do the job for you. RamDirectory has a big warning in
>>> the class description stating that the performance will get killed by an
>>> index larger than a few hundred MB, and NRTCachingDirectory is a wrapper
>>> for RamDirectory and suitable for low update rates. MMap will use the
>>> system RAM to cache as much of the index it can and only hit disk when the
>>> portion of the index you're trying to access isn't cached. I'd put my
>>> money
>>> on switching directory implementations and see what kind of performance
>>> gains that brings to the table.
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Tincu Gabriel <ti...@gmail.com>.
My bad, It's using the RamDirectory as a cache and a delegate directory
that you pass in the constructor to do the disk operations, limiting the
use of the RamDirectory to files that fit a certain size. So i guess the
underlying Directory implementation will be whatever you choose it to be.
I'd still try using a MMapDirectory and see if that improves performance.
Also, regarding the pagination, you said you're retrieving 1000 documents
at a time. Does that mean that if a query matches 10000 documents you want
all of them retrieved ?
On Mon, Jun 2, 2014 at 12:51 PM, Jamie <ja...@mailarchiva.com> wrote:
> I was under the impression that NRTCachingDirectory will instantiate an
> MMapDirectory if a 64 bit platform is detected? Is this not the case?
>
>
> On 2014/06/02, 2:09 PM, Tincu Gabriel wrote:
>
>> MMapDirectory will do the job for you. RamDirectory has a big warning in
>> the class description stating that the performance will get killed by an
>> index larger than a few hundred MB, and NRTCachingDirectory is a wrapper
>> for RamDirectory and suitable for low update rates. MMap will use the
>> system RAM to cache as much of the index it can and only hit disk when the
>> portion of the index you're trying to access isn't cached. I'd put my
>> money
>> on switching directory implementations and see what kind of performance
>> gains that brings to the table.
>>
>>
>
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
I was under the impression that NRTCachingDirectory will instantiate an
MMapDirectory if a 64 bit platform is detected? Is this not the case?
On 2014/06/02, 2:09 PM, Tincu Gabriel wrote:
> MMapDirectory will do the job for you. RamDirectory has a big warning in
> the class description stating that the performance will get killed by an
> index larger than a few hundred MB, and NRTCachingDirectory is a wrapper
> for RamDirectory and suitable for low update rates. MMap will use the
> system RAM to cache as much of the index it can and only hit disk when the
> portion of the index you're trying to access isn't cached. I'd put my money
> on switching directory implementations and see what kind of performance
> gains that brings to the table.
>
Re: search performance
Posted by Tincu Gabriel <ti...@gmail.com>.
MMapDirectory will do the job for you. RamDirectory has a big warning in
the class description stating that the performance will get killed by an
index larger than a few hundred MB, and NRTCachingDirectory is a wrapper
for RamDirectory and suitable for low update rates. MMap will use the
system RAM to cache as much of the index it can and only hit disk when the
portion of the index you're trying to access isn't cached. I'd put my money
on switching directory implementations and see what kind of performance
gains that brings to the table.
On Mon, Jun 2, 2014 at 11:50 AM, Jamie <ja...@mailarchiva.com> wrote:
> Jack
>
> First off, thanks for applying your mind to our performance problem.
>
>
> On 2014/06/02, 1:34 PM, Jack Krupansky wrote:
>
>> Do you have enough system memory to fit the entire index in OS system
>> memory so that the OS can fully cache it instead of thrashing with I/O? Do
>> you see a lot of I/O or are the queries compute-bound?
>>
> Nice idea. The index is 200GB, the machine currently has 128GB RAM. We are
> using SSDs, but disappointingly, installing them didn't reduce search times
> to acceptable levels. I'll have to check your last question regarding
> I/O... I assume it is I/O bound, though will double check...
>
> Currently, we are using
>
> fsDirectory = new NRTCachingDirectory(fsDir, 5.0, 60.0);
>
> Are you proposing we increase maxCachedMB or use the RAMDirectory? With
> the latter, we will still need to persistent the index data to disk, as it
> is undergoing constant updates.
>
>
>> You said you have a 128GB machine, so that sounds small for your index.
>> Have you tried a 256GB machine?
>>
> Nope..didn't think it would make much of a different. I suppose, assuming
> we could store the entire index in RAM it would be helpful. How does one do
> this with Lucene, while still persisting the data?
>
>
>> How frequent are your commits for updates while doing queries?
>>
> Around ten to fifteen documents are being constantly added per second.
>
> Thank again
>
>
> Jamie
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Jack
First off, thanks for applying your mind to our performance problem.
On 2014/06/02, 1:34 PM, Jack Krupansky wrote:
> Do you have enough system memory to fit the entire index in OS system
> memory so that the OS can fully cache it instead of thrashing with
> I/O? Do you see a lot of I/O or are the queries compute-bound?
Nice idea. The index is 200GB, the machine currently has 128GB RAM. We
are using SSDs, but disappointingly, installing them didn't reduce
search times to acceptable levels. I'll have to check your last question
regarding I/O... I assume it is I/O bound, though will double check...
Currently, we are using
fsDirectory = new NRTCachingDirectory(fsDir, 5.0, 60.0);
Are you proposing we increase maxCachedMB or use the RAMDirectory? With
the latter, we will still need to persistent the index data to disk, as
it is undergoing constant updates.
>
> You said you have a 128GB machine, so that sounds small for your
> index. Have you tried a 256GB machine?
Nope..didn't think it would make much of a different. I suppose,
assuming we could store the entire index in RAM it would be helpful. How
does one do this with Lucene, while still persisting the data?
>
> How frequent are your commits for updates while doing queries?
Around ten to fifteen documents are being constantly added per second.
Thank again
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jack Krupansky <ja...@basetechnology.com>.
Do you have enough system memory to fit the entire index in OS system memory
so that the OS can fully cache it instead of thrashing with I/O? Do you see
a lot of I/O or are the queries compute-bound?
You said you have a 128GB machine, so that sounds small for your index. Have
you tried a 256GB machine?
How frequent are your commits for updates while doing queries?
-- Jack Krupansky
-----Original Message-----
From: Jamie
Sent: Monday, June 2, 2014 2:51 AM
To: java-user@lucene.apache.org
Subject: search performance
Greetings
Despite following all the recommended optimizations (as described at
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed) , in some of
our installations, search performance has reached the point where is it
unacceptably slow. For instance, in one environment, the total index
size is 200GB, with 150 million documents indexed. With NRT enabled,
search speed is roughly 5 minutes on average. The server resources are:
2x6 Core Intel CPU, 128GB, 2 SSD for index and RAID 0, with Linux.
The only thing we haven't yet done, is to upgrade Lucene from 4.7.x to
4.8.x. Is this likely to make any noticeable difference in performance?
Clearly, longer term, we need to move to a distributed search model. We
thought to take advantage of the distributed search features offered in
Solr, however, our solution is very tightly integrated into Lucene
directly (since Solr didn't exist when we started out). Moving to Solr
now seems like a daunting prospect. We've also following the Katta
project with interest, but it doesn't appear support distributed
indexing, and development on it seems to have stalled. It would be nice
if there were a distributed search project on the Lucene level that we
could use.
I realize this is a rather vague question, but are there any further
suggestions on ways to improve search performance? We need cheap and
dirty ideas, as well as longer term advice on a possible path forward.
Much appreciate
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: search performance
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
Jamie [jamie@mailarchiva.com] wrote:
> It would be nice if, in future, the Lucene API could provide a
> searchAfter that takes a position (int).
It would not really help with large result sets. At least not with the current underlying implementations. This is tied into your current performance problem, if I understand it correctly.
We seem to have isolated your performance problems to large (10M+) result sets, right?
Requesting the top X results in Lucene works internally by adding to a Priority Queue. The problem with PQs is that they work really well for small result sets and really bad for large result sets (note that result set refers to the collected documents, not to the amount of matching documents). PQs rearranges the internal structure each time a hit is entered that has a score >= the lowest known score. With millions of documents in the result set, this happens all the time. Abstractly there is little difference between small result sets and large: O(n * log n) is fine scaling. In reality the rearrangements of the internal heap structure only works well when it is in CPU cache.
To test this, I created the tiny project https://github.com/tokee/luso
It simulates the workflow (for an extremely loose value of 'simulates') you described with extraction of a large result set by filling a PQ of a given size with docIDs (ints) and scores (floats) and then extracting the ordered docIDs. Running it with different sizes shows how the PQ deteriorates on a 4 core i7 with 8MB level 2 cache:
MAVEN_OPTS=-Xmx4g mvn -q exec:java -Dexec.args="pq 1 1000 10000 10000000000 1000000 5000000 10000000 20000000 30000000 40000000"
Starting 1 threads with extraction method pq
1,000 docs in mean 15 ms, 66 docs/ms.
10,000 docs in mean 47 ms, 212 docs/ms.
100,000 docs in mean 65 ms, 1,538 docs/ms.
500,000 docs in mean 385 ms, 1,298 docs/ms.
1,000,000 docs in mean 832 ms, 1,201 docs/ms.
5,000,000 docs in mean 7,566 ms, 660 docs/ms.
10,000,000 docs in mean 16,482 ms, 606 docs/ms.
20,000,000 docs in mean 39,481 ms, 506 docs/ms.
30,000,000 docs in mean 80,293 ms, 373 docs/ms.
40,000,000 docs in mean 109,537 ms, 365 docs/ms.
As can be seen, relative performance (docs/ms) drops significantly when the document count increases. To add insult to injury, this deterioration patters is optimistic as the test was the only heavy job on my computer. Running 4 of these tests in parallel (1 per core) we would ideally expect about the same speed, but instead we get
MAVEN_OPTS=-Xmx4g mvn -q exec:java -Dexec.args="pq 4 1000 10000 100000 500000 1000000 5000000 10000000 20000000 30000000 40000000"
Starting 4 threads with extraction method pq
1,000 docs in mean 34 ms, 29 docs/ms.
10,000 docs in mean 70 ms, 142 docs/ms.
100,000 docs in mean 102 ms, 980 docs/ms.
500,000 docs in mean 1,340 ms, 373 docs/ms.
1,000,000 docs in mean 2,564 ms, 390 docs/ms.
5,000,000 docs in mean 19,464 ms, 256 docs/ms.
10,000,000 docs in mean 49,985 ms, 200 docs/ms.
20,000,000 docs in mean 112,321 ms, 178 docs/ms.
(I got tired of waiting and stopped after 20M docs)
The conclusion seems clear enough: Using PQ for millions of results will take a long time.
So what can be done? I added an alternative implementation where all the docIDs and scores are collected in two parallel arrays, then merge sorted after collection. That gave the results
MAVEN_OPTS=-Xmx4g mvn -q exec:java -Dexec.args="ip 1 1000 10000 100000 500000 1000000 5000000 10000000 20000000 30000000 40000000"
Starting 1 threads with extraction method ip
1,000 docs in mean 15 ms, 66 docs/ms.
10,000 docs in mean 52 ms, 192 docs/ms.
100,000 docs in mean 73 ms, 1,369 docs/ms.
500,000 docs in mean 363 ms, 1,377 docs/ms.
1,000,000 docs in mean 780 ms, 1,282 docs/ms.
5,000,000 docs in mean 4,634 ms, 1,078 docs/ms.
10,000,000 docs in mean 9,708 ms, 1,030 docs/ms.
20,000,000 docs in mean 20,818 ms, 960 docs/ms.
30,000,000 docs in mean 32,413 ms, 925 docs/ms.
40,000,000 docs in mean 44,235 ms, 904 docs/ms.
Notice how the deterioration of relative speed is a lot less than for PQ. Running this with 4 threads gets us
MAVEN_OPTS=-Xmx4g mvn -q exec:java -Dexec.args="ip 4 1000 10000 100000 500000 1000000 5000000 10000000 20000000 30000000 40000000"
Starting 4 threads with extraction method ip
1,000 docs in mean 35 ms, 28 docs/ms.
10,000 docs in mean 221 ms, 45 docs/ms.
100,000 docs in mean 162 ms, 617 docs/ms.
500,000 docs in mean 639 ms, 782 docs/ms.
1,000,000 docs in mean 1,388 ms, 720 docs/ms.
5,000,000 docs in mean 8,372 ms, 597 docs/ms.
10,000,000 docs in mean 17,933 ms, 557 docs/ms.
20,000,000 docs in mean 36,031 ms, 555 docs/ms.
30,000,000 docs in mean 58,257 ms, 514 docs/ms.
40,000,000 docs in mean 76,763 ms, 521 docs/ms.
The speedup of the merge sorter relative to PQ increases with the size of the collected result. Unfortunately we're still talking minute-class with 60M documents. It all points to the conclusion that collecting millions of sorted document IDs should be avoided if at all possible. A searchAfter that takes a position would either need to use some clever caching or perform the giant sorted collection when called.
- Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Robert
FYI: I've modified the code to utilize the experimental function..
DirectoryReader dirReader =
DirectoryReader.openIfChanged(cachedDirectoryReader,writer, true);
In this case, the IndexReader won't be opened on each search, unless
absolutely necessary.
Regards
Jamie
On 2014/06/03, 1:25 PM, Jamie wrote:
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Greetings Lucene Users
As a follow-up to my earlier mail:
We are also using Lucene segment warmers, as per recommendation,
segments per tier is now set to five, buffer memory is set to
(Runtime.getRuntime().totalMemory()*.08)/1024/1024;
See below for code used to instantiate writer:
LimitTokenCountAnalyzer limitAnalyzer = new
LimitTokenCountAnalyzer(application.getAnalyzerFactory().getAnalyzer(language,
AnalyzerFactory.Operation.INDEX), maxPerFieldTokens);
IndexWriterConfig conf = new
IndexWriterConfig(Version.LUCENE_46, limitAnalyzer);
TieredMergePolicy logMergePolicy = new
TieredMergePolicy();
logMergePolicy.setSegmentsPerTier(5);
conf.setMergePolicy(logMergePolicy);
conf.setRAMBufferSizeMB(bufferMemoryMB);
writer = new IndexWriter(fsDirectory, conf);
writer.getConfig().setMergedSegmentWarmer(readerWarmer);
This particular monster 24 core machine has 110G of RAM. I suppose one
possibly is to load the indexes that aren' t being changed into RAM on
startup. However, the indexes are already residing on fast SSD drives.
We're using the following JRE parameters:
-XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:SurvivorRatio=3
-XX:+AggressiveOpts.
Let me know if there is anything else, we can try to obtain performance
gains.
Much appreciate
Jamie
On 2014/06/20, 9:51 AM, Jamie wrote:
> Hi All
>
> Thank you for all your suggestions. Some of the recommendations hadn't
> yet been implemented, as our code base was using older versions of
> Lucene with reduced capabilities. Thus, far, all the recommendations
> for fast search have been implemented (e.g. using pagination with
> searchAfter, DirectoryReader.openIfChanged, avoiding wrapping lucene
> scoreDoc results, option to disable sorting, etc.).
>
> While, in some environments, search performance has improved
> significantly, in other larger ones we are unfortunately, still seeing
> 1 minute - 5 minute search times. For instance, in one site, the total
> index size is 500GB with 190 million documents indexed. They are
> running a machine with 24 core and 4 SSD drives to house the indexes.
> New emails are being added to the indexes at a rate of 10 message/sec.
>
> One area possible area for improvement: Searching is being conducted
> across several indexes. To accomplish this, on each search, a
> MultiReader is constructed, that consists of several subreaders
> created by the DirectoryReader.openIfChangedMethod. Only one of the
> indexes is updated frequently, the others are never updated. For each
> search, a new IndexSearcher is created passed the MultiReader in the
> constructor. From what I've read, MultiReader and IndexSearcher are
> relatively lightweight and should not impact search performance. Is
> this correct? Is there a faster way to handle searching across
> multiple indexes? What is the performance impact of searching across
> multiple indexes?
>
> Am I correct that using SearchManager can't be used with a MultiReader
> and NRT? I would appreciate all suggestions on how to optimize our
> search performance further. Search time has become a usability issue.
>
> Much appreciate
>
> Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Vitaly Funstein <vf...@gmail.com>.
If you are using stored fields in your index, consider playing with
compression settings, or perhaps turning stored field compression off
altogether. Ways to do this have been discussed in this forum on numerous
occasions. This is highly use case dependent though, as your indexing
performance may or may not suffer, as a tradeoff.
On Fri, Jun 20, 2014 at 1:19 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
> > Am I correct that using SearchManager can't be used with a MultiReader
> and
> > NRT? I would appreciate all suggestions on how to optimize our search
> > performance further. Search time has become a usability issue.
>
> Just have a SearcherManger for every index. MultiReader construction is
> cheap (it is just a wrapper, there is no overhead), so you can ask all
> searcherManagers for the actual IndexReader and build the MultiReader on
> every search request.
>
> Uwe
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: search performance
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
> Am I correct that using SearchManager can't be used with a MultiReader and
> NRT? I would appreciate all suggestions on how to optimize our search
> performance further. Search time has become a usability issue.
Just have a SearcherManger for every index. MultiReader construction is cheap (it is just a wrapper, there is no overhead), so you can ask all searcherManagers for the actual IndexReader and build the MultiReader on every search request.
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Hi All
Thank you for all your suggestions. Some of the recommendations hadn't
yet been implemented, as our code base was using older versions of
Lucene with reduced capabilities. Thus, far, all the recommendations
for fast search have been implemented (e.g. using pagination with
searchAfter, DirectoryReader.openIfChanged, avoiding wrapping lucene
scoreDoc results, option to disable sorting, etc.).
While, in some environments, search performance has improved
significantly, in other larger ones we are unfortunately, still seeing 1
minute - 5 minute search times. For instance, in one site, the total
index size is 500GB with 190 million documents indexed. They are running
a machine with 24 core and 4 SSD drives to house the indexes. New emails
are being added to the indexes at a rate of 10 message/sec.
One area possible area for improvement: Searching is being conducted
across several indexes. To accomplish this, on each search, a
MultiReader is constructed, that consists of several subreaders created
by the DirectoryReader.openIfChangedMethod. Only one of the indexes is
updated frequently, the others are never updated. For each search, a
new IndexSearcher is created passed the MultiReader in the constructor.
From what I've read, MultiReader and IndexSearcher are relatively
lightweight and should not impact search performance. Is this correct?
Is there a faster way to handle searching across multiple indexes? What
is the performance impact of searching across multiple indexes?
Am I correct that using SearchManager can't be used with a MultiReader
and NRT? I would appreciate all suggestions on how to optimize our
search performance further. Search time has become a usability issue.
Much appreciate
Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Jon
I ended up adapting your approach. The solution involves keeping a LRU
cache of page boundary scoredocs and their respective positions. New
positions are added to the cache as new pages are discovered. To cut
down on searches, when scrolling backwards and forwards, the search
begins from nearest cached position.
Cheers
Jamie
On 2014/06/03, 3:24 PM, Jon Stewart wrote:
> With regards to pagination, is there a way for you to cache the
> IndexSearcher, Query, and TopDocs between user pagination requests (a
> lot of webapp frameworks have object caching mechanisms)? If so, you
> may have luck with code like this:
>
> void ensureTopDocs(final int rank) throws IOException {
> if (StartDocIndex > rank) {
> Docs = Searcher.search(SearchQuery, TOP_DOCS_WINDOW);
> StartDocIndex = 0;
> }
> int len = Docs.scoreDocs.length;
> while (StartDocIndex + len <= rank) {
> StartDocIndex += len;
> Docs = Searcher.searchAfter(Docs.scoreDocs[len - 1],
> SearchQuery, TOP_DOCS_WINDOW);
> len = Docs.scoreDocs.length;
> }
> }
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Thanks Jon
I'll investigate your idea further.
It would be nice if, in future, the Lucene API could provide a
searchAfter that takes a position (int).
Regards
Jamie
On 2014/06/03, 3:24 PM, Jon Stewart wrote:
> With regards to pagination, is there a way for you to cache the
> IndexSearcher, Query, and TopDocs between user pagination requests (a
> lot of webapp frameworks have object caching mechanisms)? If so, you
> may have luck with code like this:
>
> void ensureTopDocs(final int rank) throws IOException {
> if (StartDocIndex > rank) {
> Docs = Searcher.search(SearchQuery, TOP_DOCS_WINDOW);
> StartDocIndex = 0;
> }
> int len = Docs.scoreDocs.length;
> while (StartDocIndex + len <= rank) {
> StartDocIndex += len;
> Docs = Searcher.searchAfter(Docs.scoreDocs[len - 1],
> SearchQuery, TOP_DOCS_WINDOW);
> len = Docs.scoreDocs.length;
> }
> }
>
> StartDocIndex is a member variable denoting the current rank of the
> first item in TopDocs ("Docs") window. I call this function before
> each Document retrieval. The common case--of the user looking at the
> first page of results or the user advancing to the next page--is quite
> fast. But it still supports random access, albeit not in constant
> time. OTOH, if your app is concurrent, most search queries will
> probably be returned very quickly so the odd query that wants to jump
> deep into the result set will have more of the server's resources
> available to it.
>
> Also, given the size of your result sets, you have to allocate a lot
> of memory upfront which will then get gc'd after some time. From query
> to query, you will have a decent amount of memory churn. This isn't
> free. My guess is using Lucene's linear (search() & searchAfter())
> pagination will perform faster than your current approach just based
> upon not having to create such large arrays.
>
> I'm not the Lucene expert that Robert is, but this has worked alright for me.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jon Stewart <jo...@lightboxtechnologies.com>.
With regards to pagination, is there a way for you to cache the
IndexSearcher, Query, and TopDocs between user pagination requests (a
lot of webapp frameworks have object caching mechanisms)? If so, you
may have luck with code like this:
void ensureTopDocs(final int rank) throws IOException {
if (StartDocIndex > rank) {
Docs = Searcher.search(SearchQuery, TOP_DOCS_WINDOW);
StartDocIndex = 0;
}
int len = Docs.scoreDocs.length;
while (StartDocIndex + len <= rank) {
StartDocIndex += len;
Docs = Searcher.searchAfter(Docs.scoreDocs[len - 1],
SearchQuery, TOP_DOCS_WINDOW);
len = Docs.scoreDocs.length;
}
}
StartDocIndex is a member variable denoting the current rank of the
first item in TopDocs ("Docs") window. I call this function before
each Document retrieval. The common case--of the user looking at the
first page of results or the user advancing to the next page--is quite
fast. But it still supports random access, albeit not in constant
time. OTOH, if your app is concurrent, most search queries will
probably be returned very quickly so the odd query that wants to jump
deep into the result set will have more of the server's resources
available to it.
Also, given the size of your result sets, you have to allocate a lot
of memory upfront which will then get gc'd after some time. From query
to query, you will have a decent amount of memory churn. This isn't
free. My guess is using Lucene's linear (search() & searchAfter())
pagination will perform faster than your current approach just based
upon not having to create such large arrays.
I'm not the Lucene expert that Robert is, but this has worked alright for me.
cheers,
Jon
On Tue, Jun 3, 2014 at 8:47 AM, Jamie <ja...@mailarchiva.com> wrote:
> Robert. Thanks, I've already done a similar thing. Results on my test
> platform are encouraging..
>
>
> On 2014/06/03, 2:41 PM, Robert Muir wrote:
>>
>> Reopening for every search is not a good idea. this will have an
>> extremely high cost (not as high as what you are doing with "paging"
>> but still not good).
>>
>> Instead consider making it near-realtime, by doing this every second
>> or so instead. Look at SearcherManager for code that helps you do
>> this.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
Jon Stewart, Principal
(646) 719-0317 | jon@lightboxtechnologies.com | Arlington, VA
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Robert. Thanks, I've already done a similar thing. Results on my test
platform are encouraging..
On 2014/06/03, 2:41 PM, Robert Muir wrote:
> Reopening for every search is not a good idea. this will have an
> extremely high cost (not as high as what you are doing with "paging"
> but still not good).
>
> Instead consider making it near-realtime, by doing this every second
> or so instead. Look at SearcherManager for code that helps you do
> this.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Robert Muir <rc...@gmail.com>.
Reopening for every search is not a good idea. this will have an
extremely high cost (not as high as what you are doing with "paging"
but still not good).
Instead consider making it near-realtime, by doing this every second
or so instead. Look at SearcherManager for code that helps you do
this.
On Tue, Jun 3, 2014 at 7:25 AM, Jamie <ja...@mailarchiva.com> wrote:
> Robert
>
> Hmmm..... why did Mike go to all the trouble of implementing NRT search, if
> we are not supposed to be using it?
>
> The user simply wants the latest result set. To me, this doesn't appear out
> of scope for the Lucene project.
>
> Jamie
>
>
> On 2014/06/03, 1:17 PM, Robert Muir wrote:
>>
>> No, you are incorrect. The point of a search engine is to return top-N
>> most relevant.
>>
>> If you insist you need to open an indexreader on every single search,
>> and then return huge amounts of docs, maybe you should use a database
>> instead.
>>
>> On Tue, Jun 3, 2014 at 6:42 AM, Jamie <ja...@mailarchiva.com> wrote:
>>>
>>> Vitality / Robert
>>>
>>> I wouldn't go so far as to call our pagination naive!? Sub-optimal, yes.
>>> Unless I am mistaken, the Lucene library's pagination mechanism, makes
>>> the
>>> assumption that you will cache the scoredocs for the entire result set.
>>> This
>>> is not practical when you have a result set that exceeds 60M. As stated
>>> earlier, in any case, it is the first query that is slow.
>>>
>>> We do open index readers.. since we are using NRT search. Since documents
>>> are being added to the indexes on a continuous basis. When the user
>>> clicks
>>> on the Search button, the user will expect to see the latest result set.
>>> With regards to NRT search, my understanding is that we do need to open
>>> the
>>> index readers on each search operation to see the latest changes.
>>>
>>> Thus, on each search, we combine the indexreaders into a multireader, and
>>> open each reader based their corresponding writer.
>>>
>>> protected IndexReader initIndexReader() {
>>> List<IndexReader> readers = new LinkedList<>();
>>> for (Writer writer : writers) {
>>> readers.add(DirectoryReader.open(writer, true);
>>> }
>>> return MultiReader(readers,true);
>>> }
>>>
>>> Thank you for your ideas/suggestions.
>>>
>>> Regards
>>>
>>> Jamie
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Robert
Hmmm..... why did Mike go to all the trouble of implementing NRT search,
if we are not supposed to be using it?
The user simply wants the latest result set. To me, this doesn't appear
out of scope for the Lucene project.
Jamie
On 2014/06/03, 1:17 PM, Robert Muir wrote:
> No, you are incorrect. The point of a search engine is to return top-N
> most relevant.
>
> If you insist you need to open an indexreader on every single search,
> and then return huge amounts of docs, maybe you should use a database
> instead.
>
> On Tue, Jun 3, 2014 at 6:42 AM, Jamie <ja...@mailarchiva.com> wrote:
>> Vitality / Robert
>>
>> I wouldn't go so far as to call our pagination naive!? Sub-optimal, yes.
>> Unless I am mistaken, the Lucene library's pagination mechanism, makes the
>> assumption that you will cache the scoredocs for the entire result set. This
>> is not practical when you have a result set that exceeds 60M. As stated
>> earlier, in any case, it is the first query that is slow.
>>
>> We do open index readers.. since we are using NRT search. Since documents
>> are being added to the indexes on a continuous basis. When the user clicks
>> on the Search button, the user will expect to see the latest result set.
>> With regards to NRT search, my understanding is that we do need to open the
>> index readers on each search operation to see the latest changes.
>>
>> Thus, on each search, we combine the indexreaders into a multireader, and
>> open each reader based their corresponding writer.
>>
>> protected IndexReader initIndexReader() {
>> List<IndexReader> readers = new LinkedList<>();
>> for (Writer writer : writers) {
>> readers.add(DirectoryReader.open(writer, true);
>> }
>> return MultiReader(readers,true);
>> }
>>
>> Thank you for your ideas/suggestions.
>>
>> Regards
>>
>> Jamie
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Robert Muir <rc...@gmail.com>.
No, you are incorrect. The point of a search engine is to return top-N
most relevant.
If you insist you need to open an indexreader on every single search,
and then return huge amounts of docs, maybe you should use a database
instead.
On Tue, Jun 3, 2014 at 6:42 AM, Jamie <ja...@mailarchiva.com> wrote:
> Vitality / Robert
>
> I wouldn't go so far as to call our pagination naive!? Sub-optimal, yes.
> Unless I am mistaken, the Lucene library's pagination mechanism, makes the
> assumption that you will cache the scoredocs for the entire result set. This
> is not practical when you have a result set that exceeds 60M. As stated
> earlier, in any case, it is the first query that is slow.
>
> We do open index readers.. since we are using NRT search. Since documents
> are being added to the indexes on a continuous basis. When the user clicks
> on the Search button, the user will expect to see the latest result set.
> With regards to NRT search, my understanding is that we do need to open the
> index readers on each search operation to see the latest changes.
>
> Thus, on each search, we combine the indexreaders into a multireader, and
> open each reader based their corresponding writer.
>
> protected IndexReader initIndexReader() {
> List<IndexReader> readers = new LinkedList<>();
> for (Writer writer : writers) {
> readers.add(DirectoryReader.open(writer, true);
> }
> return MultiReader(readers,true);
> }
>
> Thank you for your ideas/suggestions.
>
> Regards
>
> Jamie
>
> On 2014/06/03, 12:29 PM, Vitaly Funstein wrote:
>>
>> Jamie,
>>
>> What if you were to forget for a moment the whole pagination idea, and
>> always capped your search at 1000 results for testing purposes only? This
>> is just to try and pinpoint the bottleneck here; if, regardless of the
>> query parameters, the search latency stays roughly the same and well below
>> 5 min, you now have the answer - the problem is your naive implementation
>> of pagination which results in snowballing result numbers and search
>> times,
>> the closer you get to the end of the results range. Otherwise, I would
>> focus on your query and filter next.
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Vitality / Robert
I wouldn't go so far as to call our pagination naive!? Sub-optimal, yes.
Unless I am mistaken, the Lucene library's pagination mechanism, makes
the assumption that you will cache the scoredocs for the entire result
set. This is not practical when you have a result set that exceeds 60M.
As stated earlier, in any case, it is the first query that is slow.
We do open index readers.. since we are using NRT search. Since
documents are being added to the indexes on a continuous basis. When the
user clicks on the Search button, the user will expect to see the latest
result set. With regards to NRT search, my understanding is that we do
need to open the index readers on each search operation to see the
latest changes.
Thus, on each search, we combine the indexreaders into a multireader,
and open each reader based their corresponding writer.
protected IndexReader initIndexReader() {
List<IndexReader> readers = new LinkedList<>();
for (Writer writer : writers) {
readers.add(DirectoryReader.open(writer, true);
}
return MultiReader(readers,true);
}
Thank you for your ideas/suggestions.
Regards
Jamie
On 2014/06/03, 12:29 PM, Vitaly Funstein wrote:
> Jamie,
>
> What if you were to forget for a moment the whole pagination idea, and
> always capped your search at 1000 results for testing purposes only? This
> is just to try and pinpoint the bottleneck here; if, regardless of the
> query parameters, the search latency stays roughly the same and well below
> 5 min, you now have the answer - the problem is your naive implementation
> of pagination which results in snowballing result numbers and search times,
> the closer you get to the end of the results range. Otherwise, I would
> focus on your query and filter next.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Vitaly Funstein <vf...@gmail.com>.
Jamie,
What if you were to forget for a moment the whole pagination idea, and
always capped your search at 1000 results for testing purposes only? This
is just to try and pinpoint the bottleneck here; if, regardless of the
query parameters, the search latency stays roughly the same and well below
5 min, you now have the answer - the problem is your naive implementation
of pagination which results in snowballing result numbers and search times,
the closer you get to the end of the results range. Otherwise, I would
focus on your query and filter next.
On Tue, Jun 3, 2014 at 3:21 AM, Jamie <ja...@mailarchiva.com> wrote:
> Vitaly
>
> See below:
>
>
> On 2014/06/03, 12:09 PM, Vitaly Funstein wrote:
>
>> A couple of questions.
>>
>> 1. What are you trying to achieve by setting the current thread's priority
>> to max possible value? Is it grabbing as much CPU time as possible? In my
>> experience, mucking with thread priorities like this is at best futile,
>> and
>> at worst quite detrimental to responsiveness and overall performance of
>> the
>> system as a whole. I would remove that line.
>>
> Yes, you are right to be worried about this, especially since thread
> priorities behave differently on different platforms.
>
>
>
>> 2. This seems suspicious:
>>
>> if (getPagination()) {
>> max = start + length;
>> } else {
>> max = getMaxResults();
>> }
>>
>> If start is at 100M, and length is 1000 - what do you think Lucene will
>> try
>> and do when you pass this max to the collector?
>>
> I dont see the problem here. The collector will start from zero to max
> results. I agree that from a performance perspective, ts not ideal to
> return all results from the beginning of the search, but the Lucene API us
> with no choice. I simply do not know the ScoreDoc to start from. If I did
> keep a record of it, then I would need to store all scoredocs for the
> entire result set. When there are 60M+ results, this can be problematic in
> terms of memory consumption. It would be far nicer if there was a
> searchAfter function that took a position as an integer.
>
> Regards
>
> Jamie
>
>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Vitaly
See below:
On 2014/06/03, 12:09 PM, Vitaly Funstein wrote:
> A couple of questions.
>
> 1. What are you trying to achieve by setting the current thread's priority
> to max possible value? Is it grabbing as much CPU time as possible? In my
> experience, mucking with thread priorities like this is at best futile, and
> at worst quite detrimental to responsiveness and overall performance of the
> system as a whole. I would remove that line.
Yes, you are right to be worried about this, especially since thread
priorities behave differently on different platforms.
>
> 2. This seems suspicious:
>
> if (getPagination()) {
> max = start + length;
> } else {
> max = getMaxResults();
> }
>
> If start is at 100M, and length is 1000 - what do you think Lucene will try
> and do when you pass this max to the collector?
I dont see the problem here. The collector will start from zero to max
results. I agree that from a performance perspective, ts not ideal to
return all results from the beginning of the search, but the Lucene API
us with no choice. I simply do not know the ScoreDoc to start from. If I
did keep a record of it, then I would need to store all scoredocs for
the entire result set. When there are 60M+ results, this can be
problematic in terms of memory consumption. It would be far nicer if
there was a searchAfter function that took a position as an integer.
Regards
Jamie
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Vitaly Funstein <vf...@gmail.com>.
A couple of questions.
1. What are you trying to achieve by setting the current thread's priority
to max possible value? Is it grabbing as much CPU time as possible? In my
experience, mucking with thread priorities like this is at best futile, and
at worst quite detrimental to responsiveness and overall performance of the
system as a whole. I would remove that line.
2. This seems suspicious:
if (getPagination()) {
max = start + length;
} else {
max = getMaxResults();
}
If start is at 100M, and length is 1000 - what do you think Lucene will try
and do when you pass this max to the collector?
On Tue, Jun 3, 2014 at 2:55 AM, Jamie <ja...@mailarchiva.com> wrote:
> FYI: We are also using a multireader to search over multiple index readers.
>
> Search under a million documents yields good response times. When you get
> into the 60M territory, search slows to a crawl.
>
> On 2014/06/03, 11:47 AM, Jamie wrote:
>
>> Sure... see below:
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
FYI: We are also using a multireader to search over multiple index readers.
Search under a million documents yields good response times. When you
get into the 60M territory, search slows to a crawl.
On 2014/06/03, 11:47 AM, Jamie wrote:
> Sure... see below:
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Sure... see below:
protected void search(Query query, Filter queryFilter, Sort sort)
throws BlobSearchException {
try {
logger.debug("start search {searchquery='" +
getSearchQuery() +
"',query='"+query.toString()+"',filterQuery='"+queryFilter+"',sort='"+sort+"'}");
Thread.currentThread().setPriority(Thread.MAX_PRIORITY);
results.clear();
int max;
if (getPagination()) {
max = start + length;
} else {
max = getMaxResults();
}
// release the old volume searchers
IndexReader indexReader = initIndexReader();
searcher = new IndexSearcher(indexReader,executor);
TopFieldCollector fieldCollector =
TopFieldCollector.create(sort, max,true, false, false, true);
searcher.search(query, queryFilter, fieldCollector);
TopDocs topDocs;
if (getPagination()) {
topDocs = fieldCollector.topDocs(start,length);
} else {
topDocs = fieldCollector.topDocs();
}
int count = 0;
for (int i = 0; i < topDocs.scoreDocs.length; i++) {
if ((getMaxResults()>0 && count > getMaxResults()) ||
(getPagination() && count++>=length)) { break; }
results.add(topDocs.scoreDocs[i]);
}
totalHits = fieldCollector.getTotalHits();
logger.debug("search executed successfully {query='"+
getSearchQuery() + "',returnedresults='" + results.size()+ "'}");
} catch (Exception io) {
throw new BlobSearchException("failed to execute search
query {searchquery='"+ getSearchQuery() + "}", io, logger,
ChainedException.Level.DEBUG);
}
}
On 2014/06/03, 11:41 AM, Rob Audenaerde wrote:
> Hi Jamie,
>
> What is included in the 5 minutes?
>
> Just the call to the searcher?
>
> seacher.search(...) ?
>
> Can you show a bit more of the code you use?
>
>
>
> On Tue, Jun 3, 2014 at 11:32 AM, Jamie <ja...@mailarchiva.com> wrote:
>
>> Vitaly
>>
>> Thanks for the contribution. Unfortunately, we cannot use Lucene's
>> pagination function, because in reality the user can skip pages to start
>> the search at any point, not just from the end of the previous search. Even
>> the
>> first search (without any pagination), with a max of 1000 hits, takes 5
>> minutes to complete.
>>
>> Regards
>>
>> Jamie
>>
>> On 2014/06/03, 10:54 AM, Vitaly Funstein wrote:
>>
>>> Something doesn't quite add up.
>>>
>>> TopFieldCollector fieldCollector = TopFieldCollector.create(sort,
>>> max,true,
>>>
>>>> false, false, true);
>>>>
>>>> We use pagination, so only returning 1000 documents or so at a time.
>>>>
>>>>
>>>> You say you are using pagination, yet the API you are using to create
>>> your
>>> collector isn't how you would utilize Lucene's built-in "pagination"
>>> feature (unless misunderstand the API). If the max is the snippet above is
>>> 1000, then you're simply returning top 1000 docs every time you execute
>>> your search. Otherwise... well, could you actually post a bit more of your
>>> code that runs the search here, in particular?
>>>
>>> Assuming that the max is much larger than 1000, however, you could call
>>> fieldCollector.topDocs(int, int) after accumulating hits using this
>>> collector, but this won't work multiple times per query execution,
>>> according to the javadoc. So you either have to re-execute the full
>>> search,
>>> and then get the next chunk of ScoreDocs, or use the proper API for this,
>>> one that accepts as a parameter the end of the previous page of results,
>>> i.e. IndexSearcher.searchAfter(ScoreDoc, ...)
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Rob Audenaerde <ro...@gmail.com>.
Hi Jamie,
What is included in the 5 minutes?
Just the call to the searcher?
seacher.search(...) ?
Can you show a bit more of the code you use?
On Tue, Jun 3, 2014 at 11:32 AM, Jamie <ja...@mailarchiva.com> wrote:
> Vitaly
>
> Thanks for the contribution. Unfortunately, we cannot use Lucene's
> pagination function, because in reality the user can skip pages to start
> the search at any point, not just from the end of the previous search. Even
> the
> first search (without any pagination), with a max of 1000 hits, takes 5
> minutes to complete.
>
> Regards
>
> Jamie
>
> On 2014/06/03, 10:54 AM, Vitaly Funstein wrote:
>
>> Something doesn't quite add up.
>>
>> TopFieldCollector fieldCollector = TopFieldCollector.create(sort,
>> max,true,
>>
>>> false, false, true);
>>>
>>> We use pagination, so only returning 1000 documents or so at a time.
>>>
>>>
>>> You say you are using pagination, yet the API you are using to create
>> your
>> collector isn't how you would utilize Lucene's built-in "pagination"
>> feature (unless misunderstand the API). If the max is the snippet above is
>> 1000, then you're simply returning top 1000 docs every time you execute
>> your search. Otherwise... well, could you actually post a bit more of your
>> code that runs the search here, in particular?
>>
>> Assuming that the max is much larger than 1000, however, you could call
>> fieldCollector.topDocs(int, int) after accumulating hits using this
>> collector, but this won't work multiple times per query execution,
>> according to the javadoc. So you either have to re-execute the full
>> search,
>> and then get the next chunk of ScoreDocs, or use the proper API for this,
>> one that accepts as a parameter the end of the previous page of results,
>> i.e. IndexSearcher.searchAfter(ScoreDoc, ...)
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Vitaly
Thanks for the contribution. Unfortunately, we cannot use Lucene's
pagination function, because in reality the user can skip pages to start
the search at any point, not just from the end of the previous search.
Even the
first search (without any pagination), with a max of 1000 hits, takes 5
minutes to complete.
Regards
Jamie
On 2014/06/03, 10:54 AM, Vitaly Funstein wrote:
> Something doesn't quite add up.
>
> TopFieldCollector fieldCollector = TopFieldCollector.create(sort, max,true,
>> false, false, true);
>>
>> We use pagination, so only returning 1000 documents or so at a time.
>>
>>
> You say you are using pagination, yet the API you are using to create your
> collector isn't how you would utilize Lucene's built-in "pagination"
> feature (unless misunderstand the API). If the max is the snippet above is
> 1000, then you're simply returning top 1000 docs every time you execute
> your search. Otherwise... well, could you actually post a bit more of your
> code that runs the search here, in particular?
>
> Assuming that the max is much larger than 1000, however, you could call
> fieldCollector.topDocs(int, int) after accumulating hits using this
> collector, but this won't work multiple times per query execution,
> according to the javadoc. So you either have to re-execute the full search,
> and then get the next chunk of ScoreDocs, or use the proper API for this,
> one that accepts as a parameter the end of the previous page of results,
> i.e. IndexSearcher.searchAfter(ScoreDoc, ...)
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Vitaly Funstein <vf...@gmail.com>.
Something doesn't quite add up.
TopFieldCollector fieldCollector = TopFieldCollector.create(sort, max,true,
> false, false, true);
>
> We use pagination, so only returning 1000 documents or so at a time.
>
>
You say you are using pagination, yet the API you are using to create your
collector isn't how you would utilize Lucene's built-in "pagination"
feature (unless misunderstand the API). If the max is the snippet above is
1000, then you're simply returning top 1000 docs every time you execute
your search. Otherwise... well, could you actually post a bit more of your
code that runs the search here, in particular?
Assuming that the max is much larger than 1000, however, you could call
fieldCollector.topDocs(int, int) after accumulating hits using this
collector, but this won't work multiple times per query execution,
according to the javadoc. So you either have to re-execute the full search,
and then get the next chunk of ScoreDocs, or use the proper API for this,
one that accepts as a parameter the end of the previous page of results,
i.e. IndexSearcher.searchAfter(ScoreDoc, ...)
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Tom
Thanks for the offer of assistance.
On 2014/06/02, 12:02 PM, Tincu Gabriel wrote:
> What kind of queries are you pushing into the index.
We are indexing regular emails + attachments.
Typical query is something like:
filter: to:mbox000008 from:mbox000008 cc:mbox000008 bcc:mbox000008
deliveredto:mbox000008 sender:mbox000008 recipient:mbox000008
combined with filter query "cat:email"
We also use range queries based on date.
> Do they match a lot of documents ?
Yes, although we are using a collector...
TopFieldCollector fieldCollector = TopFieldCollector.create(sort,
max,true, false, false, true);
We use pagination, so only returning 1000 documents or so at a time.
> Do you do any sorting on the result set?
Yes
> What is the average
> document size ?
approx 100KB, We are indexing email body + attachment content.
> Do you have a lot of update traffic ?
Yes we have alot of update traffic, particularly in the environment i
referred to. Is there a way to prioritize searching as apposed to update?
I suppose we could block all indexing while searching is on the go? Is
there such as option in Lucene, or should we implement this?
> What kind of schema
> does your index use ?
Not sure exactly what you are referring to here. We do have alot of
stored fields (to, from bcc, cc, etc.). The body and attachments are
analyzed.
Regards
Jamie
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Tincu Gabriel <ti...@gmail.com>.
What kind of queries are you pushing into the index. Do they match a lot of
documents ? Do you do any sorting on the result set? What is the average
document size ? Do you have a lot of update traffic ? What kind of schema
does your index use ?
On Mon, Jun 2, 2014 at 6:51 AM, Jamie <ja...@mailarchiva.com> wrote:
> Greetings
>
> Despite following all the recommended optimizations (as described at
> http://wiki.apache.org/lucene-java/ImproveSearchingSpeed) , in some of
> our installations, search performance has reached the point where is it
> unacceptably slow. For instance, in one environment, the total index size
> is 200GB, with 150 million documents indexed. With NRT enabled, search
> speed is roughly 5 minutes on average. The server resources are: 2x6 Core
> Intel CPU, 128GB, 2 SSD for index and RAID 0, with Linux.
>
> The only thing we haven't yet done, is to upgrade Lucene from 4.7.x to
> 4.8.x. Is this likely to make any noticeable difference in performance?
>
> Clearly, longer term, we need to move to a distributed search model. We
> thought to take advantage of the distributed search features offered in
> Solr, however, our solution is very tightly integrated into Lucene directly
> (since Solr didn't exist when we started out). Moving to Solr now seems
> like a daunting prospect. We've also following the Katta project with
> interest, but it doesn't appear support distributed indexing, and
> development on it seems to have stalled. It would be nice if there were a
> distributed search project on the Lucene level that we could use.
>
> I realize this is a rather vague question, but are there any further
> suggestions on ways to improve search performance? We need cheap and dirty
> ideas, as well as longer term advice on a possible path forward.
>
> Much appreciate
>
> Jamie
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: search performance
Posted by Robert Muir <rc...@gmail.com>.
Check and make sure you are not opening an indexreader for every
search. Be sure you don't do that.
On Mon, Jun 2, 2014 at 2:51 AM, Jamie <ja...@mailarchiva.com> wrote:
> Greetings
>
> Despite following all the recommended optimizations (as described at
> http://wiki.apache.org/lucene-java/ImproveSearchingSpeed) , in some of our
> installations, search performance has reached the point where is it
> unacceptably slow. For instance, in one environment, the total index size is
> 200GB, with 150 million documents indexed. With NRT enabled, search speed is
> roughly 5 minutes on average. The server resources are: 2x6 Core Intel CPU,
> 128GB, 2 SSD for index and RAID 0, with Linux.
>
> The only thing we haven't yet done, is to upgrade Lucene from 4.7.x to
> 4.8.x. Is this likely to make any noticeable difference in performance?
>
> Clearly, longer term, we need to move to a distributed search model. We
> thought to take advantage of the distributed search features offered in
> Solr, however, our solution is very tightly integrated into Lucene directly
> (since Solr didn't exist when we started out). Moving to Solr now seems like
> a daunting prospect. We've also following the Katta project with interest,
> but it doesn't appear support distributed indexing, and development on it
> seems to have stalled. It would be nice if there were a distributed search
> project on the Lucene level that we could use.
>
> I realize this is a rather vague question, but are there any further
> suggestions on ways to improve search performance? We need cheap and dirty
> ideas, as well as longer term advice on a possible path forward.
>
> Much appreciate
>
> Jamie
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Christoph Kaser <lu...@iconparc.de>.
Can you take thread stacktraces (repeatedly) during those 5 minute
searches? That might give you (or someone on the mailing list) a clue
where all that time is spent.
You could try using jstack for that:
http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstack.html
Regards
Christoph
Am 03.06.2014 08:17, schrieb Jamie:
> Toke
>
> Thanks for the comment.
>
> Unfortunately, in this instance, it is a live production system, so we
> cannot conduct experiments. The number is definitely accurate.
>
> We have many different systems with a similar load that observe the
> same performance issue. To my knowledge, the Lucene integration code
> is fairly well optimized.
>
> I've requested access to the indexes so that we can perform further
> testing.
>
> Regards
>
> Jamie
>
> On 2014/06/03, 8:09 AM, Toke Eskildsen wrote:
>> On Mon, 2014-06-02 at 08:51 +0200, Jamie wrote:
>>
>> [200GB, 150M documents]
>>
>>> With NRT enabled, search speed is roughly 5 minutes on average.
>>> The server resources are:
>>> 2x6 Core Intel CPU, 128GB, 2 SSD for index and RAID 0, with Linux.
>> 5 minutes is extremely long. Is that really the right number? I do not
>> see a hardware upgrade changing that with the fine machine you're using.
>>
>> What is your search speed if you disable continuous updates?
>>
>> When you restart the searcher, how long does the first search take?
>>
>>
>> - Toke Eskildsen, State and University Library, Denmark
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Toke
Thanks for the contact. See below:
On 2014/06/03, 9:17 AM, Toke Eskildsen wrote:
> On Tue, 2014-06-03 at 08:17 +0200, Jamie wrote:
>> Unfortunately, in this instance, it is a live production system, so we
>> cannot conduct experiments. The number is definitely accurate.
>>
>> We have many different systems with a similar load that observe the same
>> performance issue. To my knowledge, the Lucene integration code is
>> fairly well optimized.
> It is possible that the extreme slowness is a combination of factors,
> but with a bit of luck it will boil down to a single thing. Standard
> procedure it to disable features until it performs well, so
>
> - Disable running updates
No can do.
> - Limit page size
Done this.
> - Limit lookup of returned fields
Done this.
> - Disable highlighting
No highlighting.
> - Simpler queries
They are as simple as possible.
> - Whatever else you might think of
Our application has been using Lucene for seven years. It has been
constantly optimized over that period.
I'll conduct further testing...
>
> At some point along the way I would expect a sharp increase in
> performance.
>
>> I've requested access to the indexes so that we can perform further testing.
> Great.
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Tue, 2014-06-03 at 08:17 +0200, Jamie wrote:
> Unfortunately, in this instance, it is a live production system, so we
> cannot conduct experiments. The number is definitely accurate.
>
> We have many different systems with a similar load that observe the same
> performance issue. To my knowledge, the Lucene integration code is
> fairly well optimized.
It is possible that the extreme slowness is a combination of factors,
but with a bit of luck it will boil down to a single thing. Standard
procedure it to disable features until it performs well, so
- Disable running updates
- Limit page size
- Limit lookup of returned fields
- Disable highlighting
- Simpler queries
- Whatever else you might think of
At some point along the way I would expect a sharp increase in
performance.
> I've requested access to the indexes so that we can perform further testing.
Great.
- Toke Eskildsen, State and University Library, Denmark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Jamie <ja...@mailarchiva.com>.
Toke
Thanks for the comment.
Unfortunately, in this instance, it is a live production system, so we
cannot conduct experiments. The number is definitely accurate.
We have many different systems with a similar load that observe the same
performance issue. To my knowledge, the Lucene integration code is
fairly well optimized.
I've requested access to the indexes so that we can perform further testing.
Regards
Jamie
On 2014/06/03, 8:09 AM, Toke Eskildsen wrote:
> On Mon, 2014-06-02 at 08:51 +0200, Jamie wrote:
>
> [200GB, 150M documents]
>
>> With NRT enabled, search speed is roughly 5 minutes on average.
>> The server resources are:
>> 2x6 Core Intel CPU, 128GB, 2 SSD for index and RAID 0, with Linux.
> 5 minutes is extremely long. Is that really the right number? I do not
> see a hardware upgrade changing that with the fine machine you're using.
>
> What is your search speed if you disable continuous updates?
>
> When you restart the searcher, how long does the first search take?
>
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: search performance
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2014-06-02 at 08:51 +0200, Jamie wrote:
[200GB, 150M documents]
> With NRT enabled, search speed is roughly 5 minutes on average.
> The server resources are:
> 2x6 Core Intel CPU, 128GB, 2 SSD for index and RAID 0, with Linux.
5 minutes is extremely long. Is that really the right number? I do not
see a hardware upgrade changing that with the fine machine you're using.
What is your search speed if you disable continuous updates?
When you restart the searcher, how long does the first search take?
- Toke Eskildsen, State and University Library, Denmark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org