You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Chris Salem <ch...@mainsequence.net> on 2008/12/17 22:10:02 UTC
lucene 2.4 sorting slowness
Hello,
I have an index with ~4000000 documents and some 200 fields. Searching without sorting takes around 300 - 500 ms, when sorting on dates (formated as 'yyyy-mm-dd') searching time takes on average 15 seconds. Here's the code that does the search:
hits = searcher.search(query, new Sort(new SortField("slast_modified", false)));
and here's how that field is being indexed:
doc.add(new Field("slast_modified", "2008-12-12", Field.Store.NO, Field.Index.NOT_ANALYZED));
Am I doing something wrong? Is there a bug in lucene and if so is there a way to work around it so that search speed is increased to something reasonable?
Thanks
Sincerely,
Chris Salem
Development Team
Main Sequence Technologies, Inc.
PCRecruiter.net - PCRecruiter Support
chris@mainsequence.net
P: 440.946.5214 ext 5458
F: 440.856.0312
This email and any files transmitted with it may contain confidential information intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Finally, the recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. Main Sequence Technologies, Inc. 4420 Sherwin Rd. Willoughby OH 44094 www.pcrecruiter.net
(The following links were included with this email:)
http://www.pcrecruiter.net/
http://www.pcrecruiter.net/support.htm
mailto:chris@mainsequence.net
(The following links were included with this email:)
http://www.pcrecruiter.net/
http://www.pcrecruiter.net/support.htm
mailto:chris@mainsequence.net
Re: lucene 2.4 sorting slowness
Posted by Erick Erickson <er...@gmail.com>.
Are you measuring only the time to execute the searcher.search line or
are you measuring the time it takes to iterate the Hits object? The reason
I ask is that something like
for (int idx = 0; idx < hits.length(); ++idx) {
<do something with doc idx>
}
will re-execute the query every 100 documents examined or so. For
examining more than the first few documents, a HitCollector is preferred.
But Michael's question is at least as germane.
Best
Erick
On Wed, Dec 17, 2008 at 4:10 PM, Chris Salem <ch...@mainsequence.net> wrote:
> Hello,
> I have an index with ~4000000 documents and some 200 fields. Searching
> without sorting takes around 300 - 500 ms, when sorting on dates (formated
> as 'yyyy-mm-dd') searching time takes on average 15 seconds. Here's the code
> that does the search:
> hits = searcher.search(query, new Sort(new SortField("slast_modified",
> false)));
> and here's how that field is being indexed:
> doc.add(new Field("slast_modified", "2008-12-12", Field.Store.NO,
> Field.Index.NOT_ANALYZED));
> Am I doing something wrong? Is there a bug in lucene and if so is there a
> way to work around it so that search speed is increased to something
> reasonable?
> Thanks
> Sincerely,
> Chris Salem
> Development Team
> Main Sequence Technologies, Inc.
> PCRecruiter.net - PCRecruiter Support
> chris@mainsequence.net
> P: 440.946.5214 ext 5458
> F: 440.856.0312
>
> This email and any files transmitted with it may contain confidential
> information intended solely for the use of the individual or entity to whom
> they are addressed. If you have received this email in error please notify
> the sender. Please note that any views or opinions presented in this email
> are solely those of the author and do not necessarily represent those of the
> company. Finally, the recipient should check this email and any attachments
> for the presence of viruses. The company accepts no liability for any damage
> caused by any virus transmitted by this email. Main Sequence Technologies,
> Inc. 4420 Sherwin Rd. Willoughby OH 44094 www.pcrecruiter.net
>
> (The following links were included with this email:)
> http://www.pcrecruiter.net/
>
> http://www.pcrecruiter.net/support.htm
>
> mailto:chris@mainsequence.net
>
>
>
> (The following links were included with this email:)
> http://www.pcrecruiter.net/
>
> http://www.pcrecruiter.net/support.htm
>
> mailto:chris@mainsequence.net
>
>
>
Re: lucene 2.4 sorting slowness
Posted by Michael McCandless <lu...@mikemccandless.com>.
Are you warming the searcher first, and then testing the sort
performance? (The first query is slow because it populates the
FieldCache, internally, which is then reused for subsequent queries as
long as you don't close that reader/searcher).
Mike
Chris Salem wrote:
> Hello,
> I have an index with ~4000000 documents and some 200 fields.
> Searching without sorting takes around 300 - 500 ms, when sorting on
> dates (formated as 'yyyy-mm-dd') searching time takes on average 15
> seconds. Here's the code that does the search:
> hits = searcher.search(query, new Sort(new
> SortField("slast_modified", false)));
> and here's how that field is being indexed:
> doc.add(new Field("slast_modified", "2008-12-12", Field.Store.NO,
> Field.Index.NOT_ANALYZED));
> Am I doing something wrong? Is there a bug in lucene and if so is
> there a way to work around it so that search speed is increased to
> something reasonable?
> Thanks
> Sincerely,
> Chris Salem
> Development Team
> Main Sequence Technologies, Inc.
> PCRecruiter.net - PCRecruiter Support
> chris@mainsequence.net
> P: 440.946.5214 ext 5458
> F: 440.856.0312
>
> This email and any files transmitted with it may contain
> confidential information intended solely for the use of the
> individual or entity to whom they are addressed. If you have
> received this email in error please notify the sender. Please note
> that any views or opinions presented in this email are solely those
> of the author and do not necessarily represent those of the company.
> Finally, the recipient should check this email and any attachments
> for the presence of viruses. The company accepts no liability for
> any damage caused by any virus transmitted by this email. Main
> Sequence Technologies, Inc. 4420 Sherwin Rd. Willoughby OH 44094 www.pcrecruiter.net
>
> (The following links were included with this email:)
> http://www.pcrecruiter.net/
>
> http://www.pcrecruiter.net/support.htm
>
> mailto:chris@mainsequence.net
>
>
>
> (The following links were included with this email:)
> http://www.pcrecruiter.net/
>
> http://www.pcrecruiter.net/support.htm
>
> mailto:chris@mainsequence.net
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org