You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Drew Farris <dr...@gmail.com> on 2004/06/25 20:24:06 UTC
Optimizing for long queries?
Apologies if this is a FAQ, but I didn't have much luck searching the
list archives for answers on this subject:
I'm using Lucene in a context where we have frequently have queries
that search for as many as 30-50 terms in a single field. Does anyone
have any thoughts concerning ways optimize Lucene for queries of these
lengths?
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Title of PDF
Posted by Michael Giles <mg...@visionstudio.com>.
Don,
I think you misunderstood Otis. You will need to use some sort of parser
(i.e. pdfbox, xpdf) to get the title text from the PDF (I assume you are
indexing the documents, so you have this already). Then you create a
"title" field in your index and store the text of the title in there (so
that you can return it with the results). That is independent of what you
do with the rest of the PDF document.
-Mike
At 11:17 AM 6/28/2004, Don Vaillancourt wrote:
>It can't be that simple because the field will contain the whole PDF and
>not just the title. And for PDFs that are 3 or 4 megs, it is really not
>reasonable to store the who PDF in the collection just to get the title.
________________________________________________________________________
Save and share anything you find online - Furl @ http://www.furl.net
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Title of PDF
Posted by Don Vaillancourt <do...@webimpact.com>.
It can't be that simple because the field will contain the whole PDF and
not just the title. And for PDFs that are 3 or 4 megs, it is really not
reasonable to store the who PDF in the collection just to get the title.
At 11:09 AM 28/06/2004, you wrote:
>Don,
>
>Sure, just make sure you pick the Field that stores the original text
>(e.g. Keyword or Text, NOT UnStored :)). It does not matter that the
>source of the data is (PDF or HTML or Java String), as long as you use
>the appropriate Field method. Also, don't pass a Reader, as that won't
>save your original text.
>
>Otis
>
>
>--- Don Vaillancourt <do...@webimpact.com> wrote:
> > Is there a way to store the title of a PDF in a Lucene collection for
> >
> > retrieval when listing search results.
> >
> > Thanks
> >
> >
> > Don Vaillancourt
> > Director of Software Development
> >
> > WEB IMPACT INC.
> > 416-815-2000 ext. 245
> > email: donv@web-impact.com
> > web: http://www.web-impact.com
> >
> >
> >
> >
> > This email message is intended only for the addressee(s)
> > and contains information that may be confidential and/or
> > copyright. If you are not the intended recipient please
> > notify the sender by reply email and immediately delete
> > this email. Use, disclosure or reproduction of this email
> > by anyone other than the intended recipient(s) is strictly
> > prohibited. No representation is made that this email or
> > any attachments are free of viruses. Virus scanning is
> > recommended and is the responsibility of the recipient.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Don Vaillancourt
Director of Software Development
WEB IMPACT INC.
416-815-2000 ext. 245
email: donv@web-impact.com
web: http://www.web-impact.com
This email message is intended only for the addressee(s)
and contains information that may be confidential and/or
copyright. If you are not the intended recipient please
notify the sender by reply email and immediately delete
this email. Use, disclosure or reproduction of this email
by anyone other than the intended recipient(s) is strictly
prohibited. No representation is made that this email or
any attachments are free of viruses. Virus scanning is
recommended and is the responsibility of the recipient.
Re: Title of PDF
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Don,
Sure, just make sure you pick the Field that stores the original text
(e.g. Keyword or Text, NOT UnStored :)). It does not matter that the
source of the data is (PDF or HTML or Java String), as long as you use
the appropriate Field method. Also, don't pass a Reader, as that won't
save your original text.
Otis
--- Don Vaillancourt <do...@webimpact.com> wrote:
> Is there a way to store the title of a PDF in a Lucene collection for
>
> retrieval when listing search results.
>
> Thanks
>
>
> Don Vaillancourt
> Director of Software Development
>
> WEB IMPACT INC.
> 416-815-2000 ext. 245
> email: donv@web-impact.com
> web: http://www.web-impact.com
>
>
>
>
> This email message is intended only for the addressee(s)
> and contains information that may be confidential and/or
> copyright. If you are not the intended recipient please
> notify the sender by reply email and immediately delete
> this email. Use, disclosure or reproduction of this email
> by anyone other than the intended recipient(s) is strictly
> prohibited. No representation is made that this email or
> any attachments are free of viruses. Virus scanning is
> recommended and is the responsibility of the recipient.
>
>
>
>
>
>
>
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Title of PDF
Posted by Don Vaillancourt <do...@webimpact.com>.
Is there a way to store the title of a PDF in a Lucene collection for
retrieval when listing search results.
Thanks
Don Vaillancourt
Director of Software Development
WEB IMPACT INC.
416-815-2000 ext. 245
email: donv@web-impact.com
web: http://www.web-impact.com
This email message is intended only for the addressee(s)
and contains information that may be confidential and/or
copyright. If you are not the intended recipient please
notify the sender by reply email and immediately delete
this email. Use, disclosure or reproduction of this email
by anyone other than the intended recipient(s) is strictly
prohibited. No representation is made that this email or
any attachments are free of viruses. Virus scanning is
recommended and is the responsibility of the recipient.
Re: Optimizing for long queries?
Posted by Drew Farris <dr...@gmail.com>.
On Mon, 28 Jun 2004 10:04:40 +0200, Julien Nioche
<ju...@lingway.com> wrote:
> Hello Drew,
>
> I don't think it's in the FAQ.
>
Julien,
Thanks for the advice, and the in-depth exploration of INDEX_INTERVAL
here and on the developer's list. If I have the opportunity to run
similar benchmarks comparing both modified INDEX_INTERVALS and sorted
queries, I'll share the results as well.
You mentioned that you were using queries from query logs for your
application. Do you have a general idea of the average query length of
these queries?
Drew.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Optimizing for long queries? >> 40% faster by changing INDEX_INTERVAL
Posted by Julien Nioche <Ju...@lingway.com>.
The xls files did not pass. You can download them from the following URLs :
http://jnioche.freesurf.fr/shortQueries.xls
http://jnioche.freesurf.fr/longQueries.xls
----- Original Message -----
From: "Julien Nioche" <Ju...@lingway.com>
To: "Lucene Developers List" <lu...@jakarta.apache.org>
Sent: Thursday, July 01, 2004 2:32 PM
Subject: Re: Optimizing for long queries? >> 40% faster by changing
INDEX_INTERVAL
> A similar experiment with 500 shorter queries shows a 20% speed
improvement.
> (see xls file for details)
> By shorter query I mean something like that :
> ((titre:"burgundy wines"~3 titre:"burgundy wine"~3)) ((texte:"burgundy
> wines"~3^3.0 texte:"burgundy wine"~3^3.0)) ((descr:"burgundy wines"~3^4.0
> descr:"burgundy wine"~3^4.0)) ((kw:"burgundy wines"~3^4.0 kw:"burgundy
> wine"~3^4.0))
>
> ----- Original Message -----
>
> From: "Julien Nioche" <Ju...@lingway.com>
> To: "Lucene Developers List" <lu...@jakarta.apache.org>
> Cc: <dr...@gmail.com>
> Sent: Thursday, July 01, 2004 10:53 AM
> Subject: Re: Optimizing for long queries? >> 40% faster by changing
> INDEX_INTERVAL
>
>
> > I got a little bit deeper in my experiments with INDEX_INTERVAL. In a
> > previous mail to the user list I reported a 10% improvement over the
> regular
> > setting (128) with one of my application.
> > I refined the measures by taking the time spent not in the whole
> > application, but in a method that encapsulates Lucene searches. Only the
> > search time is measured, not the access to the Documents.
> >
> > Two sets of queries are generated using a log of user queries from our
> > application. Theses queries are in natural language and are expanded by
> our
> > product into a Lucene boolean query. Attached is the boolean generated
for
> > the query "Burgundy wine" - just to give you an idea of what I mean by
> large
> > query (this one is particularly big).
> >
> > These queries are used on an optimized index (INDEX_INTERVAL=16) and a
> > regular index. The index used for this test is 720 MB - FSDirectory on
> > Fedora 1 the .tii file is 3398 Kb in the modified version against 488Kb
in
> > the original. Both sets of queries have the same size (783). The xls
file
> > contains the times for both indexes sorted by decreasing order. Actually
> the
> > numbers indicates not a single search but a group of up to 4 searches.
> >
> > In average, changing the indexinterval to 16 yields an improvement of
> about
> > 40% compared to the regular setting.
> > I will try with a bigger sample of 40.000 queries and with smaller
queries
> > as well.
> >
> > The original motivation for this feature can be found at
> > http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg04092.html
> >
> > What is the best way to set up this value in IndexWriter? Maybe we could
> > limit to a few possible values like :
> > DEFAULT = 128
> > AVERAGE = 64
> > HIGH = 32
> > in order to avoid too low settings.
> >
> > Any comments or suggestions? Can anyone give feedback on this?
> >
> > Julien
> >
> >
> >
> > ----- Original Message -----
> > From: "Julien Nioche" <Ju...@lingway.com>
> > To: "Lucene Users List" <lu...@jakarta.apache.org>
> > Sent: Tuesday, June 29, 2004 3:03 PM
> > Subject: Re: Optimizing for long queries?
> >
> >
> > > I ran some tests changing TermInfosWriter.INDEX_INTERVAL to 16.
> > > On my application (which does a lot on top of lucene - including SQL
> > > transactions and so on) I won 10% percent time.
> > > I suppose this could be a bigger improvements in other applications,
> > because
> > > the search with Lucene is not 100% of my application.
> > >
> > > The index used for this test is 720 MB - FSDirectory on Fedora 1
> > > the .tii file is 3398 Kb in the modified version against 488Kb in the
> > > original (INDEX_INTERVAL=128)
> > >
> > > Has anyone tried changing this value? Do you get similar results?
> > >
> > > Julien
> > >
> > > ----- Original Message -----
> > > From: "Julien Nioche" <Ju...@lingway.com>
> > > To: "Lucene Users List" <lu...@jakarta.apache.org>
> > > Sent: Monday, June 28, 2004 10:04 AM
> > > Subject: Re: Optimizing for long queries?
> > >
> > >
> > > > Hello Drew,
> > > >
> > > > I don't think it's in the FAQ.
> > > >
> > > > 1 - What you could do is to sort your query terms by ascending
> > alphabetic
> > > > order. In my case it improved a little bit the performance. It could
> be
> > > > interesting to know how it worked in your case.
> > > >
> > > > 2- Another solution is to play with TermInfosWriter.INDEX_INTERVAL
at
> > > > indexation time. I quote Doug :
> > > >
> > > > "..., try reducing TermInfosWriter.INDEX_INTERVAL. You'll
> > > > have to re-create your indexes each time you change this constant.
> You
> > > > might try a value like 16. This would keep the number of terms in
> > > > memory from being too huge (1 of 16 terms), but would reduce the
> average
> > > > number scanned from 64 to 8, which would be substantial. Tell me
how
> > > > this works. If it makes a big difference, then perhaps we should
make
> > > > this parameter more easily changeable."
> > > >
> > > > Have you used a profiler on your application? This could be useful
to
> > spot
> > > > possible improvments.
> > > >
> > > >
> > > > ----- Original Message -----
> > > > From: "Drew Farris" <dr...@gmail.com>
> > > > To: <lu...@jakarta.apache.org>
> > > > Sent: Friday, June 25, 2004 8:24 PM
> > > > Subject: Optimizing for long queries?
> > > >
> > > >
> > > > > Apologies if this is a FAQ, but I didn't have much luck searching
> the
> > > > > list archives for answers on this subject:
> > > > >
> > > > > I'm using Lucene in a context where we have frequently have
queries
> > > > > that search for as many as 30-50 terms in a single field. Does
> anyone
> > > > > have any thoughts concerning ways optimize Lucene for queries of
> these
> > > > > lengths?
> > > > >
> > > >
> > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > > > > For additional commands, e-mail:
lucene-user-help@jakarta.apache.org
> > > > >
> > > > >
> > > >
> > > >
> > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > > > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> > > >
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> > >
> > >
> >
>
>
> --------------------------------------------------------------------------
--
> ----
>
>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>
----------------------------------------------------------------------------
----
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Optimizing for long queries? >> 40% faster by changing INDEX_INTERVAL
Posted by Julien Nioche <Ju...@lingway.com>.
A similar experiment with 500 shorter queries shows a 20% speed improvement.
(see xls file for details)
By shorter query I mean something like that :
((titre:"burgundy wines"~3 titre:"burgundy wine"~3)) ((texte:"burgundy
wines"~3^3.0 texte:"burgundy wine"~3^3.0)) ((descr:"burgundy wines"~3^4.0
descr:"burgundy wine"~3^4.0)) ((kw:"burgundy wines"~3^4.0 kw:"burgundy
wine"~3^4.0))
----- Original Message -----
From: "Julien Nioche" <Ju...@lingway.com>
To: "Lucene Developers List" <lu...@jakarta.apache.org>
Cc: <dr...@gmail.com>
Sent: Thursday, July 01, 2004 10:53 AM
Subject: Re: Optimizing for long queries? >> 40% faster by changing
INDEX_INTERVAL
> I got a little bit deeper in my experiments with INDEX_INTERVAL. In a
> previous mail to the user list I reported a 10% improvement over the
regular
> setting (128) with one of my application.
> I refined the measures by taking the time spent not in the whole
> application, but in a method that encapsulates Lucene searches. Only the
> search time is measured, not the access to the Documents.
>
> Two sets of queries are generated using a log of user queries from our
> application. Theses queries are in natural language and are expanded by
our
> product into a Lucene boolean query. Attached is the boolean generated for
> the query "Burgundy wine" - just to give you an idea of what I mean by
large
> query (this one is particularly big).
>
> These queries are used on an optimized index (INDEX_INTERVAL=16) and a
> regular index. The index used for this test is 720 MB - FSDirectory on
> Fedora 1 the .tii file is 3398 Kb in the modified version against 488Kb in
> the original. Both sets of queries have the same size (783). The xls file
> contains the times for both indexes sorted by decreasing order. Actually
the
> numbers indicates not a single search but a group of up to 4 searches.
>
> In average, changing the indexinterval to 16 yields an improvement of
about
> 40% compared to the regular setting.
> I will try with a bigger sample of 40.000 queries and with smaller queries
> as well.
>
> The original motivation for this feature can be found at
> http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg04092.html
>
> What is the best way to set up this value in IndexWriter? Maybe we could
> limit to a few possible values like :
> DEFAULT = 128
> AVERAGE = 64
> HIGH = 32
> in order to avoid too low settings.
>
> Any comments or suggestions? Can anyone give feedback on this?
>
> Julien
>
>
>
> ----- Original Message -----
> From: "Julien Nioche" <Ju...@lingway.com>
> To: "Lucene Users List" <lu...@jakarta.apache.org>
> Sent: Tuesday, June 29, 2004 3:03 PM
> Subject: Re: Optimizing for long queries?
>
>
> > I ran some tests changing TermInfosWriter.INDEX_INTERVAL to 16.
> > On my application (which does a lot on top of lucene - including SQL
> > transactions and so on) I won 10% percent time.
> > I suppose this could be a bigger improvements in other applications,
> because
> > the search with Lucene is not 100% of my application.
> >
> > The index used for this test is 720 MB - FSDirectory on Fedora 1
> > the .tii file is 3398 Kb in the modified version against 488Kb in the
> > original (INDEX_INTERVAL=128)
> >
> > Has anyone tried changing this value? Do you get similar results?
> >
> > Julien
> >
> > ----- Original Message -----
> > From: "Julien Nioche" <Ju...@lingway.com>
> > To: "Lucene Users List" <lu...@jakarta.apache.org>
> > Sent: Monday, June 28, 2004 10:04 AM
> > Subject: Re: Optimizing for long queries?
> >
> >
> > > Hello Drew,
> > >
> > > I don't think it's in the FAQ.
> > >
> > > 1 - What you could do is to sort your query terms by ascending
> alphabetic
> > > order. In my case it improved a little bit the performance. It could
be
> > > interesting to know how it worked in your case.
> > >
> > > 2- Another solution is to play with TermInfosWriter.INDEX_INTERVAL at
> > > indexation time. I quote Doug :
> > >
> > > "..., try reducing TermInfosWriter.INDEX_INTERVAL. You'll
> > > have to re-create your indexes each time you change this constant.
You
> > > might try a value like 16. This would keep the number of terms in
> > > memory from being too huge (1 of 16 terms), but would reduce the
average
> > > number scanned from 64 to 8, which would be substantial. Tell me how
> > > this works. If it makes a big difference, then perhaps we should make
> > > this parameter more easily changeable."
> > >
> > > Have you used a profiler on your application? This could be useful to
> spot
> > > possible improvments.
> > >
> > >
> > > ----- Original Message -----
> > > From: "Drew Farris" <dr...@gmail.com>
> > > To: <lu...@jakarta.apache.org>
> > > Sent: Friday, June 25, 2004 8:24 PM
> > > Subject: Optimizing for long queries?
> > >
> > >
> > > > Apologies if this is a FAQ, but I didn't have much luck searching
the
> > > > list archives for answers on this subject:
> > > >
> > > > I'm using Lucene in a context where we have frequently have queries
> > > > that search for as many as 30-50 terms in a single field. Does
anyone
> > > > have any thoughts concerning ways optimize Lucene for queries of
these
> > > > lengths?
> > > >
> > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > > > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> > > >
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> >
> >
>
----------------------------------------------------------------------------
----
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Optimizing for long queries? >> 40% faster by changing INDEX_INTERVAL
Posted by Julien Nioche <Ju...@lingway.com>.
I got a little bit deeper in my experiments with INDEX_INTERVAL. In a
previous mail to the user list I reported a 10% improvement over the regular
setting (128) with one of my application.
I refined the measures by taking the time spent not in the whole
application, but in a method that encapsulates Lucene searches. Only the
search time is measured, not the access to the Documents.
Two sets of queries are generated using a log of user queries from our
application. Theses queries are in natural language and are expanded by our
product into a Lucene boolean query. Attached is the boolean generated for
the query "Burgundy wine" - just to give you an idea of what I mean by large
query (this one is particularly big).
These queries are used on an optimized index (INDEX_INTERVAL=16) and a
regular index. The index used for this test is 720 MB - FSDirectory on
Fedora 1 the .tii file is 3398 Kb in the modified version against 488Kb in
the original. Both sets of queries have the same size (783). The xls file
contains the times for both indexes sorted by decreasing order. Actually the
numbers indicates not a single search but a group of up to 4 searches.
In average, changing the indexinterval to 16 yields an improvement of about
40% compared to the regular setting.
I will try with a bigger sample of 40.000 queries and with smaller queries
as well.
The original motivation for this feature can be found at
http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg04092.html
What is the best way to set up this value in IndexWriter? Maybe we could
limit to a few possible values like :
DEFAULT = 128
AVERAGE = 64
HIGH = 32
in order to avoid too low settings.
Any comments or suggestions? Can anyone give feedback on this?
Julien
----- Original Message -----
From: "Julien Nioche" <Ju...@lingway.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Tuesday, June 29, 2004 3:03 PM
Subject: Re: Optimizing for long queries?
> I ran some tests changing TermInfosWriter.INDEX_INTERVAL to 16.
> On my application (which does a lot on top of lucene - including SQL
> transactions and so on) I won 10% percent time.
> I suppose this could be a bigger improvements in other applications,
because
> the search with Lucene is not 100% of my application.
>
> The index used for this test is 720 MB - FSDirectory on Fedora 1
> the .tii file is 3398 Kb in the modified version against 488Kb in the
> original (INDEX_INTERVAL=128)
>
> Has anyone tried changing this value? Do you get similar results?
>
> Julien
>
> ----- Original Message -----
> From: "Julien Nioche" <Ju...@lingway.com>
> To: "Lucene Users List" <lu...@jakarta.apache.org>
> Sent: Monday, June 28, 2004 10:04 AM
> Subject: Re: Optimizing for long queries?
>
>
> > Hello Drew,
> >
> > I don't think it's in the FAQ.
> >
> > 1 - What you could do is to sort your query terms by ascending
alphabetic
> > order. In my case it improved a little bit the performance. It could be
> > interesting to know how it worked in your case.
> >
> > 2- Another solution is to play with TermInfosWriter.INDEX_INTERVAL at
> > indexation time. I quote Doug :
> >
> > "..., try reducing TermInfosWriter.INDEX_INTERVAL. You'll
> > have to re-create your indexes each time you change this constant. You
> > might try a value like 16. This would keep the number of terms in
> > memory from being too huge (1 of 16 terms), but would reduce the average
> > number scanned from 64 to 8, which would be substantial. Tell me how
> > this works. If it makes a big difference, then perhaps we should make
> > this parameter more easily changeable."
> >
> > Have you used a profiler on your application? This could be useful to
spot
> > possible improvments.
> >
> >
> > ----- Original Message -----
> > From: "Drew Farris" <dr...@gmail.com>
> > To: <lu...@jakarta.apache.org>
> > Sent: Friday, June 25, 2004 8:24 PM
> > Subject: Optimizing for long queries?
> >
> >
> > > Apologies if this is a FAQ, but I didn't have much luck searching the
> > > list archives for answers on this subject:
> > >
> > > I'm using Lucene in a context where we have frequently have queries
> > > that search for as many as 30-50 terms in a single field. Does anyone
> > > have any thoughts concerning ways optimize Lucene for queries of these
> > > lengths?
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
Re: Optimizing for long queries?
Posted by Julien Nioche <Ju...@lingway.com>.
I ran some tests changing TermInfosWriter.INDEX_INTERVAL to 16.
On my application (which does a lot on top of lucene - including SQL
transactions and so on) I won 10% percent time.
I suppose this could be a bigger improvements in other applications, because
the search with Lucene is not 100% of my application.
The index used for this test is 720 MB - FSDirectory on Fedora 1
the .tii file is 3398 Kb in the modified version against 488Kb in the
original (INDEX_INTERVAL=128)
Has anyone tried changing this value? Do you get similar results?
Julien
----- Original Message -----
From: "Julien Nioche" <Ju...@lingway.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Monday, June 28, 2004 10:04 AM
Subject: Re: Optimizing for long queries?
> Hello Drew,
>
> I don't think it's in the FAQ.
>
> 1 - What you could do is to sort your query terms by ascending alphabetic
> order. In my case it improved a little bit the performance. It could be
> interesting to know how it worked in your case.
>
> 2- Another solution is to play with TermInfosWriter.INDEX_INTERVAL at
> indexation time. I quote Doug :
>
> "..., try reducing TermInfosWriter.INDEX_INTERVAL. You'll
> have to re-create your indexes each time you change this constant. You
> might try a value like 16. This would keep the number of terms in
> memory from being too huge (1 of 16 terms), but would reduce the average
> number scanned from 64 to 8, which would be substantial. Tell me how
> this works. If it makes a big difference, then perhaps we should make
> this parameter more easily changeable."
>
> Have you used a profiler on your application? This could be useful to spot
> possible improvments.
>
>
> ----- Original Message -----
> From: "Drew Farris" <dr...@gmail.com>
> To: <lu...@jakarta.apache.org>
> Sent: Friday, June 25, 2004 8:24 PM
> Subject: Optimizing for long queries?
>
>
> > Apologies if this is a FAQ, but I didn't have much luck searching the
> > list archives for answers on this subject:
> >
> > I'm using Lucene in a context where we have frequently have queries
> > that search for as many as 30-50 terms in a single field. Does anyone
> > have any thoughts concerning ways optimize Lucene for queries of these
> > lengths?
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Optimizing for long queries?
Posted by Julien Nioche <Ju...@lingway.com>.
Hello Drew,
I don't think it's in the FAQ.
1 - What you could do is to sort your query terms by ascending alphabetic
order. In my case it improved a little bit the performance. It could be
interesting to know how it worked in your case.
2- Another solution is to play with TermInfosWriter.INDEX_INTERVAL at
indexation time. I quote Doug :
"..., try reducing TermInfosWriter.INDEX_INTERVAL. You'll
have to re-create your indexes each time you change this constant. You
might try a value like 16. This would keep the number of terms in
memory from being too huge (1 of 16 terms), but would reduce the average
number scanned from 64 to 8, which would be substantial. Tell me how
this works. If it makes a big difference, then perhaps we should make
this parameter more easily changeable."
Have you used a profiler on your application? This could be useful to spot
possible improvments.
----- Original Message -----
From: "Drew Farris" <dr...@gmail.com>
To: <lu...@jakarta.apache.org>
Sent: Friday, June 25, 2004 8:24 PM
Subject: Optimizing for long queries?
> Apologies if this is a FAQ, but I didn't have much luck searching the
> list archives for answers on this subject:
>
> I'm using Lucene in a context where we have frequently have queries
> that search for as many as 30-50 terms in a single field. Does anyone
> have any thoughts concerning ways optimize Lucene for queries of these
> lengths?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org