You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Michel Nadeau <ak...@gmail.com> on 2009/11/30 16:47:48 UTC
Performance problems with Lucene 2.9
Hi,
we use Lucene to store around 300 millions of records. We use the index both
for conventional searching, but also for all the system's data - we replaced
MySQL with Lucene because it was simply not working at all with MySQL due to
the amount or records. Our problem is that we have HUGE performance
problems... whenever we search, it takes forever to return results, and Java
uses 100% CPU/RAM.
Our index fields are like this:
TYPE
PK
FOREIGN_PK
TAG
...other information depending on type...
* All fields are Field.Index.UN_TOKENIZED
* The field "TAG" always contains the value "TAG".
Whenever we search in the index, our query is "TAG:TAG" to match all
documents, and we do the search like this:
// Search
Hits h = searcher.search(q, cluCF, cluSort);
cluCF is a ChainedFilter containing all the other filters (like
FOREIGN_PK=12345, TYPE=a, etc.).
I know that the method is probably crazy because "TAG:TAG" is matching all
300M documents and then it applies filters; so that's probably why every
little query is taking 100% CPU/RAM.... but I don't know how to do it
properly.
Help ! Any advice is welcome.
- Mike
akaris@gmail.com
Re: Performance problems with Lucene 2.9
Posted by Michel Nadeau <ak...@gmail.com>.
Great, thanks!
So what do you guys think would be the best road for my application? I NEVER
want to retrieve -all- documents, only like maximum 200. I always need to
apply some filters and some sorts. From what I understand, in all cases I
should switch from Hits to a Collector for performance reasons - but then
how will I filter and sort?
- Mike
akaris@gmail.com
On Mon, Nov 30, 2009 at 11:20 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hits is deprecated and should no longer be used. The replacements are
> TopDocs *or* Collectors.
>
> If you add a number of max-scoring results you want to have (e.g. to
> display
> the first 10 results of a google-like query on a web page), use TopDocs.
> The
> method for that is Searcher.serach(Query q, int n) (or optionally with
> filter). This will only return the most-relevant results (n hits). This is
> for normal web pages that work like google. You cannot set n to a very high
> number and try to get all results, it would use to much memory (its only to
> collect most relevant docs). Hits was for the same purpose, but was build
> like an random-access result, which it not was, confusing users like you.
> Because of that incorrect usage, Hits was very slow if you try to get all
> hits and consumes all memory.
>
> If you need all results, you have to write a Collector, which is an
> abstract
> class. The search engine then calls collect() for each hit together with
> the
> document number. In your implementation of collect() you can consume the
> document ID and optionally retrieve the score. There are more methods to
> override in Collector, sample implementations are given in JavaDocs. You
> also need to take care of IndexReaders and document base ids.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Michel Nadeau [mailto:akaris@gmail.com]
> > Sent: Monday, November 30, 2009 5:10 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Performance problems with Lucene 2.9
> >
> > What is the main difference between Hits and Collectors?
> >
> > - Mike
> > akaris@gmail.com
> >
> >
> > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> > > And if you only have a filter and apply it to all documents, make a
> > > ConstantScoreQuery on top of the filter:
> > >
> > > Query q=new ConstantScoreQuery(cluCF);
> > >
> > > Then remove the filter from your search method call and only execute
> > this
> > > query.
> > >
> > > And if you iterate over all results never-ever use Hits! (its already
> > > deprecated). Write a Collector instead (as you are not interested in
> > > scoring).
> > >
> > > And: If you replace a relational database with Lucene, be sure not to
> > think
> > > in a relational sense with foreign keys / primary keys and so on. In
> > > general
> > > you should flatten everything.
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > > > -----Original Message-----
> > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > Sent: Monday, November 30, 2009 4:56 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: Performance problems with Lucene 2.9
> > > >
> > > > Hi
> > > >
> > > > First you can use MatchAllDocsQuery, which matches all documents. It
> > will
> > > > save a HUGE posting list (TAG:TAG), and performs much faster. For
> > example
> > > > TAG:TAG computes a score for each doc, even though you don't need it.
> > > > MatchAllDocsQuery doesn't.
> > > >
> > > > Second, move away from Hits ! :) Use Collectors instead.
> > > >
> > > > If I understand the chain of filters, do you think you can code them
> > with
> > > > a
> > > > BooleanQuery that is added BooleanClauses, each with is Term
> > > > (field:value)?
> > > > You can add clauses w/ OR, AND, NOT etc.
> > > >
> > > > Note that in Lucene 2.9, you can avoid scoring documents very easily,
> > > > which
> > > > is a performance win if you don't need scores (i.e. if you just want
> > to
> > > > match everything, not caring for scores).
> > > >
> > > > Shai
> > > >
> > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > we use Lucene to store around 300 millions of records. We use the
> > index
> > > > > both
> > > > > for conventional searching, but also for all the system's data - we
> > > > > replaced
> > > > > MySQL with Lucene because it was simply not working at all with
> > MySQL
> > > > due
> > > > > to
> > > > > the amount or records. Our problem is that we have HUGE performance
> > > > > problems... whenever we search, it takes forever to return results,
> > and
> > > > > Java
> > > > > uses 100% CPU/RAM.
> > > > >
> > > > > Our index fields are like this:
> > > > >
> > > > > TYPE
> > > > > PK
> > > > > FOREIGN_PK
> > > > > TAG
> > > > > ...other information depending on type...
> > > > >
> > > > > * All fields are Field.Index.UN_TOKENIZED
> > > > > * The field "TAG" always contains the value "TAG".
> > > > >
> > > > > Whenever we search in the index, our query is "TAG:TAG" to match
> all
> > > > > documents, and we do the search like this:
> > > > >
> > > > > // Search
> > > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > > >
> > > > > cluCF is a ChainedFilter containing all the other filters (like
> > > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > > >
> > > > > I know that the method is probably crazy because "TAG:TAG" is
> > matching
> > > > all
> > > > > 300M documents and then it applies filters; so that's probably why
> > > every
> > > > > little query is taking 100% CPU/RAM.... but I don't know how to do
> > it
> > > > > properly.
> > > > >
> > > > > Help ! Any advice is welcome.
> > > > >
> > > > > - Mike
> > > > > akaris@gmail.com
> > > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Performance problems with Lucene 2.9
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hits is deprecated and should no longer be used. The replacements are
TopDocs *or* Collectors.
If you add a number of max-scoring results you want to have (e.g. to display
the first 10 results of a google-like query on a web page), use TopDocs. The
method for that is Searcher.serach(Query q, int n) (or optionally with
filter). This will only return the most-relevant results (n hits). This is
for normal web pages that work like google. You cannot set n to a very high
number and try to get all results, it would use to much memory (its only to
collect most relevant docs). Hits was for the same purpose, but was build
like an random-access result, which it not was, confusing users like you.
Because of that incorrect usage, Hits was very slow if you try to get all
hits and consumes all memory.
If you need all results, you have to write a Collector, which is an abstract
class. The search engine then calls collect() for each hit together with the
document number. In your implementation of collect() you can consume the
document ID and optionally retrieve the score. There are more methods to
override in Collector, sample implementations are given in JavaDocs. You
also need to take care of IndexReaders and document base ids.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michel Nadeau [mailto:akaris@gmail.com]
> Sent: Monday, November 30, 2009 5:10 PM
> To: java-user@lucene.apache.org
> Subject: Re: Performance problems with Lucene 2.9
>
> What is the main difference between Hits and Collectors?
>
> - Mike
> akaris@gmail.com
>
>
> On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > And if you only have a filter and apply it to all documents, make a
> > ConstantScoreQuery on top of the filter:
> >
> > Query q=new ConstantScoreQuery(cluCF);
> >
> > Then remove the filter from your search method call and only execute
> this
> > query.
> >
> > And if you iterate over all results never-ever use Hits! (its already
> > deprecated). Write a Collector instead (as you are not interested in
> > scoring).
> >
> > And: If you replace a relational database with Lucene, be sure not to
> think
> > in a relational sense with foreign keys / primary keys and so on. In
> > general
> > you should flatten everything.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Shai Erera [mailto:serera@gmail.com]
> > > Sent: Monday, November 30, 2009 4:56 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: Performance problems with Lucene 2.9
> > >
> > > Hi
> > >
> > > First you can use MatchAllDocsQuery, which matches all documents. It
> will
> > > save a HUGE posting list (TAG:TAG), and performs much faster. For
> example
> > > TAG:TAG computes a score for each doc, even though you don't need it.
> > > MatchAllDocsQuery doesn't.
> > >
> > > Second, move away from Hits ! :) Use Collectors instead.
> > >
> > > If I understand the chain of filters, do you think you can code them
> with
> > > a
> > > BooleanQuery that is added BooleanClauses, each with is Term
> > > (field:value)?
> > > You can add clauses w/ OR, AND, NOT etc.
> > >
> > > Note that in Lucene 2.9, you can avoid scoring documents very easily,
> > > which
> > > is a performance win if you don't need scores (i.e. if you just want
> to
> > > match everything, not caring for scores).
> > >
> > > Shai
> > >
> > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > we use Lucene to store around 300 millions of records. We use the
> index
> > > > both
> > > > for conventional searching, but also for all the system's data - we
> > > > replaced
> > > > MySQL with Lucene because it was simply not working at all with
> MySQL
> > > due
> > > > to
> > > > the amount or records. Our problem is that we have HUGE performance
> > > > problems... whenever we search, it takes forever to return results,
> and
> > > > Java
> > > > uses 100% CPU/RAM.
> > > >
> > > > Our index fields are like this:
> > > >
> > > > TYPE
> > > > PK
> > > > FOREIGN_PK
> > > > TAG
> > > > ...other information depending on type...
> > > >
> > > > * All fields are Field.Index.UN_TOKENIZED
> > > > * The field "TAG" always contains the value "TAG".
> > > >
> > > > Whenever we search in the index, our query is "TAG:TAG" to match all
> > > > documents, and we do the search like this:
> > > >
> > > > // Search
> > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > >
> > > > cluCF is a ChainedFilter containing all the other filters (like
> > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > >
> > > > I know that the method is probably crazy because "TAG:TAG" is
> matching
> > > all
> > > > 300M documents and then it applies filters; so that's probably why
> > every
> > > > little query is taking 100% CPU/RAM.... but I don't know how to do
> it
> > > > properly.
> > > >
> > > > Help ! Any advice is welcome.
> > > >
> > > > - Mike
> > > > akaris@gmail.com
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Performance problems with Lucene 2.9
Posted by Uwe Schindler <uw...@thetaphi.de>.
The total number is also returned in Top(Field)Docs, there is a getter
method.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michel Nadeau [mailto:akaris@gmail.com]
> Sent: Monday, November 30, 2009 6:37 PM
> To: java-user@lucene.apache.org
> Subject: Re: Performance problems with Lucene 2.9
>
> The problem with this method is that I won't be able to know how many
> total
> results / pages a search have?
>
> For example if I do a search X that returns 1,000,000 records, so 5,000
> pages of 200 items, I will only know if I have more when I'll hit "next
> page" - I won't be able to display "1,000,000 results / Page 1 of 5,000" ?
>
> - Mike
> akaris@gmail.com
>
>
> On Mon, Nov 30, 2009 at 12:28 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > > Now I have another question... is there a way to specify a "start
> from"
> > so
> > > I
> > > could get page 2, 3, 4, etc.. ?
> >
> > Search the mailing list, this was explained quite often (by others and
> me).
> > The trick is:
> > If you have 200 results per page, with n = 200 you get the top ranking
> > results for the first page. If you want to have the second page,
> reexecute
> > the query with n=400 and display scoreDocs[200..399] and so on. All full
> > text engines work like this, they can only collect top-mathcing
> documents,
> > but no documents in a slot. If n gets too big it gets very slow and uses
> > too
> > much memory. Because of that, Google limits the maximum page number.
> >
> >
> > > On Mon, Nov 30, 2009 at 12:00 PM, Uwe Schindler <uw...@thetaphi.de>
> wrote:
> > >
> > > > > And sorting is done by the
> > > > > collector, Lucene has no idea how to sort.
> > > >
> > > > Sorting is done by the internal collector behind the
> > > > Top(Field)Docs-returning method (your own collectors would have to
> do
> > it
> > > > themselves). If you call search(Query, n,... Sort), internally an
> > > collector
> > > > is created that does the sorting for you and throws away all results
> > > that
> > > > do
> > > > not fall into the first 200 hits (if n=200).
> > > >
> > > > > If you use Sort, the returned
> > > > > TopDocs will be sorted.
> > > > >
> > > > > If you do not sort at all and do not score your results, TopDocs
> is
> > > not
> > > > > very
> > > > > useful, because the first 200 hits cannot be ranked.
> > > > >
> > > > > -----
> > > > > Uwe Schindler
> > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > http://www.thetaphi.de
> > > > > eMail: uwe@thetaphi.de
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Michel Nadeau [mailto:akaris@gmail.com]
> > > > > > Sent: Monday, November 30, 2009 5:35 PM
> > > > > > To: java-user@lucene.apache.org
> > > > > > Subject: Re: Performance problems with Lucene 2.9
> > > > > >
> > > > > > I'll definitely switch to a Collector.
> > > > > >
> > > > > > It's just not clear for me if I should use BooleanQueries or
> > > > > > MatchAllDocuments+Filters ?
> > > > > >
> > > > > > And should I write my own collector or the TopDocs one is
> perfect
> > > for
> > > > me
> > > > > ?
> > > > > >
> > > > > > - Mike
> > > > > > akaris@gmail.com
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson
> > > > > > <er...@gmail.com>wrote:
> > > > > >
> > > > > > > The problem with hits is that a it re-executes the query
> > > > > > > every N documents where N is 100 (?).
> > > > > > >
> > > > > > > So, a loop like
> > > > > > > for (int idx : hits.length) {
> > > > > > > do something....
> > > > > > > }
> > > > > > >
> > > > > > > Assuming my memory is right and it's every 100, your query
> will
> > > > > > > re-execute (length/100) times. Which is unfortunate.
> > > > > > >
> > > > > > > The very quick test to see where to concentrate first would be
> to
> > > > take
> > > > > > > a time stamp just before you hit your loop.....
> > > > > > >
> > > > > > > This will tell you whether this loop is the culprit, but it
> > really
> > > > > > doesn't
> > > > > > > matter because you'll follow the advice from Uwe and Shai
> anyway
> > > <G>.
> > > > > > >
> > > > > > > Filtering and Sorting are applied to Collectors before you see
> > > > > them.....
> > > > > > >
> > > > > > > The other bit would be to investigate your sorting. Remember
> that
> > > the
> > > > > > > first sort or two take quite a while since the relevant caches
> > are
> > > > > > > populated with first used, so second+ queries should be
> faster.
> > > The
> > > > > > > Wiki has some timing/speedup advice.....
> > > > > > >
> > > > > > > Best
> > > > > > > Erick
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <
> > akaris@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > What is the main difference between Hits and Collectors?
> > > > > > > >
> > > > > > > > - Mike
> > > > > > > > akaris@gmail.com
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler
> > > <uw...@thetaphi.de>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > And if you only have a filter and apply it to all
> documents,
> > > make
> > > > > a
> > > > > > > > > ConstantScoreQuery on top of the filter:
> > > > > > > > >
> > > > > > > > > Query q=new ConstantScoreQuery(cluCF);
> > > > > > > > >
> > > > > > > > > Then remove the filter from your search method call and
> only
> > > > > execute
> > > > > > > this
> > > > > > > > > query.
> > > > > > > > >
> > > > > > > > > And if you iterate over all results never-ever use Hits!
> (its
> > > > > > already
> > > > > > > > > deprecated). Write a Collector instead (as you are not
> > > interested
> > > > > in
> > > > > > > > > scoring).
> > > > > > > > >
> > > > > > > > > And: If you replace a relational database with Lucene, be
> > sure
> > > > not
> > > > > > to
> > > > > > > > think
> > > > > > > > > in a relational sense with foreign keys / primary keys and
> so
> > > on.
> > > > > In
> > > > > > > > > general
> > > > > > > > > you should flatten everything.
> > > > > > > > >
> > > > > > > > > Uwe
> > > > > > > > >
> > > > > > > > > -----
> > > > > > > > > Uwe Schindler
> > > > > > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > > > > > http://www.thetaphi.de
> > > > > > > > > eMail: uwe@thetaphi.de
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > > > > > > Sent: Monday, November 30, 2009 4:56 PM
> > > > > > > > > > To: java-user@lucene.apache.org
> > > > > > > > > > Subject: Re: Performance problems with Lucene 2.9
> > > > > > > > > >
> > > > > > > > > > Hi
> > > > > > > > > >
> > > > > > > > > > First you can use MatchAllDocsQuery, which matches all
> > > > > documents.
> > > > > > It
> > > > > > > > will
> > > > > > > > > > save a HUGE posting list (TAG:TAG), and performs much
> > > faster.
> > > > > For
> > > > > > > > example
> > > > > > > > > > TAG:TAG computes a score for each doc, even though you
> > don't
> > > > > need
> > > > > > it.
> > > > > > > > > > MatchAllDocsQuery doesn't.
> > > > > > > > > >
> > > > > > > > > > Second, move away from Hits ! :) Use Collectors instead.
> > > > > > > > > >
> > > > > > > > > > If I understand the chain of filters, do you think you
> can
> > > code
> > > > > > them
> > > > > > > > with
> > > > > > > > > > a
> > > > > > > > > > BooleanQuery that is added BooleanClauses, each with is
> > Term
> > > > > > > > > > (field:value)?
> > > > > > > > > > You can add clauses w/ OR, AND, NOT etc.
> > > > > > > > > >
> > > > > > > > > > Note that in Lucene 2.9, you can avoid scoring documents
> > > very
> > > > > > easily,
> > > > > > > > > > which
> > > > > > > > > > is a performance win if you don't need scores (i.e. if
> you
> > > just
> > > > > > want
> > > > > > > to
> > > > > > > > > > match everything, not caring for scores).
> > > > > > > > > >
> > > > > > > > > > Shai
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau
> > > > > <ak...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > we use Lucene to store around 300 millions of records.
> We
> > > use
> > > > > > the
> > > > > > > > index
> > > > > > > > > > > both
> > > > > > > > > > > for conventional searching, but also for all the
> system's
> > > > data
> > > > > -
> > > > > > we
> > > > > > > > > > > replaced
> > > > > > > > > > > MySQL with Lucene because it was simply not working at
> > all
> > > > > with
> > > > > > > MySQL
> > > > > > > > > > due
> > > > > > > > > > > to
> > > > > > > > > > > the amount or records. Our problem is that we have
> HUGE
> > > > > > performance
> > > > > > > > > > > problems... whenever we search, it takes forever to
> > return
> > > > > > results,
> > > > > > > > and
> > > > > > > > > > > Java
> > > > > > > > > > > uses 100% CPU/RAM.
> > > > > > > > > > >
> > > > > > > > > > > Our index fields are like this:
> > > > > > > > > > >
> > > > > > > > > > > TYPE
> > > > > > > > > > > PK
> > > > > > > > > > > FOREIGN_PK
> > > > > > > > > > > TAG
> > > > > > > > > > > ...other information depending on type...
> > > > > > > > > > >
> > > > > > > > > > > * All fields are Field.Index.UN_TOKENIZED
> > > > > > > > > > > * The field "TAG" always contains the value "TAG".
> > > > > > > > > > >
> > > > > > > > > > > Whenever we search in the index, our query is
> "TAG:TAG"
> > to
> > > > > match
> > > > > > > all
> > > > > > > > > > > documents, and we do the search like this:
> > > > > > > > > > >
> > > > > > > > > > > // Search
> > > > > > > > > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > > > > > > > > >
> > > > > > > > > > > cluCF is a ChainedFilter containing all the other
> filters
> > > > > (like
> > > > > > > > > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > > > > > > > > >
> > > > > > > > > > > I know that the method is probably crazy because
> > "TAG:TAG"
> > > is
> > > > > > > > matching
> > > > > > > > > > all
> > > > > > > > > > > 300M documents and then it applies filters; so that's
> > > > probably
> > > > > > why
> > > > > > > > > every
> > > > > > > > > > > little query is taking 100% CPU/RAM.... but I don't
> know
> > > how
> > > > > to
> > > > > > do
> > > > > > > it
> > > > > > > > > > > properly.
> > > > > > > > > > >
> > > > > > > > > > > Help ! Any advice is welcome.
> > > > > > > > > > >
> > > > > > > > > > > - Mike
> > > > > > > > > > > akaris@gmail.com
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > ------------------------------------------------------------------
> > > > > --
> > > > > > -
> > > > > > > > > To unsubscribe, e-mail: java-user-
> > > unsubscribe@lucene.apache.org
> > > > > > > > > For additional commands, e-mail:
> > > > java-user-help@lucene.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > > ------------------------------------------------------------------
> ---
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > > >
> > > > --------------------------------------------------------------------
> -
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Performance problems with Lucene 2.9
Posted by Michel Nadeau <ak...@gmail.com>.
The problem with this method is that I won't be able to know how many total
results / pages a search have?
For example if I do a search X that returns 1,000,000 records, so 5,000
pages of 200 items, I will only know if I have more when I'll hit "next
page" - I won't be able to display "1,000,000 results / Page 1 of 5,000" ?
- Mike
akaris@gmail.com
On Mon, Nov 30, 2009 at 12:28 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> > Now I have another question... is there a way to specify a "start from"
> so
> > I
> > could get page 2, 3, 4, etc.. ?
>
> Search the mailing list, this was explained quite often (by others and me).
> The trick is:
> If you have 200 results per page, with n = 200 you get the top ranking
> results for the first page. If you want to have the second page, reexecute
> the query with n=400 and display scoreDocs[200..399] and so on. All full
> text engines work like this, they can only collect top-mathcing documents,
> but no documents in a slot. If n gets too big it gets very slow and uses
> too
> much memory. Because of that, Google limits the maximum page number.
>
>
> > On Mon, Nov 30, 2009 at 12:00 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> > > > And sorting is done by the
> > > > collector, Lucene has no idea how to sort.
> > >
> > > Sorting is done by the internal collector behind the
> > > Top(Field)Docs-returning method (your own collectors would have to do
> it
> > > themselves). If you call search(Query, n,... Sort), internally an
> > collector
> > > is created that does the sorting for you and throws away all results
> > that
> > > do
> > > not fall into the first 200 hits (if n=200).
> > >
> > > > If you use Sort, the returned
> > > > TopDocs will be sorted.
> > > >
> > > > If you do not sort at all and do not score your results, TopDocs is
> > not
> > > > very
> > > > useful, because the first 200 hits cannot be ranked.
> > > >
> > > > -----
> > > > Uwe Schindler
> > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > http://www.thetaphi.de
> > > > eMail: uwe@thetaphi.de
> > > >
> > > > > -----Original Message-----
> > > > > From: Michel Nadeau [mailto:akaris@gmail.com]
> > > > > Sent: Monday, November 30, 2009 5:35 PM
> > > > > To: java-user@lucene.apache.org
> > > > > Subject: Re: Performance problems with Lucene 2.9
> > > > >
> > > > > I'll definitely switch to a Collector.
> > > > >
> > > > > It's just not clear for me if I should use BooleanQueries or
> > > > > MatchAllDocuments+Filters ?
> > > > >
> > > > > And should I write my own collector or the TopDocs one is perfect
> > for
> > > me
> > > > ?
> > > > >
> > > > > - Mike
> > > > > akaris@gmail.com
> > > > >
> > > > >
> > > > > On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson
> > > > > <er...@gmail.com>wrote:
> > > > >
> > > > > > The problem with hits is that a it re-executes the query
> > > > > > every N documents where N is 100 (?).
> > > > > >
> > > > > > So, a loop like
> > > > > > for (int idx : hits.length) {
> > > > > > do something....
> > > > > > }
> > > > > >
> > > > > > Assuming my memory is right and it's every 100, your query will
> > > > > > re-execute (length/100) times. Which is unfortunate.
> > > > > >
> > > > > > The very quick test to see where to concentrate first would be to
> > > take
> > > > > > a time stamp just before you hit your loop.....
> > > > > >
> > > > > > This will tell you whether this loop is the culprit, but it
> really
> > > > > doesn't
> > > > > > matter because you'll follow the advice from Uwe and Shai anyway
> > <G>.
> > > > > >
> > > > > > Filtering and Sorting are applied to Collectors before you see
> > > > them.....
> > > > > >
> > > > > > The other bit would be to investigate your sorting. Remember that
> > the
> > > > > > first sort or two take quite a while since the relevant caches
> are
> > > > > > populated with first used, so second+ queries should be faster.
> > The
> > > > > > Wiki has some timing/speedup advice.....
> > > > > >
> > > > > > Best
> > > > > > Erick
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <
> akaris@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > What is the main difference between Hits and Collectors?
> > > > > > >
> > > > > > > - Mike
> > > > > > > akaris@gmail.com
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler
> > <uw...@thetaphi.de>
> > > > > wrote:
> > > > > > >
> > > > > > > > And if you only have a filter and apply it to all documents,
> > make
> > > > a
> > > > > > > > ConstantScoreQuery on top of the filter:
> > > > > > > >
> > > > > > > > Query q=new ConstantScoreQuery(cluCF);
> > > > > > > >
> > > > > > > > Then remove the filter from your search method call and only
> > > > execute
> > > > > > this
> > > > > > > > query.
> > > > > > > >
> > > > > > > > And if you iterate over all results never-ever use Hits! (its
> > > > > already
> > > > > > > > deprecated). Write a Collector instead (as you are not
> > interested
> > > > in
> > > > > > > > scoring).
> > > > > > > >
> > > > > > > > And: If you replace a relational database with Lucene, be
> sure
> > > not
> > > > > to
> > > > > > > think
> > > > > > > > in a relational sense with foreign keys / primary keys and so
> > on.
> > > > In
> > > > > > > > general
> > > > > > > > you should flatten everything.
> > > > > > > >
> > > > > > > > Uwe
> > > > > > > >
> > > > > > > > -----
> > > > > > > > Uwe Schindler
> > > > > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > > > > http://www.thetaphi.de
> > > > > > > > eMail: uwe@thetaphi.de
> > > > > > > >
> > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > > > > > Sent: Monday, November 30, 2009 4:56 PM
> > > > > > > > > To: java-user@lucene.apache.org
> > > > > > > > > Subject: Re: Performance problems with Lucene 2.9
> > > > > > > > >
> > > > > > > > > Hi
> > > > > > > > >
> > > > > > > > > First you can use MatchAllDocsQuery, which matches all
> > > > documents.
> > > > > It
> > > > > > > will
> > > > > > > > > save a HUGE posting list (TAG:TAG), and performs much
> > faster.
> > > > For
> > > > > > > example
> > > > > > > > > TAG:TAG computes a score for each doc, even though you
> don't
> > > > need
> > > > > it.
> > > > > > > > > MatchAllDocsQuery doesn't.
> > > > > > > > >
> > > > > > > > > Second, move away from Hits ! :) Use Collectors instead.
> > > > > > > > >
> > > > > > > > > If I understand the chain of filters, do you think you can
> > code
> > > > > them
> > > > > > > with
> > > > > > > > > a
> > > > > > > > > BooleanQuery that is added BooleanClauses, each with is
> Term
> > > > > > > > > (field:value)?
> > > > > > > > > You can add clauses w/ OR, AND, NOT etc.
> > > > > > > > >
> > > > > > > > > Note that in Lucene 2.9, you can avoid scoring documents
> > very
> > > > > easily,
> > > > > > > > > which
> > > > > > > > > is a performance win if you don't need scores (i.e. if you
> > just
> > > > > want
> > > > > > to
> > > > > > > > > match everything, not caring for scores).
> > > > > > > > >
> > > > > > > > > Shai
> > > > > > > > >
> > > > > > > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau
> > > > <ak...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > we use Lucene to store around 300 millions of records. We
> > use
> > > > > the
> > > > > > > index
> > > > > > > > > > both
> > > > > > > > > > for conventional searching, but also for all the system's
> > > data
> > > > -
> > > > > we
> > > > > > > > > > replaced
> > > > > > > > > > MySQL with Lucene because it was simply not working at
> all
> > > > with
> > > > > > MySQL
> > > > > > > > > due
> > > > > > > > > > to
> > > > > > > > > > the amount or records. Our problem is that we have HUGE
> > > > > performance
> > > > > > > > > > problems... whenever we search, it takes forever to
> return
> > > > > results,
> > > > > > > and
> > > > > > > > > > Java
> > > > > > > > > > uses 100% CPU/RAM.
> > > > > > > > > >
> > > > > > > > > > Our index fields are like this:
> > > > > > > > > >
> > > > > > > > > > TYPE
> > > > > > > > > > PK
> > > > > > > > > > FOREIGN_PK
> > > > > > > > > > TAG
> > > > > > > > > > ...other information depending on type...
> > > > > > > > > >
> > > > > > > > > > * All fields are Field.Index.UN_TOKENIZED
> > > > > > > > > > * The field "TAG" always contains the value "TAG".
> > > > > > > > > >
> > > > > > > > > > Whenever we search in the index, our query is "TAG:TAG"
> to
> > > > match
> > > > > > all
> > > > > > > > > > documents, and we do the search like this:
> > > > > > > > > >
> > > > > > > > > > // Search
> > > > > > > > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > > > > > > > >
> > > > > > > > > > cluCF is a ChainedFilter containing all the other filters
> > > > (like
> > > > > > > > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > > > > > > > >
> > > > > > > > > > I know that the method is probably crazy because
> "TAG:TAG"
> > is
> > > > > > > matching
> > > > > > > > > all
> > > > > > > > > > 300M documents and then it applies filters; so that's
> > > probably
> > > > > why
> > > > > > > > every
> > > > > > > > > > little query is taking 100% CPU/RAM.... but I don't know
> > how
> > > > to
> > > > > do
> > > > > > it
> > > > > > > > > > properly.
> > > > > > > > > >
> > > > > > > > > > Help ! Any advice is welcome.
> > > > > > > > > >
> > > > > > > > > > - Mike
> > > > > > > > > > akaris@gmail.com
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > ------------------------------------------------------------------
> > > > --
> > > > > -
> > > > > > > > To unsubscribe, e-mail: java-user-
> > unsubscribe@lucene.apache.org
> > > > > > > > For additional commands, e-mail:
> > > java-user-help@lucene.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Performance problems with Lucene 2.9
Posted by Uwe Schindler <uw...@thetaphi.de>.
> Now I have another question... is there a way to specify a "start from" so
> I
> could get page 2, 3, 4, etc.. ?
Search the mailing list, this was explained quite often (by others and me).
The trick is:
If you have 200 results per page, with n = 200 you get the top ranking
results for the first page. If you want to have the second page, reexecute
the query with n=400 and display scoreDocs[200..399] and so on. All full
text engines work like this, they can only collect top-mathcing documents,
but no documents in a slot. If n gets too big it gets very slow and uses too
much memory. Because of that, Google limits the maximum page number.
> On Mon, Nov 30, 2009 at 12:00 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > > And sorting is done by the
> > > collector, Lucene has no idea how to sort.
> >
> > Sorting is done by the internal collector behind the
> > Top(Field)Docs-returning method (your own collectors would have to do it
> > themselves). If you call search(Query, n,... Sort), internally an
> collector
> > is created that does the sorting for you and throws away all results
> that
> > do
> > not fall into the first 200 hits (if n=200).
> >
> > > If you use Sort, the returned
> > > TopDocs will be sorted.
> > >
> > > If you do not sort at all and do not score your results, TopDocs is
> not
> > > very
> > > useful, because the first 200 hits cannot be ranked.
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > > > -----Original Message-----
> > > > From: Michel Nadeau [mailto:akaris@gmail.com]
> > > > Sent: Monday, November 30, 2009 5:35 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: Performance problems with Lucene 2.9
> > > >
> > > > I'll definitely switch to a Collector.
> > > >
> > > > It's just not clear for me if I should use BooleanQueries or
> > > > MatchAllDocuments+Filters ?
> > > >
> > > > And should I write my own collector or the TopDocs one is perfect
> for
> > me
> > > ?
> > > >
> > > > - Mike
> > > > akaris@gmail.com
> > > >
> > > >
> > > > On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson
> > > > <er...@gmail.com>wrote:
> > > >
> > > > > The problem with hits is that a it re-executes the query
> > > > > every N documents where N is 100 (?).
> > > > >
> > > > > So, a loop like
> > > > > for (int idx : hits.length) {
> > > > > do something....
> > > > > }
> > > > >
> > > > > Assuming my memory is right and it's every 100, your query will
> > > > > re-execute (length/100) times. Which is unfortunate.
> > > > >
> > > > > The very quick test to see where to concentrate first would be to
> > take
> > > > > a time stamp just before you hit your loop.....
> > > > >
> > > > > This will tell you whether this loop is the culprit, but it really
> > > > doesn't
> > > > > matter because you'll follow the advice from Uwe and Shai anyway
> <G>.
> > > > >
> > > > > Filtering and Sorting are applied to Collectors before you see
> > > them.....
> > > > >
> > > > > The other bit would be to investigate your sorting. Remember that
> the
> > > > > first sort or two take quite a while since the relevant caches are
> > > > > populated with first used, so second+ queries should be faster.
> The
> > > > > Wiki has some timing/speedup advice.....
> > > > >
> > > > > Best
> > > > > Erick
> > > > >
> > > > >
> > > > > On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <ak...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > What is the main difference between Hits and Collectors?
> > > > > >
> > > > > > - Mike
> > > > > > akaris@gmail.com
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler
> <uw...@thetaphi.de>
> > > > wrote:
> > > > > >
> > > > > > > And if you only have a filter and apply it to all documents,
> make
> > > a
> > > > > > > ConstantScoreQuery on top of the filter:
> > > > > > >
> > > > > > > Query q=new ConstantScoreQuery(cluCF);
> > > > > > >
> > > > > > > Then remove the filter from your search method call and only
> > > execute
> > > > > this
> > > > > > > query.
> > > > > > >
> > > > > > > And if you iterate over all results never-ever use Hits! (its
> > > > already
> > > > > > > deprecated). Write a Collector instead (as you are not
> interested
> > > in
> > > > > > > scoring).
> > > > > > >
> > > > > > > And: If you replace a relational database with Lucene, be sure
> > not
> > > > to
> > > > > > think
> > > > > > > in a relational sense with foreign keys / primary keys and so
> on.
> > > In
> > > > > > > general
> > > > > > > you should flatten everything.
> > > > > > >
> > > > > > > Uwe
> > > > > > >
> > > > > > > -----
> > > > > > > Uwe Schindler
> > > > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > > > http://www.thetaphi.de
> > > > > > > eMail: uwe@thetaphi.de
> > > > > > >
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > > > > Sent: Monday, November 30, 2009 4:56 PM
> > > > > > > > To: java-user@lucene.apache.org
> > > > > > > > Subject: Re: Performance problems with Lucene 2.9
> > > > > > > >
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > First you can use MatchAllDocsQuery, which matches all
> > > documents.
> > > > It
> > > > > > will
> > > > > > > > save a HUGE posting list (TAG:TAG), and performs much
> faster.
> > > For
> > > > > > example
> > > > > > > > TAG:TAG computes a score for each doc, even though you don't
> > > need
> > > > it.
> > > > > > > > MatchAllDocsQuery doesn't.
> > > > > > > >
> > > > > > > > Second, move away from Hits ! :) Use Collectors instead.
> > > > > > > >
> > > > > > > > If I understand the chain of filters, do you think you can
> code
> > > > them
> > > > > > with
> > > > > > > > a
> > > > > > > > BooleanQuery that is added BooleanClauses, each with is Term
> > > > > > > > (field:value)?
> > > > > > > > You can add clauses w/ OR, AND, NOT etc.
> > > > > > > >
> > > > > > > > Note that in Lucene 2.9, you can avoid scoring documents
> very
> > > > easily,
> > > > > > > > which
> > > > > > > > is a performance win if you don't need scores (i.e. if you
> just
> > > > want
> > > > > to
> > > > > > > > match everything, not caring for scores).
> > > > > > > >
> > > > > > > > Shai
> > > > > > > >
> > > > > > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau
> > > <ak...@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > we use Lucene to store around 300 millions of records. We
> use
> > > > the
> > > > > > index
> > > > > > > > > both
> > > > > > > > > for conventional searching, but also for all the system's
> > data
> > > -
> > > > we
> > > > > > > > > replaced
> > > > > > > > > MySQL with Lucene because it was simply not working at all
> > > with
> > > > > MySQL
> > > > > > > > due
> > > > > > > > > to
> > > > > > > > > the amount or records. Our problem is that we have HUGE
> > > > performance
> > > > > > > > > problems... whenever we search, it takes forever to return
> > > > results,
> > > > > > and
> > > > > > > > > Java
> > > > > > > > > uses 100% CPU/RAM.
> > > > > > > > >
> > > > > > > > > Our index fields are like this:
> > > > > > > > >
> > > > > > > > > TYPE
> > > > > > > > > PK
> > > > > > > > > FOREIGN_PK
> > > > > > > > > TAG
> > > > > > > > > ...other information depending on type...
> > > > > > > > >
> > > > > > > > > * All fields are Field.Index.UN_TOKENIZED
> > > > > > > > > * The field "TAG" always contains the value "TAG".
> > > > > > > > >
> > > > > > > > > Whenever we search in the index, our query is "TAG:TAG" to
> > > match
> > > > > all
> > > > > > > > > documents, and we do the search like this:
> > > > > > > > >
> > > > > > > > > // Search
> > > > > > > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > > > > > > >
> > > > > > > > > cluCF is a ChainedFilter containing all the other filters
> > > (like
> > > > > > > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > > > > > > >
> > > > > > > > > I know that the method is probably crazy because "TAG:TAG"
> is
> > > > > > matching
> > > > > > > > all
> > > > > > > > > 300M documents and then it applies filters; so that's
> > probably
> > > > why
> > > > > > > every
> > > > > > > > > little query is taking 100% CPU/RAM.... but I don't know
> how
> > > to
> > > > do
> > > > > it
> > > > > > > > > properly.
> > > > > > > > >
> > > > > > > > > Help ! Any advice is welcome.
> > > > > > > > >
> > > > > > > > > - Mike
> > > > > > > > > akaris@gmail.com
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > ------------------------------------------------------------------
> > > --
> > > > -
> > > > > > > To unsubscribe, e-mail: java-user-
> unsubscribe@lucene.apache.org
> > > > > > > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Performance problems with Lucene 2.9
Posted by Uwe Schindler <uw...@thetaphi.de>.
> you think that something like this -
> TopFieldDocs tfd = searcher.search(new ConstantScoreQuery(cluCF), null,
> 200,
> cluSort);
This is little bit faster as it does not need to intersect the all queries
with the filtered ones.
> Would be more performant than using MatchAllDocsQuery with Filters like
> this
> -
> TopFieldDocs tfd = searcher.search(new MatchAllDocsQuery(), cluCF, 200,
> cluSort);
Slower, as the iterator on all document ids needs to be intersected with the
filtered iterator.
It's just one step more.
> Thanks!
>
> - Mike
> akaris@gmail.com
>
>
> On Mon, Nov 30, 2009 at 12:06 PM, Michel Nadeau <ak...@gmail.com> wrote:
>
> > I'm currently trying something like this -
> >
> > TopFieldDocs tfd = searcher.search(new MatchAllDocsQuery(), cluCF, 200,
> > cluSort);
> >
> > cluCF = filters
> > cluSort = sorts
> >
> > Now I have another question... is there a way to specify a "start from"
> so
> > I could get page 2, 3, 4, etc.. ?
> >
> > - Mike
> > akaris@gmail.com
> >
> >
> >
> > On Mon, Nov 30, 2009 at 12:00 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> >> > And sorting is done by the
> >> > collector, Lucene has no idea how to sort.
> >>
> >> Sorting is done by the internal collector behind the
> >> Top(Field)Docs-returning method (your own collectors would have to do
> it
> >> themselves). If you call search(Query, n,... Sort), internally an
> >> collector
> >> is created that does the sorting for you and throws away all results
> that
> >> do
> >> not fall into the first 200 hits (if n=200).
> >>
> >> > If you use Sort, the returned
> >> > TopDocs will be sorted.
> >> >
> >> > If you do not sort at all and do not score your results, TopDocs is
> not
> >> > very
> >> > useful, because the first 200 hits cannot be ranked.
> >> >
> >> > -----
> >> > Uwe Schindler
> >> > H.-H.-Meier-Allee 63, D-28213 Bremen
> >> > http://www.thetaphi.de
> >> > eMail: uwe@thetaphi.de
> >> >
> >> > > -----Original Message-----
> >> > > From: Michel Nadeau [mailto:akaris@gmail.com]
> >> > > Sent: Monday, November 30, 2009 5:35 PM
> >> > > To: java-user@lucene.apache.org
> >> > > Subject: Re: Performance problems with Lucene 2.9
> >> > >
> >> > > I'll definitely switch to a Collector.
> >> > >
> >> > > It's just not clear for me if I should use BooleanQueries or
> >> > > MatchAllDocuments+Filters ?
> >> > >
> >> > > And should I write my own collector or the TopDocs one is perfect
> for
> >> me
> >> > ?
> >> > >
> >> > > - Mike
> >> > > akaris@gmail.com
> >> > >
> >> > >
> >> > > On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson
> >> > > <er...@gmail.com>wrote:
> >> > >
> >> > > > The problem with hits is that a it re-executes the query
> >> > > > every N documents where N is 100 (?).
> >> > > >
> >> > > > So, a loop like
> >> > > > for (int idx : hits.length) {
> >> > > > do something....
> >> > > > }
> >> > > >
> >> > > > Assuming my memory is right and it's every 100, your query will
> >> > > > re-execute (length/100) times. Which is unfortunate.
> >> > > >
> >> > > > The very quick test to see where to concentrate first would be to
> >> take
> >> > > > a time stamp just before you hit your loop.....
> >> > > >
> >> > > > This will tell you whether this loop is the culprit, but it
> really
> >> > > doesn't
> >> > > > matter because you'll follow the advice from Uwe and Shai anyway
> >> <G>.
> >> > > >
> >> > > > Filtering and Sorting are applied to Collectors before you see
> >> > them.....
> >> > > >
> >> > > > The other bit would be to investigate your sorting. Remember that
> >> the
> >> > > > first sort or two take quite a while since the relevant caches
> are
> >> > > > populated with first used, so second+ queries should be faster.
> The
> >> > > > Wiki has some timing/speedup advice.....
> >> > > >
> >> > > > Best
> >> > > > Erick
> >> > > >
> >> > > >
> >> > > > On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau
> <ak...@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > What is the main difference between Hits and Collectors?
> >> > > > >
> >> > > > > - Mike
> >> > > > > akaris@gmail.com
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler
> <uw...@thetaphi.de>
> >> > > wrote:
> >> > > > >
> >> > > > > > And if you only have a filter and apply it to all documents,
> >> make
> >> > a
> >> > > > > > ConstantScoreQuery on top of the filter:
> >> > > > > >
> >> > > > > > Query q=new ConstantScoreQuery(cluCF);
> >> > > > > >
> >> > > > > > Then remove the filter from your search method call and only
> >> > execute
> >> > > > this
> >> > > > > > query.
> >> > > > > >
> >> > > > > > And if you iterate over all results never-ever use Hits! (its
> >> > > already
> >> > > > > > deprecated). Write a Collector instead (as you are not
> >> interested
> >> > in
> >> > > > > > scoring).
> >> > > > > >
> >> > > > > > And: If you replace a relational database with Lucene, be
> sure
> >> not
> >> > > to
> >> > > > > think
> >> > > > > > in a relational sense with foreign keys / primary keys and so
> >> on.
> >> > In
> >> > > > > > general
> >> > > > > > you should flatten everything.
> >> > > > > >
> >> > > > > > Uwe
> >> > > > > >
> >> > > > > > -----
> >> > > > > > Uwe Schindler
> >> > > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> >> > > > > > http://www.thetaphi.de
> >> > > > > > eMail: uwe@thetaphi.de
> >> > > > > >
> >> > > > > >
> >> > > > > > > -----Original Message-----
> >> > > > > > > From: Shai Erera [mailto:serera@gmail.com]
> >> > > > > > > Sent: Monday, November 30, 2009 4:56 PM
> >> > > > > > > To: java-user@lucene.apache.org
> >> > > > > > > Subject: Re: Performance problems with Lucene 2.9
> >> > > > > > >
> >> > > > > > > Hi
> >> > > > > > >
> >> > > > > > > First you can use MatchAllDocsQuery, which matches all
> >> > documents.
> >> > > It
> >> > > > > will
> >> > > > > > > save a HUGE posting list (TAG:TAG), and performs much
> faster.
> >> > For
> >> > > > > example
> >> > > > > > > TAG:TAG computes a score for each doc, even though you
> don't
> >> > need
> >> > > it.
> >> > > > > > > MatchAllDocsQuery doesn't.
> >> > > > > > >
> >> > > > > > > Second, move away from Hits ! :) Use Collectors instead.
> >> > > > > > >
> >> > > > > > > If I understand the chain of filters, do you think you can
> >> code
> >> > > them
> >> > > > > with
> >> > > > > > > a
> >> > > > > > > BooleanQuery that is added BooleanClauses, each with is
> Term
> >> > > > > > > (field:value)?
> >> > > > > > > You can add clauses w/ OR, AND, NOT etc.
> >> > > > > > >
> >> > > > > > > Note that in Lucene 2.9, you can avoid scoring documents
> very
> >> > > easily,
> >> > > > > > > which
> >> > > > > > > is a performance win if you don't need scores (i.e. if you
> >> just
> >> > > want
> >> > > > to
> >> > > > > > > match everything, not caring for scores).
> >> > > > > > >
> >> > > > > > > Shai
> >> > > > > > >
> >> > > > > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau
> >> > <ak...@gmail.com>
> >> > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi,
> >> > > > > > > >
> >> > > > > > > > we use Lucene to store around 300 millions of records. We
> >> use
> >> > > the
> >> > > > > index
> >> > > > > > > > both
> >> > > > > > > > for conventional searching, but also for all the system's
> >> data
> >> > -
> >> > > we
> >> > > > > > > > replaced
> >> > > > > > > > MySQL with Lucene because it was simply not working at
> all
> >> > with
> >> > > > MySQL
> >> > > > > > > due
> >> > > > > > > > to
> >> > > > > > > > the amount or records. Our problem is that we have HUGE
> >> > > performance
> >> > > > > > > > problems... whenever we search, it takes forever to
> return
> >> > > results,
> >> > > > > and
> >> > > > > > > > Java
> >> > > > > > > > uses 100% CPU/RAM.
> >> > > > > > > >
> >> > > > > > > > Our index fields are like this:
> >> > > > > > > >
> >> > > > > > > > TYPE
> >> > > > > > > > PK
> >> > > > > > > > FOREIGN_PK
> >> > > > > > > > TAG
> >> > > > > > > > ...other information depending on type...
> >> > > > > > > >
> >> > > > > > > > * All fields are Field.Index.UN_TOKENIZED
> >> > > > > > > > * The field "TAG" always contains the value "TAG".
> >> > > > > > > >
> >> > > > > > > > Whenever we search in the index, our query is "TAG:TAG"
> to
> >> > match
> >> > > > all
> >> > > > > > > > documents, and we do the search like this:
> >> > > > > > > >
> >> > > > > > > > // Search
> >> > > > > > > > Hits h = searcher.search(q, cluCF, cluSort);
> >> > > > > > > >
> >> > > > > > > > cluCF is a ChainedFilter containing all the other filters
> >> > (like
> >> > > > > > > > FOREIGN_PK=12345, TYPE=a, etc.).
> >> > > > > > > >
> >> > > > > > > > I know that the method is probably crazy because
> "TAG:TAG"
> >> is
> >> > > > > matching
> >> > > > > > > all
> >> > > > > > > > 300M documents and then it applies filters; so that's
> >> probably
> >> > > why
> >> > > > > > every
> >> > > > > > > > little query is taking 100% CPU/RAM.... but I don't know
> how
> >> > to
> >> > > do
> >> > > > it
> >> > > > > > > > properly.
> >> > > > > > > >
> >> > > > > > > > Help ! Any advice is welcome.
> >> > > > > > > >
> >> > > > > > > > - Mike
> >> > > > > > > > akaris@gmail.com
> >> > > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> ------------------------------------------------------------------
> >> > --
> >> > > -
> >> > > > > > To unsubscribe, e-mail: java-user-
> unsubscribe@lucene.apache.org
> >> > > > > > For additional commands, e-mail:
> >> java-user-help@lucene.apache.org
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Performance problems with Lucene 2.9
Posted by Michel Nadeau <ak...@gmail.com>.
Uwe,
you think that something like this -
TopFieldDocs tfd = searcher.search(new ConstantScoreQuery(cluCF), null, 200,
cluSort);
Would be more performant than using MatchAllDocsQuery with Filters like this
-
TopFieldDocs tfd = searcher.search(new MatchAllDocsQuery(), cluCF, 200,
cluSort);
Thanks!
- Mike
akaris@gmail.com
On Mon, Nov 30, 2009 at 12:06 PM, Michel Nadeau <ak...@gmail.com> wrote:
> I'm currently trying something like this -
>
> TopFieldDocs tfd = searcher.search(new MatchAllDocsQuery(), cluCF, 200,
> cluSort);
>
> cluCF = filters
> cluSort = sorts
>
> Now I have another question... is there a way to specify a "start from" so
> I could get page 2, 3, 4, etc.. ?
>
> - Mike
> akaris@gmail.com
>
>
>
> On Mon, Nov 30, 2009 at 12:00 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
>> > And sorting is done by the
>> > collector, Lucene has no idea how to sort.
>>
>> Sorting is done by the internal collector behind the
>> Top(Field)Docs-returning method (your own collectors would have to do it
>> themselves). If you call search(Query, n,... Sort), internally an
>> collector
>> is created that does the sorting for you and throws away all results that
>> do
>> not fall into the first 200 hits (if n=200).
>>
>> > If you use Sort, the returned
>> > TopDocs will be sorted.
>> >
>> > If you do not sort at all and do not score your results, TopDocs is not
>> > very
>> > useful, because the first 200 hits cannot be ranked.
>> >
>> > -----
>> > Uwe Schindler
>> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > http://www.thetaphi.de
>> > eMail: uwe@thetaphi.de
>> >
>> > > -----Original Message-----
>> > > From: Michel Nadeau [mailto:akaris@gmail.com]
>> > > Sent: Monday, November 30, 2009 5:35 PM
>> > > To: java-user@lucene.apache.org
>> > > Subject: Re: Performance problems with Lucene 2.9
>> > >
>> > > I'll definitely switch to a Collector.
>> > >
>> > > It's just not clear for me if I should use BooleanQueries or
>> > > MatchAllDocuments+Filters ?
>> > >
>> > > And should I write my own collector or the TopDocs one is perfect for
>> me
>> > ?
>> > >
>> > > - Mike
>> > > akaris@gmail.com
>> > >
>> > >
>> > > On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson
>> > > <er...@gmail.com>wrote:
>> > >
>> > > > The problem with hits is that a it re-executes the query
>> > > > every N documents where N is 100 (?).
>> > > >
>> > > > So, a loop like
>> > > > for (int idx : hits.length) {
>> > > > do something....
>> > > > }
>> > > >
>> > > > Assuming my memory is right and it's every 100, your query will
>> > > > re-execute (length/100) times. Which is unfortunate.
>> > > >
>> > > > The very quick test to see where to concentrate first would be to
>> take
>> > > > a time stamp just before you hit your loop.....
>> > > >
>> > > > This will tell you whether this loop is the culprit, but it really
>> > > doesn't
>> > > > matter because you'll follow the advice from Uwe and Shai anyway
>> <G>.
>> > > >
>> > > > Filtering and Sorting are applied to Collectors before you see
>> > them.....
>> > > >
>> > > > The other bit would be to investigate your sorting. Remember that
>> the
>> > > > first sort or two take quite a while since the relevant caches are
>> > > > populated with first used, so second+ queries should be faster. The
>> > > > Wiki has some timing/speedup advice.....
>> > > >
>> > > > Best
>> > > > Erick
>> > > >
>> > > >
>> > > > On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <ak...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > What is the main difference between Hits and Collectors?
>> > > > >
>> > > > > - Mike
>> > > > > akaris@gmail.com
>> > > > >
>> > > > >
>> > > > > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de>
>> > > wrote:
>> > > > >
>> > > > > > And if you only have a filter and apply it to all documents,
>> make
>> > a
>> > > > > > ConstantScoreQuery on top of the filter:
>> > > > > >
>> > > > > > Query q=new ConstantScoreQuery(cluCF);
>> > > > > >
>> > > > > > Then remove the filter from your search method call and only
>> > execute
>> > > > this
>> > > > > > query.
>> > > > > >
>> > > > > > And if you iterate over all results never-ever use Hits! (its
>> > > already
>> > > > > > deprecated). Write a Collector instead (as you are not
>> interested
>> > in
>> > > > > > scoring).
>> > > > > >
>> > > > > > And: If you replace a relational database with Lucene, be sure
>> not
>> > > to
>> > > > > think
>> > > > > > in a relational sense with foreign keys / primary keys and so
>> on.
>> > In
>> > > > > > general
>> > > > > > you should flatten everything.
>> > > > > >
>> > > > > > Uwe
>> > > > > >
>> > > > > > -----
>> > > > > > Uwe Schindler
>> > > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > > > > > http://www.thetaphi.de
>> > > > > > eMail: uwe@thetaphi.de
>> > > > > >
>> > > > > >
>> > > > > > > -----Original Message-----
>> > > > > > > From: Shai Erera [mailto:serera@gmail.com]
>> > > > > > > Sent: Monday, November 30, 2009 4:56 PM
>> > > > > > > To: java-user@lucene.apache.org
>> > > > > > > Subject: Re: Performance problems with Lucene 2.9
>> > > > > > >
>> > > > > > > Hi
>> > > > > > >
>> > > > > > > First you can use MatchAllDocsQuery, which matches all
>> > documents.
>> > > It
>> > > > > will
>> > > > > > > save a HUGE posting list (TAG:TAG), and performs much faster.
>> > For
>> > > > > example
>> > > > > > > TAG:TAG computes a score for each doc, even though you don't
>> > need
>> > > it.
>> > > > > > > MatchAllDocsQuery doesn't.
>> > > > > > >
>> > > > > > > Second, move away from Hits ! :) Use Collectors instead.
>> > > > > > >
>> > > > > > > If I understand the chain of filters, do you think you can
>> code
>> > > them
>> > > > > with
>> > > > > > > a
>> > > > > > > BooleanQuery that is added BooleanClauses, each with is Term
>> > > > > > > (field:value)?
>> > > > > > > You can add clauses w/ OR, AND, NOT etc.
>> > > > > > >
>> > > > > > > Note that in Lucene 2.9, you can avoid scoring documents very
>> > > easily,
>> > > > > > > which
>> > > > > > > is a performance win if you don't need scores (i.e. if you
>> just
>> > > want
>> > > > to
>> > > > > > > match everything, not caring for scores).
>> > > > > > >
>> > > > > > > Shai
>> > > > > > >
>> > > > > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau
>> > <ak...@gmail.com>
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi,
>> > > > > > > >
>> > > > > > > > we use Lucene to store around 300 millions of records. We
>> use
>> > > the
>> > > > > index
>> > > > > > > > both
>> > > > > > > > for conventional searching, but also for all the system's
>> data
>> > -
>> > > we
>> > > > > > > > replaced
>> > > > > > > > MySQL with Lucene because it was simply not working at all
>> > with
>> > > > MySQL
>> > > > > > > due
>> > > > > > > > to
>> > > > > > > > the amount or records. Our problem is that we have HUGE
>> > > performance
>> > > > > > > > problems... whenever we search, it takes forever to return
>> > > results,
>> > > > > and
>> > > > > > > > Java
>> > > > > > > > uses 100% CPU/RAM.
>> > > > > > > >
>> > > > > > > > Our index fields are like this:
>> > > > > > > >
>> > > > > > > > TYPE
>> > > > > > > > PK
>> > > > > > > > FOREIGN_PK
>> > > > > > > > TAG
>> > > > > > > > ...other information depending on type...
>> > > > > > > >
>> > > > > > > > * All fields are Field.Index.UN_TOKENIZED
>> > > > > > > > * The field "TAG" always contains the value "TAG".
>> > > > > > > >
>> > > > > > > > Whenever we search in the index, our query is "TAG:TAG" to
>> > match
>> > > > all
>> > > > > > > > documents, and we do the search like this:
>> > > > > > > >
>> > > > > > > > // Search
>> > > > > > > > Hits h = searcher.search(q, cluCF, cluSort);
>> > > > > > > >
>> > > > > > > > cluCF is a ChainedFilter containing all the other filters
>> > (like
>> > > > > > > > FOREIGN_PK=12345, TYPE=a, etc.).
>> > > > > > > >
>> > > > > > > > I know that the method is probably crazy because "TAG:TAG"
>> is
>> > > > > matching
>> > > > > > > all
>> > > > > > > > 300M documents and then it applies filters; so that's
>> probably
>> > > why
>> > > > > > every
>> > > > > > > > little query is taking 100% CPU/RAM.... but I don't know how
>> > to
>> > > do
>> > > > it
>> > > > > > > > properly.
>> > > > > > > >
>> > > > > > > > Help ! Any advice is welcome.
>> > > > > > > >
>> > > > > > > > - Mike
>> > > > > > > > akaris@gmail.com
>> > > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> ------------------------------------------------------------------
>> > --
>> > > -
>> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > > > > For additional commands, e-mail:
>> java-user-help@lucene.apache.org
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: Performance problems with Lucene 2.9
Posted by Michel Nadeau <ak...@gmail.com>.
I'm currently trying something like this -
TopFieldDocs tfd = searcher.search(new MatchAllDocsQuery(), cluCF, 200,
cluSort);
cluCF = filters
cluSort = sorts
Now I have another question... is there a way to specify a "start from" so I
could get page 2, 3, 4, etc.. ?
- Mike
akaris@gmail.com
On Mon, Nov 30, 2009 at 12:00 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> > And sorting is done by the
> > collector, Lucene has no idea how to sort.
>
> Sorting is done by the internal collector behind the
> Top(Field)Docs-returning method (your own collectors would have to do it
> themselves). If you call search(Query, n,... Sort), internally an collector
> is created that does the sorting for you and throws away all results that
> do
> not fall into the first 200 hits (if n=200).
>
> > If you use Sort, the returned
> > TopDocs will be sorted.
> >
> > If you do not sort at all and do not score your results, TopDocs is not
> > very
> > useful, because the first 200 hits cannot be ranked.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> > > -----Original Message-----
> > > From: Michel Nadeau [mailto:akaris@gmail.com]
> > > Sent: Monday, November 30, 2009 5:35 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: Performance problems with Lucene 2.9
> > >
> > > I'll definitely switch to a Collector.
> > >
> > > It's just not clear for me if I should use BooleanQueries or
> > > MatchAllDocuments+Filters ?
> > >
> > > And should I write my own collector or the TopDocs one is perfect for
> me
> > ?
> > >
> > > - Mike
> > > akaris@gmail.com
> > >
> > >
> > > On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson
> > > <er...@gmail.com>wrote:
> > >
> > > > The problem with hits is that a it re-executes the query
> > > > every N documents where N is 100 (?).
> > > >
> > > > So, a loop like
> > > > for (int idx : hits.length) {
> > > > do something....
> > > > }
> > > >
> > > > Assuming my memory is right and it's every 100, your query will
> > > > re-execute (length/100) times. Which is unfortunate.
> > > >
> > > > The very quick test to see where to concentrate first would be to
> take
> > > > a time stamp just before you hit your loop.....
> > > >
> > > > This will tell you whether this loop is the culprit, but it really
> > > doesn't
> > > > matter because you'll follow the advice from Uwe and Shai anyway <G>.
> > > >
> > > > Filtering and Sorting are applied to Collectors before you see
> > them.....
> > > >
> > > > The other bit would be to investigate your sorting. Remember that the
> > > > first sort or two take quite a while since the relevant caches are
> > > > populated with first used, so second+ queries should be faster. The
> > > > Wiki has some timing/speedup advice.....
> > > >
> > > > Best
> > > > Erick
> > > >
> > > >
> > > > On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <ak...@gmail.com>
> > > wrote:
> > > >
> > > > > What is the main difference between Hits and Collectors?
> > > > >
> > > > > - Mike
> > > > > akaris@gmail.com
> > > > >
> > > > >
> > > > > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de>
> > > wrote:
> > > > >
> > > > > > And if you only have a filter and apply it to all documents, make
> > a
> > > > > > ConstantScoreQuery on top of the filter:
> > > > > >
> > > > > > Query q=new ConstantScoreQuery(cluCF);
> > > > > >
> > > > > > Then remove the filter from your search method call and only
> > execute
> > > > this
> > > > > > query.
> > > > > >
> > > > > > And if you iterate over all results never-ever use Hits! (its
> > > already
> > > > > > deprecated). Write a Collector instead (as you are not interested
> > in
> > > > > > scoring).
> > > > > >
> > > > > > And: If you replace a relational database with Lucene, be sure
> not
> > > to
> > > > > think
> > > > > > in a relational sense with foreign keys / primary keys and so on.
> > In
> > > > > > general
> > > > > > you should flatten everything.
> > > > > >
> > > > > > Uwe
> > > > > >
> > > > > > -----
> > > > > > Uwe Schindler
> > > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > > http://www.thetaphi.de
> > > > > > eMail: uwe@thetaphi.de
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > > > Sent: Monday, November 30, 2009 4:56 PM
> > > > > > > To: java-user@lucene.apache.org
> > > > > > > Subject: Re: Performance problems with Lucene 2.9
> > > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > First you can use MatchAllDocsQuery, which matches all
> > documents.
> > > It
> > > > > will
> > > > > > > save a HUGE posting list (TAG:TAG), and performs much faster.
> > For
> > > > > example
> > > > > > > TAG:TAG computes a score for each doc, even though you don't
> > need
> > > it.
> > > > > > > MatchAllDocsQuery doesn't.
> > > > > > >
> > > > > > > Second, move away from Hits ! :) Use Collectors instead.
> > > > > > >
> > > > > > > If I understand the chain of filters, do you think you can code
> > > them
> > > > > with
> > > > > > > a
> > > > > > > BooleanQuery that is added BooleanClauses, each with is Term
> > > > > > > (field:value)?
> > > > > > > You can add clauses w/ OR, AND, NOT etc.
> > > > > > >
> > > > > > > Note that in Lucene 2.9, you can avoid scoring documents very
> > > easily,
> > > > > > > which
> > > > > > > is a performance win if you don't need scores (i.e. if you just
> > > want
> > > > to
> > > > > > > match everything, not caring for scores).
> > > > > > >
> > > > > > > Shai
> > > > > > >
> > > > > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau
> > <ak...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > we use Lucene to store around 300 millions of records. We use
> > > the
> > > > > index
> > > > > > > > both
> > > > > > > > for conventional searching, but also for all the system's
> data
> > -
> > > we
> > > > > > > > replaced
> > > > > > > > MySQL with Lucene because it was simply not working at all
> > with
> > > > MySQL
> > > > > > > due
> > > > > > > > to
> > > > > > > > the amount or records. Our problem is that we have HUGE
> > > performance
> > > > > > > > problems... whenever we search, it takes forever to return
> > > results,
> > > > > and
> > > > > > > > Java
> > > > > > > > uses 100% CPU/RAM.
> > > > > > > >
> > > > > > > > Our index fields are like this:
> > > > > > > >
> > > > > > > > TYPE
> > > > > > > > PK
> > > > > > > > FOREIGN_PK
> > > > > > > > TAG
> > > > > > > > ...other information depending on type...
> > > > > > > >
> > > > > > > > * All fields are Field.Index.UN_TOKENIZED
> > > > > > > > * The field "TAG" always contains the value "TAG".
> > > > > > > >
> > > > > > > > Whenever we search in the index, our query is "TAG:TAG" to
> > match
> > > > all
> > > > > > > > documents, and we do the search like this:
> > > > > > > >
> > > > > > > > // Search
> > > > > > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > > > > > >
> > > > > > > > cluCF is a ChainedFilter containing all the other filters
> > (like
> > > > > > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > > > > > >
> > > > > > > > I know that the method is probably crazy because "TAG:TAG" is
> > > > > matching
> > > > > > > all
> > > > > > > > 300M documents and then it applies filters; so that's
> probably
> > > why
> > > > > > every
> > > > > > > > little query is taking 100% CPU/RAM.... but I don't know how
> > to
> > > do
> > > > it
> > > > > > > > properly.
> > > > > > > >
> > > > > > > > Help ! Any advice is welcome.
> > > > > > > >
> > > > > > > > - Mike
> > > > > > > > akaris@gmail.com
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> ------------------------------------------------------------------
> > --
> > > -
> > > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > > For additional commands, e-mail:
> java-user-help@lucene.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Performance problems with Lucene 2.9
Posted by Uwe Schindler <uw...@thetaphi.de>.
> And sorting is done by the
> collector, Lucene has no idea how to sort.
Sorting is done by the internal collector behind the
Top(Field)Docs-returning method (your own collectors would have to do it
themselves). If you call search(Query, n,... Sort), internally an collector
is created that does the sorting for you and throws away all results that do
not fall into the first 200 hits (if n=200).
> If you use Sort, the returned
> TopDocs will be sorted.
>
> If you do not sort at all and do not score your results, TopDocs is not
> very
> useful, because the first 200 hits cannot be ranked.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Michel Nadeau [mailto:akaris@gmail.com]
> > Sent: Monday, November 30, 2009 5:35 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Performance problems with Lucene 2.9
> >
> > I'll definitely switch to a Collector.
> >
> > It's just not clear for me if I should use BooleanQueries or
> > MatchAllDocuments+Filters ?
> >
> > And should I write my own collector or the TopDocs one is perfect for me
> ?
> >
> > - Mike
> > akaris@gmail.com
> >
> >
> > On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson
> > <er...@gmail.com>wrote:
> >
> > > The problem with hits is that a it re-executes the query
> > > every N documents where N is 100 (?).
> > >
> > > So, a loop like
> > > for (int idx : hits.length) {
> > > do something....
> > > }
> > >
> > > Assuming my memory is right and it's every 100, your query will
> > > re-execute (length/100) times. Which is unfortunate.
> > >
> > > The very quick test to see where to concentrate first would be to take
> > > a time stamp just before you hit your loop.....
> > >
> > > This will tell you whether this loop is the culprit, but it really
> > doesn't
> > > matter because you'll follow the advice from Uwe and Shai anyway <G>.
> > >
> > > Filtering and Sorting are applied to Collectors before you see
> them.....
> > >
> > > The other bit would be to investigate your sorting. Remember that the
> > > first sort or two take quite a while since the relevant caches are
> > > populated with first used, so second+ queries should be faster. The
> > > Wiki has some timing/speedup advice.....
> > >
> > > Best
> > > Erick
> > >
> > >
> > > On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <ak...@gmail.com>
> > wrote:
> > >
> > > > What is the main difference between Hits and Collectors?
> > > >
> > > > - Mike
> > > > akaris@gmail.com
> > > >
> > > >
> > > > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de>
> > wrote:
> > > >
> > > > > And if you only have a filter and apply it to all documents, make
> a
> > > > > ConstantScoreQuery on top of the filter:
> > > > >
> > > > > Query q=new ConstantScoreQuery(cluCF);
> > > > >
> > > > > Then remove the filter from your search method call and only
> execute
> > > this
> > > > > query.
> > > > >
> > > > > And if you iterate over all results never-ever use Hits! (its
> > already
> > > > > deprecated). Write a Collector instead (as you are not interested
> in
> > > > > scoring).
> > > > >
> > > > > And: If you replace a relational database with Lucene, be sure not
> > to
> > > > think
> > > > > in a relational sense with foreign keys / primary keys and so on.
> In
> > > > > general
> > > > > you should flatten everything.
> > > > >
> > > > > Uwe
> > > > >
> > > > > -----
> > > > > Uwe Schindler
> > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > http://www.thetaphi.de
> > > > > eMail: uwe@thetaphi.de
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > > Sent: Monday, November 30, 2009 4:56 PM
> > > > > > To: java-user@lucene.apache.org
> > > > > > Subject: Re: Performance problems with Lucene 2.9
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > First you can use MatchAllDocsQuery, which matches all
> documents.
> > It
> > > > will
> > > > > > save a HUGE posting list (TAG:TAG), and performs much faster.
> For
> > > > example
> > > > > > TAG:TAG computes a score for each doc, even though you don't
> need
> > it.
> > > > > > MatchAllDocsQuery doesn't.
> > > > > >
> > > > > > Second, move away from Hits ! :) Use Collectors instead.
> > > > > >
> > > > > > If I understand the chain of filters, do you think you can code
> > them
> > > > with
> > > > > > a
> > > > > > BooleanQuery that is added BooleanClauses, each with is Term
> > > > > > (field:value)?
> > > > > > You can add clauses w/ OR, AND, NOT etc.
> > > > > >
> > > > > > Note that in Lucene 2.9, you can avoid scoring documents very
> > easily,
> > > > > > which
> > > > > > is a performance win if you don't need scores (i.e. if you just
> > want
> > > to
> > > > > > match everything, not caring for scores).
> > > > > >
> > > > > > Shai
> > > > > >
> > > > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau
> <ak...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > we use Lucene to store around 300 millions of records. We use
> > the
> > > > index
> > > > > > > both
> > > > > > > for conventional searching, but also for all the system's data
> -
> > we
> > > > > > > replaced
> > > > > > > MySQL with Lucene because it was simply not working at all
> with
> > > MySQL
> > > > > > due
> > > > > > > to
> > > > > > > the amount or records. Our problem is that we have HUGE
> > performance
> > > > > > > problems... whenever we search, it takes forever to return
> > results,
> > > > and
> > > > > > > Java
> > > > > > > uses 100% CPU/RAM.
> > > > > > >
> > > > > > > Our index fields are like this:
> > > > > > >
> > > > > > > TYPE
> > > > > > > PK
> > > > > > > FOREIGN_PK
> > > > > > > TAG
> > > > > > > ...other information depending on type...
> > > > > > >
> > > > > > > * All fields are Field.Index.UN_TOKENIZED
> > > > > > > * The field "TAG" always contains the value "TAG".
> > > > > > >
> > > > > > > Whenever we search in the index, our query is "TAG:TAG" to
> match
> > > all
> > > > > > > documents, and we do the search like this:
> > > > > > >
> > > > > > > // Search
> > > > > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > > > > >
> > > > > > > cluCF is a ChainedFilter containing all the other filters
> (like
> > > > > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > > > > >
> > > > > > > I know that the method is probably crazy because "TAG:TAG" is
> > > > matching
> > > > > > all
> > > > > > > 300M documents and then it applies filters; so that's probably
> > why
> > > > > every
> > > > > > > little query is taking 100% CPU/RAM.... but I don't know how
> to
> > do
> > > it
> > > > > > > properly.
> > > > > > >
> > > > > > > Help ! Any advice is welcome.
> > > > > > >
> > > > > > > - Mike
> > > > > > > akaris@gmail.com
> > > > > > >
> > > > >
> > > > >
> > > > > ------------------------------------------------------------------
> --
> > -
> > > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > > >
> > > > >
> > > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Performance problems with Lucene 2.9
Posted by Uwe Schindler <uw...@thetaphi.de>.
You should use ConstantScoreQuery(filter) as query if you want to filter all
docs and need no scoring! This disables scoring automatically. It is the
same (but more performant) like combining MatchAllDocs with a Filter.
If you only need the top 200 results, use TopDocs search(Query, int) and set
the second parameter to 200. In the returned TopDocs you get the first 200
hits (the second parameter to search) by accessing the scoreDocs array
(consult its length, may be less than 200 hits!). The maximum number of
possible hits (may be > 200!) is also accessible by TopDocs for
statistics/user display.
If you want to iterate over all hits, use a Collector. In all other cases
use the simple TopDocs based collecting. And sorting is done by the
collector, Lucene has no idea how to sort. If you use Sort, the returned
TopDocs will be sorted.
If you do not sort at all and do not score your results, TopDocs is not very
useful, because the first 200 hits cannot be ranked.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michel Nadeau [mailto:akaris@gmail.com]
> Sent: Monday, November 30, 2009 5:35 PM
> To: java-user@lucene.apache.org
> Subject: Re: Performance problems with Lucene 2.9
>
> I'll definitely switch to a Collector.
>
> It's just not clear for me if I should use BooleanQueries or
> MatchAllDocuments+Filters ?
>
> And should I write my own collector or the TopDocs one is perfect for me ?
>
> - Mike
> akaris@gmail.com
>
>
> On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson
> <er...@gmail.com>wrote:
>
> > The problem with hits is that a it re-executes the query
> > every N documents where N is 100 (?).
> >
> > So, a loop like
> > for (int idx : hits.length) {
> > do something....
> > }
> >
> > Assuming my memory is right and it's every 100, your query will
> > re-execute (length/100) times. Which is unfortunate.
> >
> > The very quick test to see where to concentrate first would be to take
> > a time stamp just before you hit your loop.....
> >
> > This will tell you whether this loop is the culprit, but it really
> doesn't
> > matter because you'll follow the advice from Uwe and Shai anyway <G>.
> >
> > Filtering and Sorting are applied to Collectors before you see them.....
> >
> > The other bit would be to investigate your sorting. Remember that the
> > first sort or two take quite a while since the relevant caches are
> > populated with first used, so second+ queries should be faster. The
> > Wiki has some timing/speedup advice.....
> >
> > Best
> > Erick
> >
> >
> > On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <ak...@gmail.com>
> wrote:
> >
> > > What is the main difference between Hits and Collectors?
> > >
> > > - Mike
> > > akaris@gmail.com
> > >
> > >
> > > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de>
> wrote:
> > >
> > > > And if you only have a filter and apply it to all documents, make a
> > > > ConstantScoreQuery on top of the filter:
> > > >
> > > > Query q=new ConstantScoreQuery(cluCF);
> > > >
> > > > Then remove the filter from your search method call and only execute
> > this
> > > > query.
> > > >
> > > > And if you iterate over all results never-ever use Hits! (its
> already
> > > > deprecated). Write a Collector instead (as you are not interested in
> > > > scoring).
> > > >
> > > > And: If you replace a relational database with Lucene, be sure not
> to
> > > think
> > > > in a relational sense with foreign keys / primary keys and so on. In
> > > > general
> > > > you should flatten everything.
> > > >
> > > > Uwe
> > > >
> > > > -----
> > > > Uwe Schindler
> > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > http://www.thetaphi.de
> > > > eMail: uwe@thetaphi.de
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > > Sent: Monday, November 30, 2009 4:56 PM
> > > > > To: java-user@lucene.apache.org
> > > > > Subject: Re: Performance problems with Lucene 2.9
> > > > >
> > > > > Hi
> > > > >
> > > > > First you can use MatchAllDocsQuery, which matches all documents.
> It
> > > will
> > > > > save a HUGE posting list (TAG:TAG), and performs much faster. For
> > > example
> > > > > TAG:TAG computes a score for each doc, even though you don't need
> it.
> > > > > MatchAllDocsQuery doesn't.
> > > > >
> > > > > Second, move away from Hits ! :) Use Collectors instead.
> > > > >
> > > > > If I understand the chain of filters, do you think you can code
> them
> > > with
> > > > > a
> > > > > BooleanQuery that is added BooleanClauses, each with is Term
> > > > > (field:value)?
> > > > > You can add clauses w/ OR, AND, NOT etc.
> > > > >
> > > > > Note that in Lucene 2.9, you can avoid scoring documents very
> easily,
> > > > > which
> > > > > is a performance win if you don't need scores (i.e. if you just
> want
> > to
> > > > > match everything, not caring for scores).
> > > > >
> > > > > Shai
> > > > >
> > > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > we use Lucene to store around 300 millions of records. We use
> the
> > > index
> > > > > > both
> > > > > > for conventional searching, but also for all the system's data -
> we
> > > > > > replaced
> > > > > > MySQL with Lucene because it was simply not working at all with
> > MySQL
> > > > > due
> > > > > > to
> > > > > > the amount or records. Our problem is that we have HUGE
> performance
> > > > > > problems... whenever we search, it takes forever to return
> results,
> > > and
> > > > > > Java
> > > > > > uses 100% CPU/RAM.
> > > > > >
> > > > > > Our index fields are like this:
> > > > > >
> > > > > > TYPE
> > > > > > PK
> > > > > > FOREIGN_PK
> > > > > > TAG
> > > > > > ...other information depending on type...
> > > > > >
> > > > > > * All fields are Field.Index.UN_TOKENIZED
> > > > > > * The field "TAG" always contains the value "TAG".
> > > > > >
> > > > > > Whenever we search in the index, our query is "TAG:TAG" to match
> > all
> > > > > > documents, and we do the search like this:
> > > > > >
> > > > > > // Search
> > > > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > > > >
> > > > > > cluCF is a ChainedFilter containing all the other filters (like
> > > > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > > > >
> > > > > > I know that the method is probably crazy because "TAG:TAG" is
> > > matching
> > > > > all
> > > > > > 300M documents and then it applies filters; so that's probably
> why
> > > > every
> > > > > > little query is taking 100% CPU/RAM.... but I don't know how to
> do
> > it
> > > > > > properly.
> > > > > >
> > > > > > Help ! Any advice is welcome.
> > > > > >
> > > > > > - Mike
> > > > > > akaris@gmail.com
> > > > > >
> > > >
> > > >
> > > > --------------------------------------------------------------------
> -
> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Performance problems with Lucene 2.9
Posted by Ian Lea <ia...@gmail.com>.
Since you are just interested in retrieving the top n hits, sounds to
me that TopDocs is the way to go. It's not a drop-in replacement for
Hits but the transition is pretty straightforward.
Then if MatchAllDocsQuery + filters gives you good enough performance
you could stop, if it doesn't look at Boolean queries.
And of course do take note of the comments about sorting.
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed is well worth a look.
--
Ian.
On Mon, Nov 30, 2009 at 4:34 PM, Michel Nadeau <ak...@gmail.com> wrote:
> I'll definitely switch to a Collector.
>
> It's just not clear for me if I should use BooleanQueries or
> MatchAllDocuments+Filters ?
>
> And should I write my own collector or the TopDocs one is perfect for me ?
>
> - Mike
> akaris@gmail.com
>
>
> On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson <er...@gmail.com>wrote:
>
>> The problem with hits is that a it re-executes the query
>> every N documents where N is 100 (?).
>>
>> So, a loop like
>> for (int idx : hits.length) {
>> do something....
>> }
>>
>> Assuming my memory is right and it's every 100, your query will
>> re-execute (length/100) times. Which is unfortunate.
>>
>> The very quick test to see where to concentrate first would be to take
>> a time stamp just before you hit your loop.....
>>
>> This will tell you whether this loop is the culprit, but it really doesn't
>> matter because you'll follow the advice from Uwe and Shai anyway <G>.
>>
>> Filtering and Sorting are applied to Collectors before you see them.....
>>
>> The other bit would be to investigate your sorting. Remember that the
>> first sort or two take quite a while since the relevant caches are
>> populated with first used, so second+ queries should be faster. The
>> Wiki has some timing/speedup advice.....
>>
>> Best
>> Erick
>>
>>
>> On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <ak...@gmail.com> wrote:
>>
>> > What is the main difference between Hits and Collectors?
>> >
>> > - Mike
>> > akaris@gmail.com
>> >
>> >
>> > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>> >
>> > > And if you only have a filter and apply it to all documents, make a
>> > > ConstantScoreQuery on top of the filter:
>> > >
>> > > Query q=new ConstantScoreQuery(cluCF);
>> > >
>> > > Then remove the filter from your search method call and only execute
>> this
>> > > query.
>> > >
>> > > And if you iterate over all results never-ever use Hits! (its already
>> > > deprecated). Write a Collector instead (as you are not interested in
>> > > scoring).
>> > >
>> > > And: If you replace a relational database with Lucene, be sure not to
>> > think
>> > > in a relational sense with foreign keys / primary keys and so on. In
>> > > general
>> > > you should flatten everything.
>> > >
>> > > Uwe
>> > >
>> > > -----
>> > > Uwe Schindler
>> > > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > > http://www.thetaphi.de
>> > > eMail: uwe@thetaphi.de
>> > >
>> > >
>> > > > -----Original Message-----
>> > > > From: Shai Erera [mailto:serera@gmail.com]
>> > > > Sent: Monday, November 30, 2009 4:56 PM
>> > > > To: java-user@lucene.apache.org
>> > > > Subject: Re: Performance problems with Lucene 2.9
>> > > >
>> > > > Hi
>> > > >
>> > > > First you can use MatchAllDocsQuery, which matches all documents. It
>> > will
>> > > > save a HUGE posting list (TAG:TAG), and performs much faster. For
>> > example
>> > > > TAG:TAG computes a score for each doc, even though you don't need it.
>> > > > MatchAllDocsQuery doesn't.
>> > > >
>> > > > Second, move away from Hits ! :) Use Collectors instead.
>> > > >
>> > > > If I understand the chain of filters, do you think you can code them
>> > with
>> > > > a
>> > > > BooleanQuery that is added BooleanClauses, each with is Term
>> > > > (field:value)?
>> > > > You can add clauses w/ OR, AND, NOT etc.
>> > > >
>> > > > Note that in Lucene 2.9, you can avoid scoring documents very easily,
>> > > > which
>> > > > is a performance win if you don't need scores (i.e. if you just want
>> to
>> > > > match everything, not caring for scores).
>> > > >
>> > > > Shai
>> > > >
>> > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com>
>> > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > we use Lucene to store around 300 millions of records. We use the
>> > index
>> > > > > both
>> > > > > for conventional searching, but also for all the system's data - we
>> > > > > replaced
>> > > > > MySQL with Lucene because it was simply not working at all with
>> MySQL
>> > > > due
>> > > > > to
>> > > > > the amount or records. Our problem is that we have HUGE performance
>> > > > > problems... whenever we search, it takes forever to return results,
>> > and
>> > > > > Java
>> > > > > uses 100% CPU/RAM.
>> > > > >
>> > > > > Our index fields are like this:
>> > > > >
>> > > > > TYPE
>> > > > > PK
>> > > > > FOREIGN_PK
>> > > > > TAG
>> > > > > ...other information depending on type...
>> > > > >
>> > > > > * All fields are Field.Index.UN_TOKENIZED
>> > > > > * The field "TAG" always contains the value "TAG".
>> > > > >
>> > > > > Whenever we search in the index, our query is "TAG:TAG" to match
>> all
>> > > > > documents, and we do the search like this:
>> > > > >
>> > > > > // Search
>> > > > > Hits h = searcher.search(q, cluCF, cluSort);
>> > > > >
>> > > > > cluCF is a ChainedFilter containing all the other filters (like
>> > > > > FOREIGN_PK=12345, TYPE=a, etc.).
>> > > > >
>> > > > > I know that the method is probably crazy because "TAG:TAG" is
>> > matching
>> > > > all
>> > > > > 300M documents and then it applies filters; so that's probably why
>> > > every
>> > > > > little query is taking 100% CPU/RAM.... but I don't know how to do
>> it
>> > > > > properly.
>> > > > >
>> > > > > Help ! Any advice is welcome.
>> > > > >
>> > > > > - Mike
>> > > > > akaris@gmail.com
>> > > > >
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > > For additional commands, e-mail: java-user-help@lucene.apache.org
>> > >
>> > >
>> >
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Performance problems with Lucene 2.9
Posted by Michel Nadeau <ak...@gmail.com>.
I'll definitely switch to a Collector.
It's just not clear for me if I should use BooleanQueries or
MatchAllDocuments+Filters ?
And should I write my own collector or the TopDocs one is perfect for me ?
- Mike
akaris@gmail.com
On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson <er...@gmail.com>wrote:
> The problem with hits is that a it re-executes the query
> every N documents where N is 100 (?).
>
> So, a loop like
> for (int idx : hits.length) {
> do something....
> }
>
> Assuming my memory is right and it's every 100, your query will
> re-execute (length/100) times. Which is unfortunate.
>
> The very quick test to see where to concentrate first would be to take
> a time stamp just before you hit your loop.....
>
> This will tell you whether this loop is the culprit, but it really doesn't
> matter because you'll follow the advice from Uwe and Shai anyway <G>.
>
> Filtering and Sorting are applied to Collectors before you see them.....
>
> The other bit would be to investigate your sorting. Remember that the
> first sort or two take quite a while since the relevant caches are
> populated with first used, so second+ queries should be faster. The
> Wiki has some timing/speedup advice.....
>
> Best
> Erick
>
>
> On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <ak...@gmail.com> wrote:
>
> > What is the main difference between Hits and Collectors?
> >
> > - Mike
> > akaris@gmail.com
> >
> >
> > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> >
> > > And if you only have a filter and apply it to all documents, make a
> > > ConstantScoreQuery on top of the filter:
> > >
> > > Query q=new ConstantScoreQuery(cluCF);
> > >
> > > Then remove the filter from your search method call and only execute
> this
> > > query.
> > >
> > > And if you iterate over all results never-ever use Hits! (its already
> > > deprecated). Write a Collector instead (as you are not interested in
> > > scoring).
> > >
> > > And: If you replace a relational database with Lucene, be sure not to
> > think
> > > in a relational sense with foreign keys / primary keys and so on. In
> > > general
> > > you should flatten everything.
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > > > -----Original Message-----
> > > > From: Shai Erera [mailto:serera@gmail.com]
> > > > Sent: Monday, November 30, 2009 4:56 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: Performance problems with Lucene 2.9
> > > >
> > > > Hi
> > > >
> > > > First you can use MatchAllDocsQuery, which matches all documents. It
> > will
> > > > save a HUGE posting list (TAG:TAG), and performs much faster. For
> > example
> > > > TAG:TAG computes a score for each doc, even though you don't need it.
> > > > MatchAllDocsQuery doesn't.
> > > >
> > > > Second, move away from Hits ! :) Use Collectors instead.
> > > >
> > > > If I understand the chain of filters, do you think you can code them
> > with
> > > > a
> > > > BooleanQuery that is added BooleanClauses, each with is Term
> > > > (field:value)?
> > > > You can add clauses w/ OR, AND, NOT etc.
> > > >
> > > > Note that in Lucene 2.9, you can avoid scoring documents very easily,
> > > > which
> > > > is a performance win if you don't need scores (i.e. if you just want
> to
> > > > match everything, not caring for scores).
> > > >
> > > > Shai
> > > >
> > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > we use Lucene to store around 300 millions of records. We use the
> > index
> > > > > both
> > > > > for conventional searching, but also for all the system's data - we
> > > > > replaced
> > > > > MySQL with Lucene because it was simply not working at all with
> MySQL
> > > > due
> > > > > to
> > > > > the amount or records. Our problem is that we have HUGE performance
> > > > > problems... whenever we search, it takes forever to return results,
> > and
> > > > > Java
> > > > > uses 100% CPU/RAM.
> > > > >
> > > > > Our index fields are like this:
> > > > >
> > > > > TYPE
> > > > > PK
> > > > > FOREIGN_PK
> > > > > TAG
> > > > > ...other information depending on type...
> > > > >
> > > > > * All fields are Field.Index.UN_TOKENIZED
> > > > > * The field "TAG" always contains the value "TAG".
> > > > >
> > > > > Whenever we search in the index, our query is "TAG:TAG" to match
> all
> > > > > documents, and we do the search like this:
> > > > >
> > > > > // Search
> > > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > > >
> > > > > cluCF is a ChainedFilter containing all the other filters (like
> > > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > > >
> > > > > I know that the method is probably crazy because "TAG:TAG" is
> > matching
> > > > all
> > > > > 300M documents and then it applies filters; so that's probably why
> > > every
> > > > > little query is taking 100% CPU/RAM.... but I don't know how to do
> it
> > > > > properly.
> > > > >
> > > > > Help ! Any advice is welcome.
> > > > >
> > > > > - Mike
> > > > > akaris@gmail.com
> > > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>
Re: Performance problems with Lucene 2.9
Posted by Erick Erickson <er...@gmail.com>.
The problem with hits is that a it re-executes the query
every N documents where N is 100 (?).
So, a loop like
for (int idx : hits.length) {
do something....
}
Assuming my memory is right and it's every 100, your query will
re-execute (length/100) times. Which is unfortunate.
The very quick test to see where to concentrate first would be to take
a time stamp just before you hit your loop.....
This will tell you whether this loop is the culprit, but it really doesn't
matter because you'll follow the advice from Uwe and Shai anyway <G>.
Filtering and Sorting are applied to Collectors before you see them.....
The other bit would be to investigate your sorting. Remember that the
first sort or two take quite a while since the relevant caches are
populated with first used, so second+ queries should be faster. The
Wiki has some timing/speedup advice.....
Best
Erick
On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <ak...@gmail.com> wrote:
> What is the main difference between Hits and Collectors?
>
> - Mike
> akaris@gmail.com
>
>
> On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
>
> > And if you only have a filter and apply it to all documents, make a
> > ConstantScoreQuery on top of the filter:
> >
> > Query q=new ConstantScoreQuery(cluCF);
> >
> > Then remove the filter from your search method call and only execute this
> > query.
> >
> > And if you iterate over all results never-ever use Hits! (its already
> > deprecated). Write a Collector instead (as you are not interested in
> > scoring).
> >
> > And: If you replace a relational database with Lucene, be sure not to
> think
> > in a relational sense with foreign keys / primary keys and so on. In
> > general
> > you should flatten everything.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Shai Erera [mailto:serera@gmail.com]
> > > Sent: Monday, November 30, 2009 4:56 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: Performance problems with Lucene 2.9
> > >
> > > Hi
> > >
> > > First you can use MatchAllDocsQuery, which matches all documents. It
> will
> > > save a HUGE posting list (TAG:TAG), and performs much faster. For
> example
> > > TAG:TAG computes a score for each doc, even though you don't need it.
> > > MatchAllDocsQuery doesn't.
> > >
> > > Second, move away from Hits ! :) Use Collectors instead.
> > >
> > > If I understand the chain of filters, do you think you can code them
> with
> > > a
> > > BooleanQuery that is added BooleanClauses, each with is Term
> > > (field:value)?
> > > You can add clauses w/ OR, AND, NOT etc.
> > >
> > > Note that in Lucene 2.9, you can avoid scoring documents very easily,
> > > which
> > > is a performance win if you don't need scores (i.e. if you just want to
> > > match everything, not caring for scores).
> > >
> > > Shai
> > >
> > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > we use Lucene to store around 300 millions of records. We use the
> index
> > > > both
> > > > for conventional searching, but also for all the system's data - we
> > > > replaced
> > > > MySQL with Lucene because it was simply not working at all with MySQL
> > > due
> > > > to
> > > > the amount or records. Our problem is that we have HUGE performance
> > > > problems... whenever we search, it takes forever to return results,
> and
> > > > Java
> > > > uses 100% CPU/RAM.
> > > >
> > > > Our index fields are like this:
> > > >
> > > > TYPE
> > > > PK
> > > > FOREIGN_PK
> > > > TAG
> > > > ...other information depending on type...
> > > >
> > > > * All fields are Field.Index.UN_TOKENIZED
> > > > * The field "TAG" always contains the value "TAG".
> > > >
> > > > Whenever we search in the index, our query is "TAG:TAG" to match all
> > > > documents, and we do the search like this:
> > > >
> > > > // Search
> > > > Hits h = searcher.search(q, cluCF, cluSort);
> > > >
> > > > cluCF is a ChainedFilter containing all the other filters (like
> > > > FOREIGN_PK=12345, TYPE=a, etc.).
> > > >
> > > > I know that the method is probably crazy because "TAG:TAG" is
> matching
> > > all
> > > > 300M documents and then it applies filters; so that's probably why
> > every
> > > > little query is taking 100% CPU/RAM.... but I don't know how to do it
> > > > properly.
> > > >
> > > > Help ! Any advice is welcome.
> > > >
> > > > - Mike
> > > > akaris@gmail.com
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
Re: Performance problems with Lucene 2.9
Posted by Michel Nadeau <ak...@gmail.com>.
What is the main difference between Hits and Collectors?
- Mike
akaris@gmail.com
On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> And if you only have a filter and apply it to all documents, make a
> ConstantScoreQuery on top of the filter:
>
> Query q=new ConstantScoreQuery(cluCF);
>
> Then remove the filter from your search method call and only execute this
> query.
>
> And if you iterate over all results never-ever use Hits! (its already
> deprecated). Write a Collector instead (as you are not interested in
> scoring).
>
> And: If you replace a relational database with Lucene, be sure not to think
> in a relational sense with foreign keys / primary keys and so on. In
> general
> you should flatten everything.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Shai Erera [mailto:serera@gmail.com]
> > Sent: Monday, November 30, 2009 4:56 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Performance problems with Lucene 2.9
> >
> > Hi
> >
> > First you can use MatchAllDocsQuery, which matches all documents. It will
> > save a HUGE posting list (TAG:TAG), and performs much faster. For example
> > TAG:TAG computes a score for each doc, even though you don't need it.
> > MatchAllDocsQuery doesn't.
> >
> > Second, move away from Hits ! :) Use Collectors instead.
> >
> > If I understand the chain of filters, do you think you can code them with
> > a
> > BooleanQuery that is added BooleanClauses, each with is Term
> > (field:value)?
> > You can add clauses w/ OR, AND, NOT etc.
> >
> > Note that in Lucene 2.9, you can avoid scoring documents very easily,
> > which
> > is a performance win if you don't need scores (i.e. if you just want to
> > match everything, not caring for scores).
> >
> > Shai
> >
> > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > we use Lucene to store around 300 millions of records. We use the index
> > > both
> > > for conventional searching, but also for all the system's data - we
> > > replaced
> > > MySQL with Lucene because it was simply not working at all with MySQL
> > due
> > > to
> > > the amount or records. Our problem is that we have HUGE performance
> > > problems... whenever we search, it takes forever to return results, and
> > > Java
> > > uses 100% CPU/RAM.
> > >
> > > Our index fields are like this:
> > >
> > > TYPE
> > > PK
> > > FOREIGN_PK
> > > TAG
> > > ...other information depending on type...
> > >
> > > * All fields are Field.Index.UN_TOKENIZED
> > > * The field "TAG" always contains the value "TAG".
> > >
> > > Whenever we search in the index, our query is "TAG:TAG" to match all
> > > documents, and we do the search like this:
> > >
> > > // Search
> > > Hits h = searcher.search(q, cluCF, cluSort);
> > >
> > > cluCF is a ChainedFilter containing all the other filters (like
> > > FOREIGN_PK=12345, TYPE=a, etc.).
> > >
> > > I know that the method is probably crazy because "TAG:TAG" is matching
> > all
> > > 300M documents and then it applies filters; so that's probably why
> every
> > > little query is taking 100% CPU/RAM.... but I don't know how to do it
> > > properly.
> > >
> > > Help ! Any advice is welcome.
> > >
> > > - Mike
> > > akaris@gmail.com
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Performance problems with Lucene 2.9
Posted by Uwe Schindler <uw...@thetaphi.de>.
And if you only have a filter and apply it to all documents, make a
ConstantScoreQuery on top of the filter:
Query q=new ConstantScoreQuery(cluCF);
Then remove the filter from your search method call and only execute this
query.
And if you iterate over all results never-ever use Hits! (its already
deprecated). Write a Collector instead (as you are not interested in
scoring).
And: If you replace a relational database with Lucene, be sure not to think
in a relational sense with foreign keys / primary keys and so on. In general
you should flatten everything.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Shai Erera [mailto:serera@gmail.com]
> Sent: Monday, November 30, 2009 4:56 PM
> To: java-user@lucene.apache.org
> Subject: Re: Performance problems with Lucene 2.9
>
> Hi
>
> First you can use MatchAllDocsQuery, which matches all documents. It will
> save a HUGE posting list (TAG:TAG), and performs much faster. For example
> TAG:TAG computes a score for each doc, even though you don't need it.
> MatchAllDocsQuery doesn't.
>
> Second, move away from Hits ! :) Use Collectors instead.
>
> If I understand the chain of filters, do you think you can code them with
> a
> BooleanQuery that is added BooleanClauses, each with is Term
> (field:value)?
> You can add clauses w/ OR, AND, NOT etc.
>
> Note that in Lucene 2.9, you can avoid scoring documents very easily,
> which
> is a performance win if you don't need scores (i.e. if you just want to
> match everything, not caring for scores).
>
> Shai
>
> On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com> wrote:
>
> > Hi,
> >
> > we use Lucene to store around 300 millions of records. We use the index
> > both
> > for conventional searching, but also for all the system's data - we
> > replaced
> > MySQL with Lucene because it was simply not working at all with MySQL
> due
> > to
> > the amount or records. Our problem is that we have HUGE performance
> > problems... whenever we search, it takes forever to return results, and
> > Java
> > uses 100% CPU/RAM.
> >
> > Our index fields are like this:
> >
> > TYPE
> > PK
> > FOREIGN_PK
> > TAG
> > ...other information depending on type...
> >
> > * All fields are Field.Index.UN_TOKENIZED
> > * The field "TAG" always contains the value "TAG".
> >
> > Whenever we search in the index, our query is "TAG:TAG" to match all
> > documents, and we do the search like this:
> >
> > // Search
> > Hits h = searcher.search(q, cluCF, cluSort);
> >
> > cluCF is a ChainedFilter containing all the other filters (like
> > FOREIGN_PK=12345, TYPE=a, etc.).
> >
> > I know that the method is probably crazy because "TAG:TAG" is matching
> all
> > 300M documents and then it applies filters; so that's probably why every
> > little query is taking 100% CPU/RAM.... but I don't know how to do it
> > properly.
> >
> > Help ! Any advice is welcome.
> >
> > - Mike
> > akaris@gmail.com
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Performance problems with Lucene 2.9
Posted by Michel Nadeau <ak...@gmail.com>.
Hi !
Thanks so much !!
* I'll check the documentation for MatchAllDocsQuery.
* I'm already changing my code to create BooleanQueries instead of filters -
is that better than MatchAllDocsQuery or it's the same?
* Is using MatchAllDocsQuery the only way to disable scoring?
* Would you have any good example of how to use Collectors instead of Hits?
- Mike
akaris@gmail.com
On Mon, Nov 30, 2009 at 10:56 AM, Shai Erera <se...@gmail.com> wrote:
> Hi
>
> First you can use MatchAllDocsQuery, which matches all documents. It will
> save a HUGE posting list (TAG:TAG), and performs much faster. For example
> TAG:TAG computes a score for each doc, even though you don't need it.
> MatchAllDocsQuery doesn't.
>
> Second, move away from Hits ! :) Use Collectors instead.
>
> If I understand the chain of filters, do you think you can code them with a
> BooleanQuery that is added BooleanClauses, each with is Term (field:value)?
> You can add clauses w/ OR, AND, NOT etc.
>
> Note that in Lucene 2.9, you can avoid scoring documents very easily, which
> is a performance win if you don't need scores (i.e. if you just want to
> match everything, not caring for scores).
>
> Shai
>
> On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com> wrote:
>
> > Hi,
> >
> > we use Lucene to store around 300 millions of records. We use the index
> > both
> > for conventional searching, but also for all the system's data - we
> > replaced
> > MySQL with Lucene because it was simply not working at all with MySQL due
> > to
> > the amount or records. Our problem is that we have HUGE performance
> > problems... whenever we search, it takes forever to return results, and
> > Java
> > uses 100% CPU/RAM.
> >
> > Our index fields are like this:
> >
> > TYPE
> > PK
> > FOREIGN_PK
> > TAG
> > ...other information depending on type...
> >
> > * All fields are Field.Index.UN_TOKENIZED
> > * The field "TAG" always contains the value "TAG".
> >
> > Whenever we search in the index, our query is "TAG:TAG" to match all
> > documents, and we do the search like this:
> >
> > // Search
> > Hits h = searcher.search(q, cluCF, cluSort);
> >
> > cluCF is a ChainedFilter containing all the other filters (like
> > FOREIGN_PK=12345, TYPE=a, etc.).
> >
> > I know that the method is probably crazy because "TAG:TAG" is matching
> all
> > 300M documents and then it applies filters; so that's probably why every
> > little query is taking 100% CPU/RAM.... but I don't know how to do it
> > properly.
> >
> > Help ! Any advice is welcome.
> >
> > - Mike
> > akaris@gmail.com
> >
>
Re: Performance problems with Lucene 2.9
Posted by Shai Erera <se...@gmail.com>.
Hi
First you can use MatchAllDocsQuery, which matches all documents. It will
save a HUGE posting list (TAG:TAG), and performs much faster. For example
TAG:TAG computes a score for each doc, even though you don't need it.
MatchAllDocsQuery doesn't.
Second, move away from Hits ! :) Use Collectors instead.
If I understand the chain of filters, do you think you can code them with a
BooleanQuery that is added BooleanClauses, each with is Term (field:value)?
You can add clauses w/ OR, AND, NOT etc.
Note that in Lucene 2.9, you can avoid scoring documents very easily, which
is a performance win if you don't need scores (i.e. if you just want to
match everything, not caring for scores).
Shai
On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <ak...@gmail.com> wrote:
> Hi,
>
> we use Lucene to store around 300 millions of records. We use the index
> both
> for conventional searching, but also for all the system's data - we
> replaced
> MySQL with Lucene because it was simply not working at all with MySQL due
> to
> the amount or records. Our problem is that we have HUGE performance
> problems... whenever we search, it takes forever to return results, and
> Java
> uses 100% CPU/RAM.
>
> Our index fields are like this:
>
> TYPE
> PK
> FOREIGN_PK
> TAG
> ...other information depending on type...
>
> * All fields are Field.Index.UN_TOKENIZED
> * The field "TAG" always contains the value "TAG".
>
> Whenever we search in the index, our query is "TAG:TAG" to match all
> documents, and we do the search like this:
>
> // Search
> Hits h = searcher.search(q, cluCF, cluSort);
>
> cluCF is a ChainedFilter containing all the other filters (like
> FOREIGN_PK=12345, TYPE=a, etc.).
>
> I know that the method is probably crazy because "TAG:TAG" is matching all
> 300M documents and then it applies filters; so that's probably why every
> little query is taking 100% CPU/RAM.... but I don't know how to do it
> properly.
>
> Help ! Any advice is welcome.
>
> - Mike
> akaris@gmail.com
>