You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Christian Robert <cr...@arcor.de> on 2009/10/01 13:17:07 UTC
Pagination and Sorting
Hello everybody,
I'm looking at quite an interesting challenge right now, so I
hope that somebody out there will be able to assist me.
What I'm trying to do is returning search results both sorted and
paginated. So far I haven't been able to come up with a working solution.
Pagination without sorting is no problem - simply looping through the
document identifiers and grabbing the documents within my "pagination
window".
Sorting is also no problem - create the Sort object and executing the
search.
However, when using a Sort I always get a TopDocs object in which all
the identifiers of the documents are contained. No problem when dealing
with a small index, but I've some 700.000 documents indexed and each
time I'm trying to call
TopDocs topDocs =
indexSearcher.search(query, null, Integer.MAX_VALUE, sort);
I'm ending up with an OutOfMemoryError.
So it seems that Lucene needs all the documents loaded into memory for
the sorting to work where in contrast I only want to load the documents
to be displayed in the current pagination window.
Am I lost here and have to find another way or is there a working
solution to combine sorting and pagination?
Thanks in advance!
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Pagination and Sorting
Posted by Uwe Schindler <uw...@thetaphi.de>.
But a collector will not output the documents in sorted order...
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Anshum [mailto:anshumg@gmail.com]
> Sent: Thursday, October 01, 2009 1:58 PM
> To: java-user@lucene.apache.org
> Subject: Re: Pagination and Sorting
>
> Hey Christian,
> Try what I wrote in the last reply. Would work absolutely fine. Have
> tested
> that for very large datasets.
> --
> Anshum Gupta
> Naukri Labs!
> http://ai-cafe.blogspot.com
>
> The facts expressed here belong to everybody, the opinions to me. The
> distinction is yours to draw............
>
>
> On Thu, Oct 1, 2009 at 5:23 PM, Christian Robert <cr...@arcor.de>
> wrote:
>
> > Uwe,
> >
> > > You are using TopDocs incorrectly. Normally you use *not*
> > Integer.MAX_VALUE,
> > > as the upper bound of your pagination window as numer of documents. So
> if
> > > user wants to display documents 90 to 100, just set the number to 100
> > docs.
> > > If the user then goes to docs 100 to 110, just reexecute ther query
> with
> > a
> > > larger value. [...] If somebody goes further, just raise and reexecute
> > query.
> >
> > That's what I thought of as workaround for the first pages. But at
> > some point the user might want to see the last page of the results,
> > so eventually I will have to call
> >
> > TopDocs topDocs = indexSearcher.search(query, null, 300500, sort);
> >
> > and still run into an error. Right now I'd present the user with an
> > error message like "The search returned too many results, please further
> > restrict your search" but a more generic solution would sound a lot
> > better ;)
> >
> > Chris
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Pagination and Sorting
Posted by Anshum <an...@gmail.com>.
Hey Christian,
Try what I wrote in the last reply. Would work absolutely fine. Have tested
that for very large datasets.
--
Anshum Gupta
Naukri Labs!
http://ai-cafe.blogspot.com
The facts expressed here belong to everybody, the opinions to me. The
distinction is yours to draw............
On Thu, Oct 1, 2009 at 5:23 PM, Christian Robert <cr...@arcor.de> wrote:
> Uwe,
>
> > You are using TopDocs incorrectly. Normally you use *not*
> Integer.MAX_VALUE,
> > as the upper bound of your pagination window as numer of documents. So if
> > user wants to display documents 90 to 100, just set the number to 100
> docs.
> > If the user then goes to docs 100 to 110, just reexecute ther query with
> a
> > larger value. [...] If somebody goes further, just raise and reexecute
> query.
>
> That's what I thought of as workaround for the first pages. But at
> some point the user might want to see the last page of the results,
> so eventually I will have to call
>
> TopDocs topDocs = indexSearcher.search(query, null, 300500, sort);
>
> and still run into an error. Right now I'd present the user with an
> error message like "The search returned too many results, please further
> restrict your search" but a more generic solution would sound a lot
> better ;)
>
> Chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Pagination and Sorting
Posted by Uwe Schindler <uw...@thetaphi.de>.
I forgot to mention: Because of this, e.g. even Google (who do not use
Lucene :-]) does not let you go beyond a limit to a very large page number.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Thursday, October 01, 2009 2:25 PM
> To: java-user@lucene.apache.org
> Subject: RE: Pagination and Sorting
>
> Hi Chris,
>
> > Uwe,
> >
> > > You are using TopDocs incorrectly. Normally you use *not*
> > Integer.MAX_VALUE,
> > > as the upper bound of your pagination window as numer of documents. So
> > if
> > > user wants to display documents 90 to 100, just set the number to 100
> > docs.
> > > If the user then goes to docs 100 to 110, just reexecute ther query
> with
> > a
> > > larger value. [...] If somebody goes further, just raise and reexecute
> > query.
> >
> > That's what I thought of as workaround for the first pages.
>
> That's not a workaround, it is designed to work like this!
>
> > But at
> > some point the user might want to see the last page of the results,
> > so eventually I will have to call
> >
> > TopDocs topDocs = indexSearcher.search(query, null, 300500, sort);
> >
> > and still run into an error. Right now I'd present the user with an
> > error message like "The search returned too many results, please further
> > restrict your search" but a more generic solution would sound a lot
> > better ;)
>
> How about sorting backwards in this case (and print out the page in
> reverse
> to get a forward-printed page again)?
>
> Uwe
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Pagination and Sorting
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Chris,
> Uwe,
>
> > You are using TopDocs incorrectly. Normally you use *not*
> Integer.MAX_VALUE,
> > as the upper bound of your pagination window as numer of documents. So
> if
> > user wants to display documents 90 to 100, just set the number to 100
> docs.
> > If the user then goes to docs 100 to 110, just reexecute ther query with
> a
> > larger value. [...] If somebody goes further, just raise and reexecute
> query.
>
> That's what I thought of as workaround for the first pages.
That's not a workaround, it is designed to work like this!
> But at
> some point the user might want to see the last page of the results,
> so eventually I will have to call
>
> TopDocs topDocs = indexSearcher.search(query, null, 300500, sort);
>
> and still run into an error. Right now I'd present the user with an
> error message like "The search returned too many results, please further
> restrict your search" but a more generic solution would sound a lot
> better ;)
How about sorting backwards in this case (and print out the page in reverse
to get a forward-printed page again)?
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Pagination and Sorting
Posted by Christian Robert <cr...@arcor.de>.
Uwe,
> You are using TopDocs incorrectly. Normally you use *not* Integer.MAX_VALUE,
> as the upper bound of your pagination window as numer of documents. So if
> user wants to display documents 90 to 100, just set the number to 100 docs.
> If the user then goes to docs 100 to 110, just reexecute ther query with a
> larger value. [...] If somebody goes further, just raise and reexecute query.
That's what I thought of as workaround for the first pages. But at
some point the user might want to see the last page of the results,
so eventually I will have to call
TopDocs topDocs = indexSearcher.search(query, null, 300500, sort);
and still run into an error. Right now I'd present the user with an
error message like "The search returned too many results, please further
restrict your search" but a more generic solution would sound a lot
better ;)
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Pagination and Sorting
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hallo Chris,
You are using TopDocs incorrectly. Normally you use *not* Integer.MAX_VALUE,
as the upper bound of your pagination window as numer of documents. So if
user wants to display documents 90 to 100, just set the number to 100 docs.
If the user then goes to docs 100 to 110, just reexecute ther query with a
larger value. This is exactly how search engines work, user gets the most
relevant docs presented first and then goes forward in pagination, but most
users will not go beyond the first 5 or 10 pages. So an initial max value of
100 is good. If somebody goes further, just raise and reexecute query.
If you want to collect all Hots, do not use TopDocs, instead use a
Collector.
This is the correct way for both sorting and not sorting, only that with
sorting, the memory usage is bigger. Internally an array of
Integer.MAX_VALUE is allocated, which leads to the OOM.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Christian Robert [mailto:cr_usenet@arcor.de]
> Sent: Thursday, October 01, 2009 1:17 PM
> To: java-user@lucene.apache.org
> Subject: Pagination and Sorting
>
> Hello everybody,
>
> I'm looking at quite an interesting challenge right now, so I
> hope that somebody out there will be able to assist me.
>
> What I'm trying to do is returning search results both sorted and
> paginated. So far I haven't been able to come up with a working solution.
>
> Pagination without sorting is no problem - simply looping through the
> document identifiers and grabbing the documents within my "pagination
> window".
>
> Sorting is also no problem - create the Sort object and executing the
> search.
>
> However, when using a Sort I always get a TopDocs object in which all
> the identifiers of the documents are contained. No problem when dealing
> with a small index, but I've some 700.000 documents indexed and each
> time I'm trying to call
>
> TopDocs topDocs =
> indexSearcher.search(query, null, Integer.MAX_VALUE, sort);
>
> I'm ending up with an OutOfMemoryError.
>
> So it seems that Lucene needs all the documents loaded into memory for
> the sorting to work where in contrast I only want to load the documents
> to be displayed in the current pagination window.
>
> Am I lost here and have to find another way or is there a working
> solution to combine sorting and pagination?
>
> Thanks in advance!
> Chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Pagination and Sorting
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Anshum,
That is exactly the same code he is using (only that he does not instantiate
the collector; IndexSearcher.search(query, int) does exactly that
internally :-)
His problem was, that if offset+limit is large or Integer.MAX_VALUE that he
runs out of memory.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Anshum [mailto:anshumg@gmail.com]
> Sent: Thursday, October 01, 2009 2:31 PM
> To: java-user@lucene.apache.org
> Subject: Re: Pagination and Sorting
>
> @Christian : Which version of Lucene are you using?
>
> For lucene 2.9 this would work.
> *__code snippet__*
> IndexReader r = IndexReader.open("/home/anshum/index/indexname", true);
> IndexSearcher s = new IndexSearcher(r);
> QueryParser qp = new QueryParser("testfield",new StopAnalyzer());
> Query q = qp.parse("test query");
> int offset = 10000;
> int limit = 100;
> Sort sort = new Sort(new SortField("testfield",SortField.DOUBLE));* // You
> may want to change field type to something else*
> TopFieldCollector c = TopFieldCollector.create(sort, offset+limit, true,
> true, true, true);
> s.search(q, c);
> for(int i=offset;i<offset + limit;i++)
> System.out.println("Document : " +
> s.doc(c.topDocs().scoreDocs[i].doc));
>
> *__code snippet ends__*
> --
> Anshum Gupta
> Naukri Labs!
> http://ai-cafe.blogspot.com
>
> The facts expressed here belong to everybody, the opinions to me. The
> distinction is yours to draw............
>
>
> On Thu, Oct 1, 2009 at 5:37 PM, Christian Robert <cr...@arcor.de>
> wrote:
>
> > Anshum,
> >
> > > You could get the hits in a collector and pass the sort to the
> > > collector as it would be the collect function that handles the
> > > sorting.
> > >
> > > searcherObject.search(query,collector);
> > >
> > > Hope that gives you some headway. :)
> >
> > Not quite (yet?) ;-)
> >
> > What do you mean by passing the Sort to the collector as collect
> > function? Could you provide a little code snippet that describes
> > the procedure?
> >
> > Thanks!
> > Chris
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Pagination and Sorting
Posted by Anshum <an...@gmail.com>.
@Christian : Which version of Lucene are you using?
For lucene 2.9 this would work.
*__code snippet__*
IndexReader r = IndexReader.open("/home/anshum/index/indexname", true);
IndexSearcher s = new IndexSearcher(r);
QueryParser qp = new QueryParser("testfield",new StopAnalyzer());
Query q = qp.parse("test query");
int offset = 10000;
int limit = 100;
Sort sort = new Sort(new SortField("testfield",SortField.DOUBLE));* // You
may want to change field type to something else*
TopFieldCollector c = TopFieldCollector.create(sort, offset+limit, true,
true, true, true);
s.search(q, c);
for(int i=offset;i<offset + limit;i++)
System.out.println("Document : " + s.doc(c.topDocs().scoreDocs[i].doc));
*__code snippet ends__*
--
Anshum Gupta
Naukri Labs!
http://ai-cafe.blogspot.com
The facts expressed here belong to everybody, the opinions to me. The
distinction is yours to draw............
On Thu, Oct 1, 2009 at 5:37 PM, Christian Robert <cr...@arcor.de> wrote:
> Anshum,
>
> > You could get the hits in a collector and pass the sort to the
> > collector as it would be the collect function that handles the
> > sorting.
> >
> > searcherObject.search(query,collector);
> >
> > Hope that gives you some headway. :)
>
> Not quite (yet?) ;-)
>
> What do you mean by passing the Sort to the collector as collect
> function? Could you provide a little code snippet that describes
> the procedure?
>
> Thanks!
> Chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Pagination and Sorting
Posted by Christian Robert <cr...@arcor.de>.
Anshum,
> You could get the hits in a collector and pass the sort to the
> collector as it would be the collect function that handles the
> sorting.
>
> searcherObject.search(query,collector);
>
> Hope that gives you some headway. :)
Not quite (yet?) ;-)
What do you mean by passing the Sort to the collector as collect
function? Could you provide a little code snippet that describes
the procedure?
Thanks!
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Pagination and Sorting
Posted by Anshum <an...@gmail.com>.
You could get the hits in a collector and pass the sort to the collector as
it would be the collect function that handles the sorting.
searcherObject.search(query,collector);
Hope that gives you some headway. :)
--
Anshum Gupta
Naukri Labs!
http://ai-cafe.blogspot.com
The facts expressed here belong to everybody, the opinions to me. The
distinction is yours to draw............
On Thu, Oct 1, 2009 at 4:47 PM, Christian Robert <cr...@arcor.de> wrote:
> Hello everybody,
>
> I'm looking at quite an interesting challenge right now, so I
> hope that somebody out there will be able to assist me.
>
> What I'm trying to do is returning search results both sorted and
> paginated. So far I haven't been able to come up with a working solution.
>
> Pagination without sorting is no problem - simply looping through the
> document identifiers and grabbing the documents within my "pagination
> window".
>
> Sorting is also no problem - create the Sort object and executing the
> search.
>
> However, when using a Sort I always get a TopDocs object in which all
> the identifiers of the documents are contained. No problem when dealing
> with a small index, but I've some 700.000 documents indexed and each
> time I'm trying to call
>
> TopDocs topDocs =
> indexSearcher.search(query, null, Integer.MAX_VALUE, sort);
>
> I'm ending up with an OutOfMemoryError.
>
> So it seems that Lucene needs all the documents loaded into memory for
> the sorting to work where in contrast I only want to load the documents
> to be displayed in the current pagination window.
>
> Am I lost here and have to find another way or is there a working
> solution to combine sorting and pagination?
>
> Thanks in advance!
> Chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>