paging and maintaingin a cursor just like ScrollableResultSet

["Hiller, Dean x66079" <de...@broadridge.com>]

As you probably know, using Query in hibernate/JPA gets slower and slower each page since it starts all over on the index tree :( WHILE ScrollableResultSet does NOT because the database maintains a cursor into the index that just picks up where it left off so as you go to the next page, next page, the speed stays linearly the same!!!!

Does something like that exist in solr?

I was looking at the api and all the examples are just for returning all results from what I could tell.

I went into Lucene and it looks like it can do it kind of if you code up your own Collector and unfortunately make the Collector.collect(int doc) block on a lock while waiting for the client to ask for the next page(or ask to release the resource since it is complete).

Ie. ScrollableResultSet obviously has to be closed when complete and so would this method as well.

Any ideas on how to achieve this as my client is a computer not a webapp with a human clicking next page and we want the resultset paging to be linear as it really hurts our performance.

Thanks,
Dean

This message and any attachments are intended only for the use of the addressee and
may contain information that is privileged and confidential. If the reader of the 
message is not the intended recipient or an authorized representative of the
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.

Re: paging and maintaingin a cursor just like ScrollableResultSet

[Michael Sokolov <so...@ifactory.com>]

One technique I've used to page through huge result sets that could 
help: if you have a sortable key (like an id), you can just fetch all 
docs, sorted by the key, and then on subsequent page requests use the 
last value from the previous page as a filter in a range term like:

id:[<last-id> TO *]

where you substitute for <last-id>

there may be a better approach though...

-Mike

On 6/19/2011 6:02 PM, Hiller, Dean x66079 wrote:
> As you probably know, using Query in hibernate/JPA gets slower and slower each page since it starts all over on the index tree :( WHILE ScrollableResultSet does NOT because the database maintains a cursor into the index that just picks up where it left off so as you go to the next page, next page, the speed stays linearly the same!!!!
>
> Does something like that exist in solr?
>
> I was looking at the api and all the examples are just for returning all results from what I could tell.
>
> I went into Lucene and it looks like it can do it kind of if you code up your own Collector and unfortunately make the Collector.collect(int doc) block on a lock while waiting for the client to ask for the next page(or ask to release the resource since it is complete).
>
> Ie. ScrollableResultSet obviously has to be closed when complete and so would this method as well.
>
> Any ideas on how to achieve this as my client is a computer not a webapp with a human clicking next page and we want the resultset paging to be linear as it really hurts our performance.
>
> Thanks,
> Dean
>
> This message and any attachments are intended only for the use of the addressee and
> may contain information that is privileged and confidential. If the reader of the
> message is not the intended recipient or an authorized representative of the
> intended recipient, you are hereby notified that any dissemination of this
> communication is strictly prohibited. If you have received this communication in
> error, please notify us immediately by e-mail and delete the message and any
> attachments from your system.
>
>

Re: paging and maintaingin a cursor just like ScrollableResultSet

[Erick Erickson <er...@gmail.com>]

See the queryResultCache setting in your solrconfig.xml file. For a
given query it keeps a
(configurable) list of results, you could simply make this very large,
at the usual memory
cost.

Best
Erick

On Sun, Jun 19, 2011 at 6:02 PM, Hiller, Dean  x66079
<de...@broadridge.com> wrote:
> As you probably know, using Query in hibernate/JPA gets slower and slower each page since it starts all over on the index tree :( WHILE ScrollableResultSet does NOT because the database maintains a cursor into the index that just picks up where it left off so as you go to the next page, next page, the speed stays linearly the same!!!!
>
> Does something like that exist in solr?
>
> I was looking at the api and all the examples are just for returning all results from what I could tell.
>
> I went into Lucene and it looks like it can do it kind of if you code up your own Collector and unfortunately make the Collector.collect(int doc) block on a lock while waiting for the client to ask for the next page(or ask to release the resource since it is complete).
>
> Ie. ScrollableResultSet obviously has to be closed when complete and so would this method as well.
>
> Any ideas on how to achieve this as my client is a computer not a webapp with a human clicking next page and we want the resultset paging to be linear as it really hurts our performance.
>
> Thanks,
> Dean
>
> This message and any attachments are intended only for the use of the addressee and
> may contain information that is privileged and confidential. If the reader of the
> message is not the intended recipient or an authorized representative of the
> intended recipient, you are hereby notified that any dissemination of this
> communication is strictly prohibited. If you have received this communication in
> error, please notify us immediately by e-mail and delete the message and any
> attachments from your system.
>
>