You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Naomi Dushay <nd...@stanford.edu> on 2008/11/26 23:43:52 UTC
range queries on string field with millions of values
I have a performance problem and I haven't thought of a clever way
around it.
I work at the Stanford University Libraries. We have a collection of
over 8 million items. Each item has a call number. I have been asked
to provide a way to browse forward and backward from an arbitrary call
number.
I have managed to create a fields that present the call numbers in
appropriate sorts, both forward and reverse. (This is necessary
because raw call numbers don't sort properly: A123 AZ27 B99 B999
BBB111111).
We can ignore the reverse sorted range query problem; it's the same
as the forward sorted range query.
So I use a query like this:
sortCallNum["A123 B34 1970" TO *]&rows=10.
Call numbers are squirrelly, so we can't predict the string that will
appropriately grab at least 10 subsequent documents. They are
certainly not consecutive!
so from
A123 B34 1970
we're unable to predict if any of these will return at least 10 results:
A123 B34 1980 or
A123 B34 V.8 or
A123 B44 or
A123 B67 or
A123 C27 or
A124* or
A22* or
AA* or
You get the idea.
I have not figured out a way to efficiently query for "the next 10
call numbers in sort order". I have also mucked about with the cache
initialization, but that's not working either:
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<!-- populate query result cache for sorted queries -->
<lst>
<str name="q">shelfkey:[0 TO *]</str>
<str name="sort">shelfkey asc</str>
</lst>
</arr>
Can anyone help me with this?
- Naomi
Re: range queries on string field with millions of values
Posted by Glen Newton <gl...@gmail.com>.
Hi Naomi,
Try fixing your data. :-)
No, really:
1 - Sort all of your call numbers using whatever sort makes sense to you.
2 - Assign them - in your sort order - sort keys that are floats, starting:
0.01
0.02
...
1.01
1.02
...
79,999.98
79,999.99
This should approx. cover all of your 8M items.
3 - To guarantee at least 10 items:
sortKey:[234.87 TO 234.96]
4 - OK, as your library is not static, and new items are not added to
the end of the call numbers but interleaved, you need to be able to
deal with this. Here is how.
a - Add the new item to your call numbers.
b - sort by call numbers, as in #1 above
c - Find the sort keys of the two items on either side of your new
item. Let's say these are: 3288.78 and 3288.79. You assign the new
item the sort key one decimal down from the other two: 3288.785, which
places it between the other 2 items, sort-wise.
Of course, as the distribution of additions is not even, so over time
some query ranges will start being much larger than 10. When this
starts to be a problem, re-sort all and re-assign sort keys.
Oh, you will need a mapping from call numbers to the sort keys (and
perhaps vice versa). Just use a Lucene index as a hash lookup.
Is this what you needed?
-glen :-)
2008/11/28 Naomi Dushay <nd...@stanford.edu>:
> The point isn't really how the exact sort works - it's the performance
> issues, coupled with an unpredictable distribution along the entire possible
> sort space.
>
> the sort works
> the range queries work
> the performance sucks
>
> and I haven't thought of a clever work around.
>
> - Naomi
>
> On Nov 27, 2008, at 9:41 AM, Alexander Ramos Jardim wrote:
>
>> I did not even understand what you are considering to be the order on your
>> call numbers.
>>
>> 2008/11/26 Naomi Dushay <nd...@stanford.edu>
>>
>>> I have a performance problem and I haven't thought of a clever way around
>>> it.
>>>
>>> I work at the Stanford University Libraries. We have a collection of
>>> over
>>> 8 million items. Each item has a call number. I have been asked to
>>> provide
>>> a way to browse forward and backward from an arbitrary call number.
>>>
>>> I have managed to create a fields that present the call numbers in
>>> appropriate sorts, both forward and reverse. (This is necessary because
>>> raw
>>> call numbers don't sort properly: A123 AZ27 B99 B999 BBB111111).
>>>
>>> We can ignore the reverse sorted range query problem; it's the same as
>>> the
>>> forward sorted range query.
>>>
>>> So I use a query like this:
>>>
>>> sortCallNum["A123 B34 1970" TO *]&rows=10.
>>>
>>>
>>> Call numbers are squirrelly, so we can't predict the string that will
>>> appropriately grab at least 10 subsequent documents. They are certainly
>>> not
>>> consecutive!
>>>
>>> so from
>>> A123 B34 1970
>>>
>>> we're unable to predict if any of these will return at least 10 results:
>>>
>>> A123 B34 1980 or
>>> A123 B34 V.8 or
>>> A123 B44 or
>>> A123 B67 or
>>> A123 C27 or
>>> A124* or
>>> A22* or
>>> AA* or
>>>
>>> You get the idea.
>>>
>>> I have not figured out a way to efficiently query for "the next 10 call
>>> numbers in sort order". I have also mucked about with the cache
>>> initialization, but that's not working either:
>>>
>>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>>> <arr name="queries">
>>> <!-- populate query result cache for sorted queries -->
>>> <lst>
>>> <str name="q">shelfkey:[0 TO *]</str>
>>> <str name="sort">shelfkey asc</str>
>>> </lst>
>>> </arr>
>>>
>>> Can anyone help me with this?
>>>
>>> - Naomi
>>>
>>>
>>
>>
>> --
>> Alexander Ramos Jardim
>
> Naomi Dushay
> ndushay@stanford.edu
>
>
>
>
--
-
Re: range queries on string field with millions of values
Posted by Naomi Dushay <nd...@stanford.edu>.
The point isn't really how the exact sort works - it's the performance
issues, coupled with an unpredictable distribution along the entire
possible sort space.
the sort works
the range queries work
the performance sucks
and I haven't thought of a clever work around.
- Naomi
On Nov 27, 2008, at 9:41 AM, Alexander Ramos Jardim wrote:
> I did not even understand what you are considering to be the order
> on your
> call numbers.
>
> 2008/11/26 Naomi Dushay <nd...@stanford.edu>
>
>> I have a performance problem and I haven't thought of a clever way
>> around
>> it.
>>
>> I work at the Stanford University Libraries. We have a collection
>> of over
>> 8 million items. Each item has a call number. I have been asked
>> to provide
>> a way to browse forward and backward from an arbitrary call number.
>>
>> I have managed to create a fields that present the call numbers in
>> appropriate sorts, both forward and reverse. (This is necessary
>> because raw
>> call numbers don't sort properly: A123 AZ27 B99 B999 BBB111111).
>>
>> We can ignore the reverse sorted range query problem; it's the
>> same as the
>> forward sorted range query.
>>
>> So I use a query like this:
>>
>> sortCallNum["A123 B34 1970" TO *]&rows=10.
>>
>>
>> Call numbers are squirrelly, so we can't predict the string that will
>> appropriately grab at least 10 subsequent documents. They are
>> certainly not
>> consecutive!
>>
>> so from
>> A123 B34 1970
>>
>> we're unable to predict if any of these will return at least 10
>> results:
>>
>> A123 B34 1980 or
>> A123 B34 V.8 or
>> A123 B44 or
>> A123 B67 or
>> A123 C27 or
>> A124* or
>> A22* or
>> AA* or
>>
>> You get the idea.
>>
>> I have not figured out a way to efficiently query for "the next 10
>> call
>> numbers in sort order". I have also mucked about with the cache
>> initialization, but that's not working either:
>>
>> <listener event="firstSearcher" class="solr.QuerySenderListener">
>> <arr name="queries">
>> <!-- populate query result cache for sorted queries -->
>> <lst>
>> <str name="q">shelfkey:[0 TO *]</str>
>> <str name="sort">shelfkey asc</str>
>> </lst>
>> </arr>
>>
>> Can anyone help me with this?
>>
>> - Naomi
>>
>>
>
>
> --
> Alexander Ramos Jardim
Naomi Dushay
ndushay@stanford.edu
Re: range queries on string field with millions of values
Posted by Alexander Ramos Jardim <al...@gmail.com>.
I did not even understand what you are considering to be the order on your
call numbers.
2008/11/26 Naomi Dushay <nd...@stanford.edu>
> I have a performance problem and I haven't thought of a clever way around
> it.
>
> I work at the Stanford University Libraries. We have a collection of over
> 8 million items. Each item has a call number. I have been asked to provide
> a way to browse forward and backward from an arbitrary call number.
>
> I have managed to create a fields that present the call numbers in
> appropriate sorts, both forward and reverse. (This is necessary because raw
> call numbers don't sort properly: A123 AZ27 B99 B999 BBB111111).
>
> We can ignore the reverse sorted range query problem; it's the same as the
> forward sorted range query.
>
> So I use a query like this:
>
> sortCallNum["A123 B34 1970" TO *]&rows=10.
>
>
> Call numbers are squirrelly, so we can't predict the string that will
> appropriately grab at least 10 subsequent documents. They are certainly not
> consecutive!
>
> so from
> A123 B34 1970
>
> we're unable to predict if any of these will return at least 10 results:
>
> A123 B34 1980 or
> A123 B34 V.8 or
> A123 B44 or
> A123 B67 or
> A123 C27 or
> A124* or
> A22* or
> AA* or
>
> You get the idea.
>
> I have not figured out a way to efficiently query for "the next 10 call
> numbers in sort order". I have also mucked about with the cache
> initialization, but that's not working either:
>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <!-- populate query result cache for sorted queries -->
> <lst>
> <str name="q">shelfkey:[0 TO *]</str>
> <str name="sort">shelfkey asc</str>
> </lst>
> </arr>
>
> Can anyone help me with this?
>
> - Naomi
>
>
--
Alexander Ramos Jardim
Re: range queries on string field with millions of values
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Nov 26, 2008 at 5:43 PM, Naomi Dushay <nd...@stanford.edu> wrote:
> sortCallNum["A123 B34 1970" TO *]&rows=10.
If you really just want to get call numbers X through X+10, then you
are in luck:
https://issues.apache.org/jira/browse/SOLR-877
http://wiki.apache.org/solr/TermsComponent
But looking at what Hoss wrote you might actually want the first 10
results sorted by field myField, that meet the restriction that
callNumer>X (a filter). If so, this is a long standing problem that
isn't solved yet (slowness of range queries with many terms), but we
do know methods of fixing it, most involving indexing prefixes.
-Yonik
>
> Call numbers are squirrelly, so we can't predict the string that will
> appropriately grab at least 10 subsequent documents. They are certainly not
> consecutive!
>
> so from
> A123 B34 1970
>
> we're unable to predict if any of these will return at least 10 results:
>
> A123 B34 1980 or
> A123 B34 V.8 or
> A123 B44 or
> A123 B67 or
> A123 C27 or
> A124* or
> A22* or
> AA* or
>
> You get the idea.
>
> I have not figured out a way to efficiently query for "the next 10 call
> numbers in sort order". I have also mucked about with the cache
> initialization, but that's not working either:
>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <!-- populate query result cache for sorted queries -->
> <lst>
> <str name="q">shelfkey:[0 TO *]</str>
> <str name="sort">shelfkey asc</str>
> </lst>
> </arr>
>
> Can anyone help me with this?
>
> - Naomi
>
>
Re: range queries on string field with millions of values
Posted by Yonik Seeley <yo...@apache.org>.
On Sun, Nov 30, 2008 at 2:04 AM, Naomi Dushay <nd...@stanford.edu> wrote:
> The terms component approach, if i understand it correctly, will be
> problematic. I need to present not only the next X call numbers in
> sequence, but other fields in those documents (e.g. title, author).
You can still use the method Hoss suggested of doing 2 requests to
satisfy this type of search:
>> But as Yonik said: the new TermsComponent may actually be a better option
>> for you -- doing two requests for every page (the first to get the N Terms
>> in your id field starting with your input, the second to do an query for
>> docs matching any of those N ids) might actually be faster even though
>> there won't likely even be any cache hits.
So TermsComponent gets the next 10 IDs, then you do a standard query
with those 10 IDs.
-Yonik
> assume the Terms Component approach will only give me the next X call number
> values, not the documents.
>
> It sounds like Glen Newton's suggestion of mapping the call numbers to a
> float number is the most likely solution.
>
> I know it sounds ridiculous to do all this for a "call number browse" but
> our faculty have explicitly asked for this. For humanities scholars
> especially, they know the call numbers that are of interest to them, and
> they browse the stacks that way (ML 1500s are opera, V35 is verdi ...).
> They are using the research methods that have been successful for their
> entire careers. Plus, library materials are going to off site, high density
> storage, so the only way for them to to browse all materials, regardless of
> location, via call number is online. I doubt they'll find this feature as
> useful as they expect, but it behooves us to give the users what they ask
> for.
>
> So yeah, our user needs are perhaps a little outside of your expectations.
> :-)
>
> - Naomi
>
>
> On Nov 29, 2008, at 2:58 PM, Chris Hostetter wrote:
>
>>
>> : The results are correct. But the response time sucks.
>> :
>> : Reading the docs about caches, I thought I could populate the query
>> result
>> : cache with an autowarming query and the response time would be okay.
>> But that
>> : hasn't worked. (See excerpts from my solrConfig file below.)
>> :
>> : A repeated query is very fast, implying caching happens for a particular
>> : starting point ("42" above).
>> :
>> : Is there a way to populate the cache with the ENTIRE sorted list of
>> values for
>> : the field, so any arbitrary starting point will get results from the
>> cache,
>> : rather than grabbing all results from (x) to the end, then sorting all
>> these
>> : results, then returning the first 10?
>>
>> there's two "caches" that come into play for something like this...
>>
>> the first cache is a low level Lucene cache called the "FieldCache" that
>> is completley hidden from you (and for the most part: from Solr).
>> anytime you sort on a field, it get's built, and reuse for all sorts on
>> that field. My originl concern was that it wasn't getting warmed on
>> "newSearcher" (because you have to be explicit about that.
>>
>> the second cache is the queryResultsCache which caches a "window" of an
>> ordered list of documents based on a query, and a sort. you can see this
>> cache in your Solr stats, and yes: these two requests results in different
>> cache keys for the queryResultsCache...
>>
>> q=yourField:[42+TO+*]&sort=yourField+asc&rows=10
>> q=yourField:[52+TO+*]&sort=yourField+asc&rows=10
>>
>> ...BUT! ... the two queries below will result in the same cache key, and
>> the second will be a cache hit, provided a sufficient value for
>> the "queryResultWindowSize" ...
>>
>> q=yourField:[42+TO+*]&sort=yourField+asc&rows=10
>> q=yourField:[42+TO+*]&sort=yourField+asc&rows=10&start=10
>>
>> so perhaps the key to your problem is to just make sure that once the user
>> gives you an id to start with, you "scroll" by increasing the start param
>> (not altering the id) ... the first query might be "slow" but every query
>> after that should be a cache hit (depending on your page size, and how far
>> you expect people to scroll, you should consider increasing
>> queryResultWindowSize)
>>
>> But as Yonik said: the new TermsComponent may actually be a better option
>> for you -- doing two requests for every page (the first to get the N Terms
>> in your id field starting with your input, the second to do an query for
>> docs matching any of those N ids) might actually be faster even though
>> there won't likely even be any cache hits.
>>
>>
>> My opinion: Your use case sounds like a waste of effort. I can't imagine
>> anyone using a library catalog system ever wanting to lookup a callnumber,
>> and then scroll through all posisble books with similar call numbers -- it
>> seems much more likely that i'd want to look at other books with similar
>> authors, or keywords, or tags ... all things that are actaully *easier* to
>> do with Solr. (but then again: i don't work in a library. i trust that
>> you know something i don't about what your users want.)
>>
>>
>> -Hoss
>>
>
> Naomi Dushay
> ndushay@stanford.edu
>
>
>
>
Re: range queries on string field with millions of values
Posted by Naomi Dushay <nd...@stanford.edu>.
Hi Hoss,
Thanks for this.
The terms component approach, if i understand it correctly, will be
problematic. I need to present not only the next X call numbers in
sequence, but other fields in those documents (e.g. title, author). I
assume the Terms Component approach will only give me the next X call
number values, not the documents.
It sounds like Glen Newton's suggestion of mapping the call numbers to
a float number is the most likely solution.
I know it sounds ridiculous to do all this for a "call number browse"
but our faculty have explicitly asked for this. For humanities
scholars especially, they know the call numbers that are of interest
to them, and they browse the stacks that way (ML 1500s are opera, V35
is verdi ...). They are using the research methods that have been
successful for their entire careers. Plus, library materials are
going to off site, high density storage, so the only way for them to
to browse all materials, regardless of location, via call number is
online. I doubt they'll find this feature as useful as they expect,
but it behooves us to give the users what they ask for.
So yeah, our user needs are perhaps a little outside of your
expectations. :-)
- Naomi
On Nov 29, 2008, at 2:58 PM, Chris Hostetter wrote:
>
> : The results are correct. But the response time sucks.
> :
> : Reading the docs about caches, I thought I could populate the
> query result
> : cache with an autowarming query and the response time would be
> okay. But that
> : hasn't worked. (See excerpts from my solrConfig file below.)
> :
> : A repeated query is very fast, implying caching happens for a
> particular
> : starting point ("42" above).
> :
> : Is there a way to populate the cache with the ENTIRE sorted list
> of values for
> : the field, so any arbitrary starting point will get results from
> the cache,
> : rather than grabbing all results from (x) to the end, then sorting
> all these
> : results, then returning the first 10?
>
> there's two "caches" that come into play for something like this...
>
> the first cache is a low level Lucene cache called the "FieldCache"
> that
> is completley hidden from you (and for the most part: from Solr).
> anytime you sort on a field, it get's built, and reuse for all sorts
> on
> that field. My originl concern was that it wasn't getting warmed on
> "newSearcher" (because you have to be explicit about that.
>
> the second cache is the queryResultsCache which caches a "window" of
> an
> ordered list of documents based on a query, and a sort. you can see
> this
> cache in your Solr stats, and yes: these two requests results in
> different
> cache keys for the queryResultsCache...
>
> q=yourField:[42+TO+*]&sort=yourField+asc&rows=10
> q=yourField:[52+TO+*]&sort=yourField+asc&rows=10
>
> ...BUT! ... the two queries below will result in the same cache key,
> and
> the second will be a cache hit, provided a sufficient value for
> the "queryResultWindowSize" ...
>
> q=yourField:[42+TO+*]&sort=yourField+asc&rows=10
> q=yourField:[42+TO+*]&sort=yourField+asc&rows=10&start=10
>
> so perhaps the key to your problem is to just make sure that once
> the user
> gives you an id to start with, you "scroll" by increasing the start
> param
> (not altering the id) ... the first query might be "slow" but every
> query
> after that should be a cache hit (depending on your page size, and
> how far
> you expect people to scroll, you should consider increasing
> queryResultWindowSize)
>
> But as Yonik said: the new TermsComponent may actually be a better
> option
> for you -- doing two requests for every page (the first to get the N
> Terms
> in your id field starting with your input, the second to do an query
> for
> docs matching any of those N ids) might actually be faster even though
> there won't likely even be any cache hits.
>
>
> My opinion: Your use case sounds like a waste of effort. I can't
> imagine
> anyone using a library catalog system ever wanting to lookup a
> callnumber,
> and then scroll through all posisble books with similar call numbers
> -- it
> seems much more likely that i'd want to look at other books with
> similar
> authors, or keywords, or tags ... all things that are actaully
> *easier* to
> do with Solr. (but then again: i don't work in a library. i trust
> that
> you know something i don't about what your users want.)
>
>
> -Hoss
>
Naomi Dushay
ndushay@stanford.edu
Re: range queries on string field with millions of values
Posted by Chris Hostetter <ho...@fucit.org>.
: The results are correct. But the response time sucks.
:
: Reading the docs about caches, I thought I could populate the query result
: cache with an autowarming query and the response time would be okay. But that
: hasn't worked. (See excerpts from my solrConfig file below.)
:
: A repeated query is very fast, implying caching happens for a particular
: starting point ("42" above).
:
: Is there a way to populate the cache with the ENTIRE sorted list of values for
: the field, so any arbitrary starting point will get results from the cache,
: rather than grabbing all results from (x) to the end, then sorting all these
: results, then returning the first 10?
there's two "caches" that come into play for something like this...
the first cache is a low level Lucene cache called the "FieldCache" that
is completley hidden from you (and for the most part: from Solr).
anytime you sort on a field, it get's built, and reuse for all sorts on
that field. My originl concern was that it wasn't getting warmed on
"newSearcher" (because you have to be explicit about that.
the second cache is the queryResultsCache which caches a "window" of an
ordered list of documents based on a query, and a sort. you can see this
cache in your Solr stats, and yes: these two requests results in different
cache keys for the queryResultsCache...
q=yourField:[42+TO+*]&sort=yourField+asc&rows=10
q=yourField:[52+TO+*]&sort=yourField+asc&rows=10
...BUT! ... the two queries below will result in the same cache key, and
the second will be a cache hit, provided a sufficient value for
the "queryResultWindowSize" ...
q=yourField:[42+TO+*]&sort=yourField+asc&rows=10
q=yourField:[42+TO+*]&sort=yourField+asc&rows=10&start=10
so perhaps the key to your problem is to just make sure that once the user
gives you an id to start with, you "scroll" by increasing the start param
(not altering the id) ... the first query might be "slow" but every query
after that should be a cache hit (depending on your page size, and how far
you expect people to scroll, you should consider increasing
queryResultWindowSize)
But as Yonik said: the new TermsComponent may actually be a better option
for you -- doing two requests for every page (the first to get the N Terms
in your id field starting with your input, the second to do an query for
docs matching any of those N ids) might actually be faster even though
there won't likely even be any cache hits.
My opinion: Your use case sounds like a waste of effort. I can't imagine
anyone using a library catalog system ever wanting to lookup a callnumber,
and then scroll through all posisble books with similar call numbers -- it
seems much more likely that i'd want to look at other books with similar
authors, or keywords, or tags ... all things that are actaully *easier* to
do with Solr. (but then again: i don't work in a library. i trust that
you know something i don't about what your users want.)
-Hoss
Re: range queries on string field with millions of values
Posted by Walter Underwood <wu...@netflix.com>.
Do you want to page through all items or through the result of a
query (like all hits for "civil war" in call number order).
If you want the former, then a text search engine is really
the wrong tool. This problem only requires indexed sequential
file formats (like B-trees), something that worked quite well
30 or 40 years ago, even before relational databases were invented.
Text search engines, like Lucene/Solr, have sorting and traversal
as a secondary feature. Their primary feature is relevance ranking.
With only 8M items, I'd be inclined to put them in a big array
sorted by call number, and use binary search. Sounds dumb, but
it is really fast. The entries would be a simple pair, call
number and key.
wunder
On 11/28/08 4:41 PM, "Naomi Dushay" <nd...@stanford.edu> wrote:
> Gosh, I'm sorry to be so unclear. Hmm. Trying to clarify below:
>
> On Nov 28, 2008, at 3:52 PM, Chris Hostetter wrote:
>
>> Having read through this thread, i'm not sure i understand what
>> exactly
>> the problem is. my naive understanding is...
>>
>> 1) you want to sort by a field
>> 2) you want to be able to "paginate" through all docs in order of this
>> field.
>> 3) you want to be able to start your pagination at any arbitrary
>> value for
>> this field.
>>
>> so (assuming the field is a simple number for now) you could us
>> something
>> like
>>
>> q=yourField:[42 TO *&sort=yourField+asc&rows=10&start-0
>>
>> where "42" is the arbitrary ID someone wants to start at.
>>
>
> perfect. This is the query I'm using.
>
> The results are correct. But the response time sucks.
>
> Reading the docs about caches, I thought I could populate the query
> result cache with an autowarming query and the response time would be
> okay. But that hasn't worked. (See excerpts from my solrConfig file
> below.)
>
> A repeated query is very fast, implying caching happens for a
> particular starting point ("42" above).
>
> Is there a way to populate the cache with the ENTIRE sorted list of
> values for the field, so any arbitrary starting point will get results
> from the cache, rather than grabbing all results from (x) to the end,
> then sorting all these results, then returning the first 10?
>
>
>> This sentence below seems to imply that you have a solution which
>> produces
>> correct results, but doesn't produce results quickly...
>
> right.
>
>> : I have a performance problem and I haven't thought of a clever way
>> around it.
>>
>> ...however this lines seems to suggest that you're having trouble
>> getting at least 10 results from any query (?)
>>
>> : Call numbers are squirrelly, so we can't predict the string that
>> will
>> : appropriately grab at least 10 subsequent documents. They are
>> certainly not
>> : consecutive!
>> :
>> : so from
>> : A123 B34 1970
>> :
>> : we're unable to predict if any of these will return at least 10
>> results:
>
> I was trying to express that I couldn't do this:
>
> myfield:[X TO Y]
>
> because I can't algorithmically compute Y.
>
> Glen Newton suggested a work around, whereby I represent my
> squirrelly, but sortable, field values as floating point numbers, and
> then I can compute Y.
>
>> ...but i'm not sure what exactly that means. for any given field,
>> there
>> is always going to be some values X such that myField:[X TO *] won't
>> return at least 10 docs ... the are the last values in the index in
>> order
>> -- surely it's okay for your app to have an "end" state when you run
>> out
>> of data? :)
>
> yes. Understood. This is not an issue.
>
>> Oh, and BTW...
>>
>> : numbers in sort order". I have also mucked about with the cache
>> : initialization, but that's not working either:
>> :
>> : <listener event="firstSearcher"
>> class="solr.QuerySenderListener">
>>
>> ...make sure you also do a newSearcher listener that does the same
>> thing,
>> otherwise your FieldCache (used for sorting) may not be warmed when
>> commits happen)
>
> Yup yup yup.
>
> from solrconfig:
>
> <filterCache
> class="solr.LRUCache"
> size="20000000"
> initialSize="10000000"
> autowarmCount="500000"/>
>
> <queryResultCache
> class="solr.LRUCache"
> size="10000000"
> initialSize="5000000"
> autowarmCount="5000000"/>
>
>
> <listener event="newSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <!-- populate query result cache for sorted queries -->
> <lst>
> <str name="q">shelfkey:[0 TO *]</str>
> <str name="sort">shelfkey asc</str>
> </lst>
> </arr>
> </listener>
>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <!-- populate query result cache for sorted queries -->
> <lst>
> <str name="q">shelfkey:[0 TO *]</str>
> <str name="sort">shelfkey asc</str>
> </lst>
>
>
Re: range queries on string field with millions of values
Posted by Naomi Dushay <nd...@stanford.edu>.
Gosh, I'm sorry to be so unclear. Hmm. Trying to clarify below:
On Nov 28, 2008, at 3:52 PM, Chris Hostetter wrote:
> Having read through this thread, i'm not sure i understand what
> exactly
> the problem is. my naive understanding is...
>
> 1) you want to sort by a field
> 2) you want to be able to "paginate" through all docs in order of this
> field.
> 3) you want to be able to start your pagination at any arbitrary
> value for
> this field.
>
> so (assuming the field is a simple number for now) you could us
> something
> like
>
> q=yourField:[42 TO *&sort=yourField+asc&rows=10&start-0
>
> where "42" is the arbitrary ID someone wants to start at.
>
perfect. This is the query I'm using.
The results are correct. But the response time sucks.
Reading the docs about caches, I thought I could populate the query
result cache with an autowarming query and the response time would be
okay. But that hasn't worked. (See excerpts from my solrConfig file
below.)
A repeated query is very fast, implying caching happens for a
particular starting point ("42" above).
Is there a way to populate the cache with the ENTIRE sorted list of
values for the field, so any arbitrary starting point will get results
from the cache, rather than grabbing all results from (x) to the end,
then sorting all these results, then returning the first 10?
> This sentence below seems to imply that you have a solution which
> produces
> correct results, but doesn't produce results quickly...
right.
> : I have a performance problem and I haven't thought of a clever way
> around it.
>
> ...however this lines seems to suggest that you're having trouble
> getting at least 10 results from any query (?)
>
> : Call numbers are squirrelly, so we can't predict the string that
> will
> : appropriately grab at least 10 subsequent documents. They are
> certainly not
> : consecutive!
> :
> : so from
> : A123 B34 1970
> :
> : we're unable to predict if any of these will return at least 10
> results:
I was trying to express that I couldn't do this:
myfield:[X TO Y]
because I can't algorithmically compute Y.
Glen Newton suggested a work around, whereby I represent my
squirrelly, but sortable, field values as floating point numbers, and
then I can compute Y.
> ...but i'm not sure what exactly that means. for any given field,
> there
> is always going to be some values X such that myField:[X TO *] won't
> return at least 10 docs ... the are the last values in the index in
> order
> -- surely it's okay for your app to have an "end" state when you run
> out
> of data? :)
yes. Understood. This is not an issue.
> Oh, and BTW...
>
> : numbers in sort order". I have also mucked about with the cache
> : initialization, but that's not working either:
> :
> : <listener event="firstSearcher"
> class="solr.QuerySenderListener">
>
> ...make sure you also do a newSearcher listener that does the same
> thing,
> otherwise your FieldCache (used for sorting) may not be warmed when
> commits happen)
Yup yup yup.
from solrconfig:
<filterCache
class="solr.LRUCache"
size="20000000"
initialSize="10000000"
autowarmCount="500000"/>
<queryResultCache
class="solr.LRUCache"
size="10000000"
initialSize="5000000"
autowarmCount="5000000"/>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<!-- populate query result cache for sorted queries -->
<lst>
<str name="q">shelfkey:[0 TO *]</str>
<str name="sort">shelfkey asc</str>
</lst>
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<!-- populate query result cache for sorted queries -->
<lst>
<str name="q">shelfkey:[0 TO *]</str>
<str name="sort">shelfkey asc</str>
</lst>
Re: range queries on string field with millions of values
Posted by Chris Hostetter <ho...@fucit.org>.
Having read through this thread, i'm not sure i understand what exactly
the problem is. my naive understanding is...
1) you want to sort by a field
2) you want to be able to "paginate" through all docs in order of this
field.
3) you want to be able to start your pagination at any arbitrary value for
this field.
so (assuming the field is a simple number for now) you could us something
like
q=yourField:[42 TO *&sort=yourField+asc&rows=10&start-0
where "42" is the arbitrary ID someone wants to start at.
This sentence below seems to imply that you have a solution which produces
correct results, but doesn't produce results quickly...
: I have a performance problem and I haven't thought of a clever way around it.
...however this lines seems to suggest that you're having trouble
getting at least 10 results from any query (?)
: Call numbers are squirrelly, so we can't predict the string that will
: appropriately grab at least 10 subsequent documents. They are certainly not
: consecutive!
:
: so from
: A123 B34 1970
:
: we're unable to predict if any of these will return at least 10 results:
...but i'm not sure what exactly that means. for any given field, there
is always going to be some values X such that myField:[X TO *] won't
return at least 10 docs ... the are the last values in the index in order
-- surely it's okay for your app to have an "end" state when you run out
of data? :)
But like i said, i'm not relaly clear on whether i'm clear on what your
problem is.
Oh, and BTW...
: numbers in sort order". I have also mucked about with the cache
: initialization, but that's not working either:
:
: <listener event="firstSearcher" class="solr.QuerySenderListener">
...make sure you also do a newSearcher listener that does the same thing,
otherwise your FieldCache (used for sorting) may not be warmed when
commits happen)
-Hoss