Date range performance

[Jonathan Ariel <io...@gmail.com>]

Hi,
I'm experiencing a really poor performance when using date ranges in solr
query. Is it a know issue? is there any special consideration when using
date ranges? It seems weird because I always thought date dates are
translated to strings, so internally lucene resolves everything the same
way. So maybe the problem is with parsing the dates and traslating it to the
internal value?
Any suggestion?

Thanks!

Re: Date range performance

[Norberto Meijome <fr...@meijome.net>]

On Thu, 3 Apr 2008 18:14:56 -0300
"Jonathan Ariel" <io...@gmail.com> wrote:

> I'm experiencing a really poor performance when using date ranges in solr
> query. Is it a know issue? is there any special consideration when using
> date ranges? It seems weird because I always thought date dates are
> translated to strings, so internally lucene resolves everything the same
> way. So maybe the problem is with parsing the dates and traslating it to the
> internal value?
> Any suggestion?

Hi Jonathan,
I've seen this mentioned somewhere, but I cannot find the reference to it now. from what I recall, it goes down to the DateField type having too much precision (down to milliseconds, from memory). When you perform a range search, it would check for each of those possibilities making it a huge and slow task. 
The advice in that post / page was to use a string field which encoded the date as YYYYMMDD...to as much precision as you want, and then handling the date maths yourself on the app.

It may not work for you, and I could be completely wrong :)

b
_________________________
{Beto|Norberto|Numard} Meijome

"A Man that is good at excuses is usually good at nothing else"
  Benjamin Franklin

I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.

Re: Date range performance

[Jonathan Ariel <io...@gmail.com>]

Ok. Just to give some feedback.
I reindexed with less precision as you told me and it's working really fast.
Thanks for your help!

Jonathan

On Fri, Apr 4, 2008 at 6:02 PM, Chris Hostetter <ho...@fucit.org>
wrote:

>
> : Looking into the code it seems like a Lucene problem, more than Solr. It
> is
> : in the RangeQuery and RangeFilter classes. The problem with changing
> this to
> : have a sorted index and than binary search is that you have to sort it,
> : which is slow. Unless we can store the ordered index somewhere and reuse
> it,
> : it will be even slower than now. And if we store it, we will have to
> face
> : the problem with updating ordered index with new terms.
>
> FWIW: Lucene Term enumeration is already indexed, it's just not a binary
> search tree (the details escape me at the moment, but there there is an
> interval value of N somewhere in the code, and every Nth Term is loaded
> into memory so a TermEnum.seek can skip ahead N terms at a time).
>
> But the number of unique terms can be a bottle neck ... rounding to the
> level of precision you absolutely need can save you in these cases by
> reducing the number of unique terms.
>
>
>
>
> -Hoss
>
>

Re: Date range performance

[Chris Hostetter <ho...@fucit.org>]

: Looking into the code it seems like a Lucene problem, more than Solr. It is
: in the RangeQuery and RangeFilter classes. The problem with changing this to
: have a sorted index and than binary search is that you have to sort it,
: which is slow. Unless we can store the ordered index somewhere and reuse it,
: it will be even slower than now. And if we store it, we will have to face
: the problem with updating ordered index with new terms.

FWIW: Lucene Term enumeration is already indexed, it's just not a binary 
search tree (the details escape me at the moment, but there there is an 
interval value of N somewhere in the code, and every Nth Term is loaded 
into memory so a TermEnum.seek can skip ahead N terms at a time).

But the number of unique terms can be a bottle neck ... rounding to the 
level of precision you absolutely need can save you in these cases by 
reducing the number of unique terms.




-Hoss

Re: Date range performance

[Jonathan Ariel <io...@gmail.com>]

Thanks! I'll try taking some precision and let you know about the result.

Looking into the code it seems like a Lucene problem, more than Solr. It is
in the RangeQuery and RangeFilter classes. The problem with changing this to
have a sorted index and than binary search is that you have to sort it,
which is slow. Unless we can store the ordered index somewhere and reuse it,
it will be even slower than now. And if we store it, we will have to face
the problem with updating ordered index with new terms.


On Fri, Apr 4, 2008 at 3:30 PM, Mike Klaas <mi...@gmail.com> wrote:

> On 3-Apr-08, at 4:24 PM, Jonathan Ariel wrote:
>
> > Is this depends on the number of documents that matches the query or the
> > number of documents in the index?
> >
>
> This aspect is more depedent on the number of terms that the date query
> translates into.
>
>  If in a 3 million documents index my query matches 4, having date with a
> > precision of seconds could slow down the query?
> >
>
> Yes.  Solr does range queries by taking the disjunction of a bunch of term
> queries, so it is the total number of terms checked that is the limiting
> factor.
>
> It would be better to implement this using an ordered index that could be
> binary-searched, but Solr isn't currently designed for that (though I think
> range optimization algorithms would be a cool addition).
>
> -Mike
>
>
>
> > On Thu, Apr 3, 2008 at 7:45 PM, Mike Klaas <mi...@gmail.com> wrote:
> >
> >
> > > On 3-Apr-08, at 2:14 PM, Jonathan Ariel wrote:
> > >
> > >  Hi,
> > > > I'm experiencing a really poor performance when using date ranges in
> > > > solr
> > > > query. Is it a know issue? is there any special consideration when
> > > > using
> > > > date ranges? It seems weird because I always thought date dates are
> > > > translated to strings, so internally lucene resolves everything the
> > > > same
> > > > way. So maybe the problem is with parsing the dates and traslating
> > > > it to
> > > > the
> > > > internal value?
> > > > Any suggestion?
> > > >
> > > >
> > > Range query is highly dependent on the total number of unique terms
> > > covered by the range.  If you are indexing dates with very high
> > > precision
> > > (e.g., milliseconds), this can consist of ridiculous numbers of terms.
> > >
> > > Try rounding the dates to something more granular when indexing.
> > >
> > > -Mike
> > >
> > >
>

Re: Date range performance

[Mike Klaas <mi...@gmail.com>]

On 3-Apr-08, at 4:24 PM, Jonathan Ariel wrote:
> Is this depends on the number of documents that matches the query or  
> the
> number of documents in the index?

This aspect is more depedent on the number of terms that the date  
query translates into.

> If in a 3 million documents index my query matches 4, having date  
> with a
> precision of seconds could slow down the query?

Yes.  Solr does range queries by taking the disjunction of a bunch of  
term queries, so it is the total number of terms checked that is the  
limiting factor.

It would be better to implement this using an ordered index that could  
be binary-searched, but Solr isn't currently designed for that (though  
I think range optimization algorithms would be a cool addition).

-Mike

>
> On Thu, Apr 3, 2008 at 7:45 PM, Mike Klaas <mi...@gmail.com>  
> wrote:
>
>>
>> On 3-Apr-08, at 2:14 PM, Jonathan Ariel wrote:
>>
>>> Hi,
>>> I'm experiencing a really poor performance when using date ranges in
>>> solr
>>> query. Is it a know issue? is there any special consideration when  
>>> using
>>> date ranges? It seems weird because I always thought date dates are
>>> translated to strings, so internally lucene resolves everything  
>>> the same
>>> way. So maybe the problem is with parsing the dates and traslating  
>>> it to
>>> the
>>> internal value?
>>> Any suggestion?
>>>
>>
>> Range query is highly dependent on the total number of unique terms
>> covered by the range.  If you are indexing dates with very high  
>> precision
>> (e.g., milliseconds), this can consist of ridiculous numbers of  
>> terms.
>>
>> Try rounding the dates to something more granular when indexing.
>>
>> -Mike
>>

Re: Date range performance

[Jonathan Ariel <io...@gmail.com>]

Is this depends on the number of documents that matches the query or the
number of documents in the index?
If in a 3 million documents index my query matches 4, having date with a
precision of seconds could slow down the query?

On Thu, Apr 3, 2008 at 7:45 PM, Mike Klaas <mi...@gmail.com> wrote:

>
> On 3-Apr-08, at 2:14 PM, Jonathan Ariel wrote:
>
> > Hi,
> > I'm experiencing a really poor performance when using date ranges in
> > solr
> > query. Is it a know issue? is there any special consideration when using
> > date ranges? It seems weird because I always thought date dates are
> > translated to strings, so internally lucene resolves everything the same
> > way. So maybe the problem is with parsing the dates and traslating it to
> > the
> > internal value?
> > Any suggestion?
> >
>
> Range query is highly dependent on the total number of unique terms
> covered by the range.  If you are indexing dates with very high precision
> (e.g., milliseconds), this can consist of ridiculous numbers of terms.
>
> Try rounding the dates to something more granular when indexing.
>
> -Mike
>

Re: Date range performance

[Mike Klaas <mi...@gmail.com>]

On 3-Apr-08, at 2:14 PM, Jonathan Ariel wrote:
> Hi,
> I'm experiencing a really poor performance when using date ranges in  
> solr
> query. Is it a know issue? is there any special consideration when  
> using
> date ranges? It seems weird because I always thought date dates are
> translated to strings, so internally lucene resolves everything the  
> same
> way. So maybe the problem is with parsing the dates and traslating  
> it to the
> internal value?
> Any suggestion?

Range query is highly dependent on the total number of unique terms  
covered by the range.  If you are indexing dates with very high  
precision (e.g., milliseconds), this can consist of ridiculous numbers  
of terms.

Try rounding the dates to something more granular when indexing.

-Mike