You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jan Simon Winkelmann <wi...@newsfactory.de> on 2010/04/29 10:35:57 UTC
Slow Date-Range Queries
Hi,
I am currently having serious performance problems with date range queries. What I am doing, is validating a datasets published status by a valid_from and a valid_till date field.
I did get a performance boost of ~ 100% by switching from a normal solr.DateField to a solr.TrieDateField with precisionStep="8", however my query still takes about 1,3 seconds.
My field defintion looks like this:
<fieldType name="date" class="solr.TrieDateField" precisionStep="8" sortMissingLast="true" omitNorms="true"/>
<field name="valid_from" type="date" indexed="true" stored="false" required="false" />
<field name="valid_till" type="date" indexed="true" stored="false" required="false" />
And the query looks like this:
((valid_from:[* TO 2010-04-29T10:34:12Z]) AND (valid_till:[2010-04-29T10:34:12Z TO *])) OR ((*:* -valid_from:[* TO *]) AND (*:* -valid_till:[* TO *])))
I use the empty checks for datasets which do not have a valid from/till range.
Is there any way to get this any faster? Would it be faster using unix-timestamps with int fields?
I would appreciate any insight and help on this.
regards,
Jan-Simon
RE: Slow Date-Range Queries
Posted by "Nagelberg, Kallin" <KN...@globeandmail.com>.
You might want to look at DateMath, http://lucene.apache.org/solr/api/org/apache/solr/util/DateMathParser.html. I believe the default precision is to the millisecond, so if you afford to round to the nearest second or even minute you might see some performance gains.
-Kallin Nagelberg
-----Original Message-----
From: Jan Simon Winkelmann [mailto:winkelmann@newsfactory.de]
Sent: Thursday, April 29, 2010 4:36 AM
To: solr-user@lucene.apache.org
Subject: Slow Date-Range Queries
Hi,
I am currently having serious performance problems with date range queries. What I am doing, is validating a datasets published status by a valid_from and a valid_till date field.
I did get a performance boost of ~ 100% by switching from a normal solr.DateField to a solr.TrieDateField with precisionStep="8", however my query still takes about 1,3 seconds.
My field defintion looks like this:
<fieldType name="date" class="solr.TrieDateField" precisionStep="8" sortMissingLast="true" omitNorms="true"/>
<field name="valid_from" type="date" indexed="true" stored="false" required="false" />
<field name="valid_till" type="date" indexed="true" stored="false" required="false" />
And the query looks like this:
((valid_from:[* TO 2010-04-29T10:34:12Z]) AND (valid_till:[2010-04-29T10:34:12Z TO *])) OR ((*:* -valid_from:[* TO *]) AND (*:* -valid_till:[* TO *])))
I use the empty checks for datasets which do not have a valid from/till range.
Is there any way to get this any faster? Would it be faster using unix-timestamps with int fields?
I would appreciate any insight and help on this.
regards,
Jan-Simon
AW: Slow Date-Range Queries
Posted by Jan Simon Winkelmann <wi...@newsfactory.de>.
For now I need them, I will however most likely (as suggested by Ahmet Arslan), create another boolean field to get rid of them, just simply due to the fact that I am switching to Solr 1.4 frange queries.
On the topic of frange queries, is there a way to simulate the date range wildcards here? They don't seem to be working for the frange.
> Do you really need the *:* stuff in the date range subqueries? That
> may add to the execution time.
Re: Slow Date-Range Queries
Posted by Lance Norskog <go...@gmail.com>.
Do you really need the *:* stuff in the date range subqueries? That
may add to the execution time.
On Thu, Apr 29, 2010 at 9:52 AM, Erick Erickson <er...@gmail.com> wrote:
> Hmmm, what does the rest of your query look like? And does adding
> &debugQuery=on show anything interesting?
>
> Best
> Erick
>
> On Thu, Apr 29, 2010 at 6:54 AM, Jan Simon Winkelmann <
> winkelmann@newsfactory.de> wrote:
>
>> > > ((valid_from:[* TO 2010-04-29T10:34:12Z]) AND
>> > > (valid_till:[2010-04-29T10:34:12Z TO *])) OR ((*:*
>> > > -valid_from:[* TO *]) AND (*:* -valid_till:[* TO *])))
>> > >
>> > > I use the empty checks for datasets which do not have a
>> > > valid from/till range.
>> > >
>> > >
>> > > Is there any way to get this any faster?
>> >
>> > I can suggest you two things.
>> >
>> > 1-) valid_till:[* TO *] and valid_from:[* TO *] type queries can be
>> > performance killer. You can create a new boolean field ( populated via
>> > conditional copy or populated client side) that holds the information
>> > whether valid_from exists or not. So that valid_till:[* TO *] can be
>> > rewritten as valid_till_bool:true.
>>
>> That may be an idea, however i checked what happens when I simply leave
>> them out. It does affect the performance but the query is still somewhere
>> around 1 second.
>>
>> > 2-) If you are embedding these queries into q parameter, you can write
>> > your clauses into (filter query) fq parameters so that they are cached.
>>
>> The problem here is, that the timestamp itself does change quite a bit and
>> hence cannot be properly cached. It could be for a few seconds, but
>> occasional response times of more than a second is still unacceptable for
>> us. We need a solution that responds quickly ALL the time, not just most of
>> the time.
>>
>> Thanks for your ideas though :)
>>
>> regards,
>> Jan-Simon
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Slow Date-Range Queries
Posted by Erick Erickson <er...@gmail.com>.
Hmmm, what does the rest of your query look like? And does adding
&debugQuery=on show anything interesting?
Best
Erick
On Thu, Apr 29, 2010 at 6:54 AM, Jan Simon Winkelmann <
winkelmann@newsfactory.de> wrote:
> > > ((valid_from:[* TO 2010-04-29T10:34:12Z]) AND
> > > (valid_till:[2010-04-29T10:34:12Z TO *])) OR ((*:*
> > > -valid_from:[* TO *]) AND (*:* -valid_till:[* TO *])))
> > >
> > > I use the empty checks for datasets which do not have a
> > > valid from/till range.
> > >
> > >
> > > Is there any way to get this any faster?
> >
> > I can suggest you two things.
> >
> > 1-) valid_till:[* TO *] and valid_from:[* TO *] type queries can be
> > performance killer. You can create a new boolean field ( populated via
> > conditional copy or populated client side) that holds the information
> > whether valid_from exists or not. So that valid_till:[* TO *] can be
> > rewritten as valid_till_bool:true.
>
> That may be an idea, however i checked what happens when I simply leave
> them out. It does affect the performance but the query is still somewhere
> around 1 second.
>
> > 2-) If you are embedding these queries into q parameter, you can write
> > your clauses into (filter query) fq parameters so that they are cached.
>
> The problem here is, that the timestamp itself does change quite a bit and
> hence cannot be properly cached. It could be for a few seconds, but
> occasional response times of more than a second is still unacceptable for
> us. We need a solution that responds quickly ALL the time, not just most of
> the time.
>
> Thanks for your ideas though :)
>
> regards,
> Jan-Simon
>
>
AW: Slow Date-Range Queries
Posted by Jan Simon Winkelmann <wi...@newsfactory.de>.
> > ((valid_from:[* TO 2010-04-29T10:34:12Z]) AND
> > (valid_till:[2010-04-29T10:34:12Z TO *])) OR ((*:*
> > -valid_from:[* TO *]) AND (*:* -valid_till:[* TO *])))
> >
> > I use the empty checks for datasets which do not have a
> > valid from/till range.
> >
> >
> > Is there any way to get this any faster?
>
> I can suggest you two things.
>
> 1-) valid_till:[* TO *] and valid_from:[* TO *] type queries can be
> performance killer. You can create a new boolean field ( populated via
> conditional copy or populated client side) that holds the information
> whether valid_from exists or not. So that valid_till:[* TO *] can be
> rewritten as valid_till_bool:true.
That may be an idea, however i checked what happens when I simply leave them out. It does affect the performance but the query is still somewhere around 1 second.
> 2-) If you are embedding these queries into q parameter, you can write
> your clauses into (filter query) fq parameters so that they are cached.
The problem here is, that the timestamp itself does change quite a bit and hence cannot be properly cached. It could be for a few seconds, but occasional response times of more than a second is still unacceptable for us. We need a solution that responds quickly ALL the time, not just most of the time.
Thanks for your ideas though :)
regards,
Jan-Simon
Re: Slow Date-Range Queries
Posted by Ahmet Arslan <io...@yahoo.com>.
> I am currently having serious performance problems with
> date range queries. What I am doing, is validating a
> datasets published status by a valid_from and a valid_till
> date field.
>
> I did get a performance boost of ~ 100% by switching from a
> normal solr.DateField to a solr.TrieDateField with
> precisionStep="8", however my query still takes about 1,3
> seconds.
>
> My field defintion looks like this:
>
> <fieldType name="date" class="solr.TrieDateField"
> precisionStep="8" sortMissingLast="true"
> omitNorms="true"/>
>
> <field name="valid_from" type="date" indexed="true"
> stored="false" required="false" />
> <field name="valid_till" type="date" indexed="true"
> stored="false" required="false" />
>
>
> And the query looks like this:
> ((valid_from:[* TO 2010-04-29T10:34:12Z]) AND
> (valid_till:[2010-04-29T10:34:12Z TO *])) OR ((*:*
> -valid_from:[* TO *]) AND (*:* -valid_till:[* TO *])))
>
> I use the empty checks for datasets which do not have a
> valid from/till range.
>
>
> Is there any way to get this any faster?
I can suggest you two things.
1-) valid_till:[* TO *] and valid_from:[* TO *] type queries can be performance killer. You can create a new boolean field ( populated via conditional copy or populated client side) that holds the information whether valid_from exists or not. So that valid_till:[* TO *] can be rewritten as valid_till_bool:true.
2-) If you are embedding these queries into q parameter, you can write your clauses into (filter query) fq parameters so that they are cached.