You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Callum Lamb <cl...@mintel.com> on 2016/07/01 10:52:48 UTC
Date range warming.
We want to warm some FQ's. The main ones we want to do being date presets
like "last 6 months", "last year" .etc
The queries for the last 6 months get generated to look like this from site
(it's really 6 months -1 day):
*date_published:([2016-01-02T00:00:00.000Z TO 2016-07-01T23:59:59.999Z])*
But since I have to represent this is in the firstSearcher section of
solrconfig.xml I need to use the date math features (Is there another way?
There doesn't seem to be a JVM system properties with the date in it, and I
don't want to have to restart solr everyday to update a solr env variable).
So I have this:
*date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-1SECOND])*
Which should resolve to the same thing. Is there someway I can check this
for sure? I get the same results when I run them.
I have a couple questions though:
1. Is Solr smart enough to see that it if the current explicit queries that
come through are the same as my date math queries and re-use the fq in this
case? Is there a way to confirm this? I can go and change them to be the
same as well, not much of an issue, more curious than anything.
2. Can Solr re-use fq's with NOW in them at all? Since NOW is changing all
the time I'm worried there some kind of checker than just sets cache=false
on all queries containing NOW or worse expands them to the current time and
caches that, and none of the fq's will ever match it (assuming solr just
does a strcmp for fq's).
Cheers,
Callum.
--
Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
Contact details for our other offices can be found at
http://www.mintel.com/office-locations.
This email and any attachments may include content that is confidential,
privileged
or otherwise protected under applicable law. Unauthorised disclosure,
copying, distribution
or use of the contents is prohibited and may be unlawful. If you have
received this email in error,
including without appropriate authorisation, then please reply to the
sender about the error
and delete this email and any attachments.
Re: Date range warming.
Posted by Erick Erickson <er...@gmail.com>.
Glad you figured it out. BTW, your query can be a little simpler
since you can mix and match inclusive/exclusive end points as:
date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY})
Note curly brace rather than square bracket at the end.
As to whether the filter cache is actually used, if you look at the
admin UI>>core>>plugins/stats>>filterCache you can see
both the hit count for the filter cache for the current searcher as well
as cumulative stats. So a manual test would be
1> reopen the searcher (or start Solr)
2> submit a filter query and see if the hit count bumped.
Best,
Erick
On Fri, Jul 1, 2016 at 6:22 AM, Callum Lamb <cl...@mintel.com> wrote:
> Okay I figured it out. Answer here if anyone ever stumbled across this in
> future.
>
> With debugQuery on you can see filter_queries actually get processed into
> what's in the parsed_filter_queries and it's those that get cached. In this
> case solr converts them to unix timestamps TO unix_timestamp and It's these
> that get cached. So you can see if they'll match by looking at those.
>
> On Fri, Jul 1, 2016 at 12:00 PM, Callum Lamb <cl...@mintel.com> wrote:
>
>> Woops, Just realised it's meant to be:
>>
>> date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-*1MILLISECOND*])
>>
>> instead.
>>
>> On Fri, Jul 1, 2016 at 11:52 AM, Callum Lamb <cl...@mintel.com> wrote:
>>
>>> We want to warm some FQ's. The main ones we want to do being date presets
>>> like "last 6 months", "last year" .etc
>>>
>>> The queries for the last 6 months get generated to look like this from
>>> site (it's really 6 months -1 day):
>>>
>>> *date_published:([2016-01-02T00:00:00.000Z TO 2016-07-01T23:59:59.999Z])*
>>>
>>> But since I have to represent this is in the firstSearcher section of
>>> solrconfig.xml I need to use the date math features (Is there another way?
>>> There doesn't seem to be a JVM system properties with the date in it, and I
>>> don't want to have to restart solr everyday to update a solr env variable).
>>>
>>> So I have this:
>>>
>>> *date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-1SECOND])*
>>>
>>> Which should resolve to the same thing. Is there someway I can check this
>>> for sure? I get the same results when I run them.
>>>
>>> I have a couple questions though:
>>>
>>> 1. Is Solr smart enough to see that it if the current explicit queries
>>> that come through are the same as my date math queries and re-use the fq in
>>> this case? Is there a way to confirm this? I can go and change them to be
>>> the same as well, not much of an issue, more curious than anything.
>>>
>>> 2. Can Solr re-use fq's with NOW in them at all? Since NOW is changing
>>> all the time I'm worried there some kind of checker than just sets
>>> cache=false on all queries containing NOW or worse expands them to the
>>> current time and caches that, and none of the fq's will ever match it
>>> (assuming solr just does a strcmp for fq's).
>>>
>>> Cheers,
>>>
>>> Callum.
>>>
>>>
>>>
>>>
>>>
>>
>
> --
>
> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
>
> Contact details for our other offices can be found at
> http://www.mintel.com/office-locations.
>
> This email and any attachments may include content that is confidential,
> privileged
> or otherwise protected under applicable law. Unauthorised disclosure,
> copying, distribution
> or use of the contents is prohibited and may be unlawful. If you have
> received this email in error,
> including without appropriate authorisation, then please reply to the
> sender about the error
> and delete this email and any attachments.
>
Re: Date range warming.
Posted by Callum Lamb <cl...@mintel.com>.
Okay I figured it out. Answer here if anyone ever stumbled across this in
future.
With debugQuery on you can see filter_queries actually get processed into
what's in the parsed_filter_queries and it's those that get cached. In this
case solr converts them to unix timestamps TO unix_timestamp and It's these
that get cached. So you can see if they'll match by looking at those.
On Fri, Jul 1, 2016 at 12:00 PM, Callum Lamb <cl...@mintel.com> wrote:
> Woops, Just realised it's meant to be:
>
> date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-*1MILLISECOND*])
>
> instead.
>
> On Fri, Jul 1, 2016 at 11:52 AM, Callum Lamb <cl...@mintel.com> wrote:
>
>> We want to warm some FQ's. The main ones we want to do being date presets
>> like "last 6 months", "last year" .etc
>>
>> The queries for the last 6 months get generated to look like this from
>> site (it's really 6 months -1 day):
>>
>> *date_published:([2016-01-02T00:00:00.000Z TO 2016-07-01T23:59:59.999Z])*
>>
>> But since I have to represent this is in the firstSearcher section of
>> solrconfig.xml I need to use the date math features (Is there another way?
>> There doesn't seem to be a JVM system properties with the date in it, and I
>> don't want to have to restart solr everyday to update a solr env variable).
>>
>> So I have this:
>>
>> *date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-1SECOND])*
>>
>> Which should resolve to the same thing. Is there someway I can check this
>> for sure? I get the same results when I run them.
>>
>> I have a couple questions though:
>>
>> 1. Is Solr smart enough to see that it if the current explicit queries
>> that come through are the same as my date math queries and re-use the fq in
>> this case? Is there a way to confirm this? I can go and change them to be
>> the same as well, not much of an issue, more curious than anything.
>>
>> 2. Can Solr re-use fq's with NOW in them at all? Since NOW is changing
>> all the time I'm worried there some kind of checker than just sets
>> cache=false on all queries containing NOW or worse expands them to the
>> current time and caches that, and none of the fq's will ever match it
>> (assuming solr just does a strcmp for fq's).
>>
>> Cheers,
>>
>> Callum.
>>
>>
>>
>>
>>
>
--
Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
Contact details for our other offices can be found at
http://www.mintel.com/office-locations.
This email and any attachments may include content that is confidential,
privileged
or otherwise protected under applicable law. Unauthorised disclosure,
copying, distribution
or use of the contents is prohibited and may be unlawful. If you have
received this email in error,
including without appropriate authorisation, then please reply to the
sender about the error
and delete this email and any attachments.
Re: Date range warming.
Posted by Callum Lamb <cl...@mintel.com>.
Woops, Just realised it's meant to be:
date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-*1MILLISECOND*])
instead.
On Fri, Jul 1, 2016 at 11:52 AM, Callum Lamb <cl...@mintel.com> wrote:
> We want to warm some FQ's. The main ones we want to do being date presets
> like "last 6 months", "last year" .etc
>
> The queries for the last 6 months get generated to look like this from
> site (it's really 6 months -1 day):
>
> *date_published:([2016-01-02T00:00:00.000Z TO 2016-07-01T23:59:59.999Z])*
>
> But since I have to represent this is in the firstSearcher section of
> solrconfig.xml I need to use the date math features (Is there another way?
> There doesn't seem to be a JVM system properties with the date in it, and I
> don't want to have to restart solr everyday to update a solr env variable).
>
> So I have this:
>
> *date_published:([NOW/DAY-6MONTH+1DAY TO NOW/DAY+1DAY-1SECOND])*
>
> Which should resolve to the same thing. Is there someway I can check this
> for sure? I get the same results when I run them.
>
> I have a couple questions though:
>
> 1. Is Solr smart enough to see that it if the current explicit queries
> that come through are the same as my date math queries and re-use the fq in
> this case? Is there a way to confirm this? I can go and change them to be
> the same as well, not much of an issue, more curious than anything.
>
> 2. Can Solr re-use fq's with NOW in them at all? Since NOW is changing all
> the time I'm worried there some kind of checker than just sets cache=false
> on all queries containing NOW or worse expands them to the current time and
> caches that, and none of the fq's will ever match it (assuming solr just
> does a strcmp for fq's).
>
> Cheers,
>
> Callum.
>
>
>
>
>
--
Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
Contact details for our other offices can be found at
http://www.mintel.com/office-locations.
This email and any attachments may include content that is confidential,
privileged
or otherwise protected under applicable law. Unauthorised disclosure,
copying, distribution
or use of the contents is prohibited and may be unlawful. If you have
received this email in error,
including without appropriate authorisation, then please reply to the
sender about the error
and delete this email and any attachments.