You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alok Dhir <ad...@symplicity.com> on 2008/10/29 21:30:41 UTC

date range query performance

Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.

Fairly simple schema -- no large text fields, standard request  
handler.  4 small facet fields.

The index is an event log -- a primary search/retrieval requirement is  
date range queries.

A simple query without a date range subquery is ridiculously fast -  
2ms.  The same query with a date range takes up to 30s (30,000ms).

Concrete example, this query just look 18s:

	instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z TO  
2008-10-30T03:59:59Z] AND label_facet:"Added to Position"

The exact same query without the date range took 2ms.

I saw a thread from Apr 2008 which explains the problem being due to  
too much precision on the DateField type, and the range expansion  
leading to far too many elements being checked.  Proposed solution  
appears to be a hack where you index date fields as strings and  
hacking together date functions to generate proper queries/format  
results.

Does this remain the recommended solution to this issue?

Thanks

---
Alok K. Dhir
Symplicity Corporation
www.symplicity.com
(703) 351-0200 x 8080
adhir@symplicity.com


Re: SOLR Performance

Posted by Otis Gospodnetic <ot...@yahoo.com>.
That depends largely on your ramBufferSizeMB setting in solrconfig.xml and the memory you are willing to give to the JVM via -Xmx.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----
> From: Alok Dhir <ad...@symplicity.com>
> To: solr-user@lucene.apache.org
> Sent: Monday, November 3, 2008 5:16:27 PM
> Subject: Re: SOLR Performance
> 
> in terms of RAM -- how to size that on the indexer?
> 
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
> 
> On Nov 3, 2008, at 4:07 PM, Walter Underwood wrote:
> 
> > The indexing box can be much smaller, especially in terms of CPU.
> > It just needs one fast thread and enough disk.
> > 
> > wunder
> > 
> > On 11/3/08 2:58 PM, "Alok Dhir" wrote:
> > 
> >> I was afraid of that.  Was hoping not to need another big fat box like
> >> this one...
> >> 
> >> ---
> >> Alok K. Dhir
> >> Symplicity Corporation
> >> www.symplicity.com
> >> (703) 351-0200 x 8080
> >> adhir@symplicity.com
> >> 
> >> On Nov 3, 2008, at 4:53 PM, Feak, Todd wrote:
> >> 
> >>> I believe this is one of the reasons that a master/slave configuration
> >>> comes in handy. Commits to the Master don't slow down queries on the
> >>> Slave.
> >>> 
> >>> -Todd
> >>> 
> >>> -----Original Message-----
> >>> From: Alok Dhir [mailto:adhir@symplicity.com]
> >>> Sent: Monday, November 03, 2008 1:47 PM
> >>> To: solr-user@lucene.apache.org
> >>> Subject: SOLR Performance
> >>> 
> >>> We've moved past this issue by reducing date precision -- thanks to
> >>> all for the help.  Now we're at another problem.
> >>> 
> >>> There is relatively constant updating of the index -- new log entries
> >>> are pumped in from several applications continuously.  Obviously, new
> >>> entries do not appear in searches until after a commit occurs.
> >>> 
> >>> The problem is, issuing a commit causes searches to come to a
> >>> screeching halt for up to 2 minutes.  We're up to around 80M docs.
> >>> Index size is 27G.  The number of docs will soon be 800M, which
> >>> doesn't bode well for these "pauses" in search performance.
> >>> 
> >>> I'd appreciate any suggestions.
> >>> 
> >>> ---
> >>> Alok K. Dhir
> >>> Symplicity Corporation
> >>> www.symplicity.com
> >>> (703) 351-0200 x 8080
> >>> adhir@symplicity.com
> >>> 
> >>> On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:
> >>> 
> >>>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
> >>>> 
> >>>> Fairly simple schema -- no large text fields, standard request
> >>>> handler.  4 small facet fields.
> >>>> 
> >>>> The index is an event log -- a primary search/retrieval requirement
> >>>> is date range queries.
> >>>> 
> >>>> A simple query without a date range subquery is ridiculously fast -
> >>>> 2ms.  The same query with a date range takes up to 30s (30,000ms).
> >>>> 
> >>>> Concrete example, this query just look 18s:
> >>>> 
> >>>> instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z
> >>> TO
> >>>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
> >>>> 
> >>>> The exact same query without the date range took 2ms.
> >>>> 
> >>>> I saw a thread from Apr 2008 which explains the problem being due to
> >>>> too much precision on the DateField type, and the range expansion
> >>>> leading to far too many elements being checked.  Proposed solution
> >>>> appears to be a hack where you index date fields as strings and
> >>>> hacking together date functions to generate proper queries/format
> >>>> results.
> >>>> 
> >>>> Does this remain the recommended solution to this issue?
> >>>> 
> >>>> Thanks
> >>>> 
> >>>> ---
> >>>> Alok K. Dhir
> >>>> Symplicity Corporation
> >>>> www.symplicity.com
> >>>> (703) 351-0200 x 8080
> >>>> adhir@symplicity.com
> >>>> 
> >>> 
> >>> 
> >> 
> > 


RE: SOLR Performance

Posted by "Feak, Todd" <To...@smss.sony.com>.
Most desktops nowadays have at least a dual-core and 1GB, you may be
able to get a semi-realistic feel for performance on a local desktop. If
you have access to something meaty in a desktop, you may not have to
spend a dime to find out what it's going to take in a server.

-T

-----Original Message-----
From: Mike Klaas [mailto:mike.klaas@gmail.com] 
Sent: Monday, November 03, 2008 4:25 PM
To: solr-user@lucene.apache.org
Subject: Re: SOLR Performance

If you never execute any queries, a gig should be more than enough.

Of course, I've never played around with a .8 billion doc corpus on  
one machine.

-Mike

On 3-Nov-08, at 2:16 PM, Alok Dhir wrote:

> in terms of RAM -- how to size that on the indexer?
>
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
>
> On Nov 3, 2008, at 4:07 PM, Walter Underwood wrote:
>
>> The indexing box can be much smaller, especially in terms of CPU.
>> It just needs one fast thread and enough disk.
>>
>> wunder
>>
>> On 11/3/08 2:58 PM, "Alok Dhir" <ad...@symplicity.com> wrote:
>>
>>> I was afraid of that.  Was hoping not to need another big fat box  
>>> like
>>> this one...
>>>
>>> ---
>>> Alok K. Dhir
>>> Symplicity Corporation
>>> www.symplicity.com
>>> (703) 351-0200 x 8080
>>> adhir@symplicity.com
>>>
>>> On Nov 3, 2008, at 4:53 PM, Feak, Todd wrote:
>>>
>>>> I believe this is one of the reasons that a master/slave  
>>>> configuration
>>>> comes in handy. Commits to the Master don't slow down queries on  
>>>> the
>>>> Slave.
>>>>
>>>> -Todd
>>>>
>>>> -----Original Message-----
>>>> From: Alok Dhir [mailto:adhir@symplicity.com]
>>>> Sent: Monday, November 03, 2008 1:47 PM
>>>> To: solr-user@lucene.apache.org
>>>> Subject: SOLR Performance
>>>>
>>>> We've moved past this issue by reducing date precision -- thanks to
>>>> all for the help.  Now we're at another problem.
>>>>
>>>> There is relatively constant updating of the index -- new log  
>>>> entries
>>>> are pumped in from several applications continuously.  Obviously,  
>>>> new
>>>> entries do not appear in searches until after a commit occurs.
>>>>
>>>> The problem is, issuing a commit causes searches to come to a
>>>> screeching halt for up to 2 minutes.  We're up to around 80M docs.
>>>> Index size is 27G.  The number of docs will soon be 800M, which
>>>> doesn't bode well for these "pauses" in search performance.
>>>>
>>>> I'd appreciate any suggestions.
>>>>
>>>> ---
>>>> Alok K. Dhir
>>>> Symplicity Corporation
>>>> www.symplicity.com
>>>> (703) 351-0200 x 8080
>>>> adhir@symplicity.com
>>>>
>>>> On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:
>>>>
>>>>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core  
>>>>> machine.
>>>>>
>>>>> Fairly simple schema -- no large text fields, standard request
>>>>> handler.  4 small facet fields.
>>>>>
>>>>> The index is an event log -- a primary search/retrieval  
>>>>> requirement
>>>>> is date range queries.
>>>>>
>>>>> A simple query without a date range subquery is ridiculously  
>>>>> fast -
>>>>> 2ms.  The same query with a date range takes up to 30s (30,000ms).
>>>>>
>>>>> Concrete example, this query just look 18s:
>>>>>
>>>>> instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z
>>>> TO
>>>>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>>>>
>>>>> The exact same query without the date range took 2ms.
>>>>>
>>>>> I saw a thread from Apr 2008 which explains the problem being  
>>>>> due to
>>>>> too much precision on the DateField type, and the range expansion
>>>>> leading to far too many elements being checked.  Proposed solution
>>>>> appears to be a hack where you index date fields as strings and
>>>>> hacking together date functions to generate proper queries/format
>>>>> results.
>>>>>
>>>>> Does this remain the recommended solution to this issue?
>>>>>
>>>>> Thanks
>>>>>
>>>>> ---
>>>>> Alok K. Dhir
>>>>> Symplicity Corporation
>>>>> www.symplicity.com
>>>>> (703) 351-0200 x 8080
>>>>> adhir@symplicity.com
>>>>>
>>>>
>>>>
>>>
>>
>



Re: SOLR Performance

Posted by Walter Underwood <wu...@netflix.com>.
Funny, that is exactly what Infoseek did back in 1996. A big index that
changed rarely and a small index with real-time changes. Once each week,
merge to make a new big index and start over with the small one.

You also need to handle deletes specially.

wunder

On 11/3/08 6:44 PM, "Lance Norskog" <go...@gmail.com> wrote:

> The logistics of handling giant index files hit us before search
> performance. We switched to a set of indexes running inside one server
> (tomcat) instance with the Multicore+Distributed Search tools, with a frozen
> old index and a new index actively taking updates. The smaller new index
> takes much less time to recover after a commit.
> 
> The DS code does not handle cases where the new and old index have different
> versions of the same document. We wrote a custom distributed search that
> favored the "new" index over the "old".
> 
> Lance
> 
> -----Original Message-----
> From: Mike Klaas [mailto:mike.klaas@gmail.com]
> Sent: Monday, November 03, 2008 4:25 PM
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR Performance
> 
> If you never execute any queries, a gig should be more than enough.
> 
> Of course, I've never played around with a .8 billion doc corpus on one
> machine.
> 
> -Mike
> 
> On 3-Nov-08, at 2:16 PM, Alok Dhir wrote:
> 
>> in terms of RAM -- how to size that on the indexer?
>> 
>> ---
>> Alok K. Dhir
>> Symplicity Corporation
>> www.symplicity.com
>> (703) 351-0200 x 8080
>> adhir@symplicity.com
>> 
>> On Nov 3, 2008, at 4:07 PM, Walter Underwood wrote:
>> 
>>> The indexing box can be much smaller, especially in terms of CPU.
>>> It just needs one fast thread and enough disk.
>>> 
>>> wunder
>>> 
>>> On 11/3/08 2:58 PM, "Alok Dhir" <ad...@symplicity.com> wrote:
>>> 
>>>> I was afraid of that.  Was hoping not to need another big fat box
>>>> like this one...
>>>> 
>>>> ---
>>>> Alok K. Dhir
>>>> Symplicity Corporation
>>>> www.symplicity.com
>>>> (703) 351-0200 x 8080
>>>> adhir@symplicity.com
>>>> 
>>>> On Nov 3, 2008, at 4:53 PM, Feak, Todd wrote:
>>>> 
>>>>> I believe this is one of the reasons that a master/slave
>>>>> configuration comes in handy. Commits to the Master don't slow down
>>>>> queries on the Slave.
>>>>> 
>>>>> -Todd
>>>>> 
>>>>> -----Original Message-----
>>>>> From: Alok Dhir [mailto:adhir@symplicity.com]
>>>>> Sent: Monday, November 03, 2008 1:47 PM
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: SOLR Performance
>>>>> 
>>>>> We've moved past this issue by reducing date precision -- thanks to
>>>>> all for the help.  Now we're at another problem.
>>>>> 
>>>>> There is relatively constant updating of the index -- new log
>>>>> entries are pumped in from several applications continuously.
>>>>> Obviously, new entries do not appear in searches until after a
>>>>> commit occurs.
>>>>> 
>>>>> The problem is, issuing a commit causes searches to come to a
>>>>> screeching halt for up to 2 minutes.  We're up to around 80M docs.
>>>>> Index size is 27G.  The number of docs will soon be 800M, which
>>>>> doesn't bode well for these "pauses" in search performance.
>>>>> 
>>>>> I'd appreciate any suggestions.
>>>>> 
>>>>> ---
>>>>> Alok K. Dhir
>>>>> Symplicity Corporation
>>>>> www.symplicity.com
>>>>> (703) 351-0200 x 8080
>>>>> adhir@symplicity.com
>>>>> 
>>>>> On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:
>>>>> 
>>>>>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core
>>>>>> machine.
>>>>>> 
>>>>>> Fairly simple schema -- no large text fields, standard request
>>>>>> handler.  4 small facet fields.
>>>>>> 
>>>>>> The index is an event log -- a primary search/retrieval
>>>>>> requirement is date range queries.
>>>>>> 
>>>>>> A simple query without a date range subquery is ridiculously fast
>>>>>> - 2ms.  The same query with a date range takes up to 30s
>>>>>> (30,000ms).
>>>>>> 
>>>>>> Concrete example, this query just look 18s:
>>>>>> 
>>>>>> instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z
>>>>> TO
>>>>>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>>>>> 
>>>>>> The exact same query without the date range took 2ms.
>>>>>> 
>>>>>> I saw a thread from Apr 2008 which explains the problem being due
>>>>>> to too much precision on the DateField type, and the range
>>>>>> expansion leading to far too many elements being checked.
>>>>>> Proposed solution appears to be a hack where you index date fields
>>>>>> as strings and hacking together date functions to generate proper
>>>>>> queries/format results.
>>>>>> 
>>>>>> Does this remain the recommended solution to this issue?
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>> ---
>>>>>> Alok K. Dhir
>>>>>> Symplicity Corporation
>>>>>> www.symplicity.com
>>>>>> (703) 351-0200 x 8080
>>>>>> adhir@symplicity.com
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 
> 


RE: SOLR Performance

Posted by Lance Norskog <go...@gmail.com>.
The logistics of handling giant index files hit us before search
performance. We switched to a set of indexes running inside one server
(tomcat) instance with the Multicore+Distributed Search tools, with a frozen
old index and a new index actively taking updates. The smaller new index
takes much less time to recover after a commit.

The DS code does not handle cases where the new and old index have different
versions of the same document. We wrote a custom distributed search that
favored the "new" index over the "old".

Lance

-----Original Message-----
From: Mike Klaas [mailto:mike.klaas@gmail.com] 
Sent: Monday, November 03, 2008 4:25 PM
To: solr-user@lucene.apache.org
Subject: Re: SOLR Performance

If you never execute any queries, a gig should be more than enough.

Of course, I've never played around with a .8 billion doc corpus on one
machine.

-Mike

On 3-Nov-08, at 2:16 PM, Alok Dhir wrote:

> in terms of RAM -- how to size that on the indexer?
>
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
>
> On Nov 3, 2008, at 4:07 PM, Walter Underwood wrote:
>
>> The indexing box can be much smaller, especially in terms of CPU.
>> It just needs one fast thread and enough disk.
>>
>> wunder
>>
>> On 11/3/08 2:58 PM, "Alok Dhir" <ad...@symplicity.com> wrote:
>>
>>> I was afraid of that.  Was hoping not to need another big fat box 
>>> like this one...
>>>
>>> ---
>>> Alok K. Dhir
>>> Symplicity Corporation
>>> www.symplicity.com
>>> (703) 351-0200 x 8080
>>> adhir@symplicity.com
>>>
>>> On Nov 3, 2008, at 4:53 PM, Feak, Todd wrote:
>>>
>>>> I believe this is one of the reasons that a master/slave 
>>>> configuration comes in handy. Commits to the Master don't slow down 
>>>> queries on the Slave.
>>>>
>>>> -Todd
>>>>
>>>> -----Original Message-----
>>>> From: Alok Dhir [mailto:adhir@symplicity.com]
>>>> Sent: Monday, November 03, 2008 1:47 PM
>>>> To: solr-user@lucene.apache.org
>>>> Subject: SOLR Performance
>>>>
>>>> We've moved past this issue by reducing date precision -- thanks to 
>>>> all for the help.  Now we're at another problem.
>>>>
>>>> There is relatively constant updating of the index -- new log 
>>>> entries are pumped in from several applications continuously.  
>>>> Obviously, new entries do not appear in searches until after a 
>>>> commit occurs.
>>>>
>>>> The problem is, issuing a commit causes searches to come to a 
>>>> screeching halt for up to 2 minutes.  We're up to around 80M docs.
>>>> Index size is 27G.  The number of docs will soon be 800M, which 
>>>> doesn't bode well for these "pauses" in search performance.
>>>>
>>>> I'd appreciate any suggestions.
>>>>
>>>> ---
>>>> Alok K. Dhir
>>>> Symplicity Corporation
>>>> www.symplicity.com
>>>> (703) 351-0200 x 8080
>>>> adhir@symplicity.com
>>>>
>>>> On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:
>>>>
>>>>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core 
>>>>> machine.
>>>>>
>>>>> Fairly simple schema -- no large text fields, standard request 
>>>>> handler.  4 small facet fields.
>>>>>
>>>>> The index is an event log -- a primary search/retrieval 
>>>>> requirement is date range queries.
>>>>>
>>>>> A simple query without a date range subquery is ridiculously fast 
>>>>> - 2ms.  The same query with a date range takes up to 30s 
>>>>> (30,000ms).
>>>>>
>>>>> Concrete example, this query just look 18s:
>>>>>
>>>>> instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z
>>>> TO
>>>>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>>>>
>>>>> The exact same query without the date range took 2ms.
>>>>>
>>>>> I saw a thread from Apr 2008 which explains the problem being due 
>>>>> to too much precision on the DateField type, and the range 
>>>>> expansion leading to far too many elements being checked.  
>>>>> Proposed solution appears to be a hack where you index date fields 
>>>>> as strings and hacking together date functions to generate proper 
>>>>> queries/format results.
>>>>>
>>>>> Does this remain the recommended solution to this issue?
>>>>>
>>>>> Thanks
>>>>>
>>>>> ---
>>>>> Alok K. Dhir
>>>>> Symplicity Corporation
>>>>> www.symplicity.com
>>>>> (703) 351-0200 x 8080
>>>>> adhir@symplicity.com
>>>>>
>>>>
>>>>
>>>
>>
>



Re: SOLR Performance

Posted by Mike Klaas <mi...@gmail.com>.
If you never execute any queries, a gig should be more than enough.

Of course, I've never played around with a .8 billion doc corpus on  
one machine.

-Mike

On 3-Nov-08, at 2:16 PM, Alok Dhir wrote:

> in terms of RAM -- how to size that on the indexer?
>
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
>
> On Nov 3, 2008, at 4:07 PM, Walter Underwood wrote:
>
>> The indexing box can be much smaller, especially in terms of CPU.
>> It just needs one fast thread and enough disk.
>>
>> wunder
>>
>> On 11/3/08 2:58 PM, "Alok Dhir" <ad...@symplicity.com> wrote:
>>
>>> I was afraid of that.  Was hoping not to need another big fat box  
>>> like
>>> this one...
>>>
>>> ---
>>> Alok K. Dhir
>>> Symplicity Corporation
>>> www.symplicity.com
>>> (703) 351-0200 x 8080
>>> adhir@symplicity.com
>>>
>>> On Nov 3, 2008, at 4:53 PM, Feak, Todd wrote:
>>>
>>>> I believe this is one of the reasons that a master/slave  
>>>> configuration
>>>> comes in handy. Commits to the Master don't slow down queries on  
>>>> the
>>>> Slave.
>>>>
>>>> -Todd
>>>>
>>>> -----Original Message-----
>>>> From: Alok Dhir [mailto:adhir@symplicity.com]
>>>> Sent: Monday, November 03, 2008 1:47 PM
>>>> To: solr-user@lucene.apache.org
>>>> Subject: SOLR Performance
>>>>
>>>> We've moved past this issue by reducing date precision -- thanks to
>>>> all for the help.  Now we're at another problem.
>>>>
>>>> There is relatively constant updating of the index -- new log  
>>>> entries
>>>> are pumped in from several applications continuously.  Obviously,  
>>>> new
>>>> entries do not appear in searches until after a commit occurs.
>>>>
>>>> The problem is, issuing a commit causes searches to come to a
>>>> screeching halt for up to 2 minutes.  We're up to around 80M docs.
>>>> Index size is 27G.  The number of docs will soon be 800M, which
>>>> doesn't bode well for these "pauses" in search performance.
>>>>
>>>> I'd appreciate any suggestions.
>>>>
>>>> ---
>>>> Alok K. Dhir
>>>> Symplicity Corporation
>>>> www.symplicity.com
>>>> (703) 351-0200 x 8080
>>>> adhir@symplicity.com
>>>>
>>>> On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:
>>>>
>>>>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core  
>>>>> machine.
>>>>>
>>>>> Fairly simple schema -- no large text fields, standard request
>>>>> handler.  4 small facet fields.
>>>>>
>>>>> The index is an event log -- a primary search/retrieval  
>>>>> requirement
>>>>> is date range queries.
>>>>>
>>>>> A simple query without a date range subquery is ridiculously  
>>>>> fast -
>>>>> 2ms.  The same query with a date range takes up to 30s (30,000ms).
>>>>>
>>>>> Concrete example, this query just look 18s:
>>>>>
>>>>> instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z
>>>> TO
>>>>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>>>>
>>>>> The exact same query without the date range took 2ms.
>>>>>
>>>>> I saw a thread from Apr 2008 which explains the problem being  
>>>>> due to
>>>>> too much precision on the DateField type, and the range expansion
>>>>> leading to far too many elements being checked.  Proposed solution
>>>>> appears to be a hack where you index date fields as strings and
>>>>> hacking together date functions to generate proper queries/format
>>>>> results.
>>>>>
>>>>> Does this remain the recommended solution to this issue?
>>>>>
>>>>> Thanks
>>>>>
>>>>> ---
>>>>> Alok K. Dhir
>>>>> Symplicity Corporation
>>>>> www.symplicity.com
>>>>> (703) 351-0200 x 8080
>>>>> adhir@symplicity.com
>>>>>
>>>>
>>>>
>>>
>>
>


Re: SOLR Performance

Posted by Alok Dhir <ad...@symplicity.com>.
in terms of RAM -- how to size that on the indexer?

---
Alok K. Dhir
Symplicity Corporation
www.symplicity.com
(703) 351-0200 x 8080
adhir@symplicity.com

On Nov 3, 2008, at 4:07 PM, Walter Underwood wrote:

> The indexing box can be much smaller, especially in terms of CPU.
> It just needs one fast thread and enough disk.
>
> wunder
>
> On 11/3/08 2:58 PM, "Alok Dhir" <ad...@symplicity.com> wrote:
>
>> I was afraid of that.  Was hoping not to need another big fat box  
>> like
>> this one...
>>
>> ---
>> Alok K. Dhir
>> Symplicity Corporation
>> www.symplicity.com
>> (703) 351-0200 x 8080
>> adhir@symplicity.com
>>
>> On Nov 3, 2008, at 4:53 PM, Feak, Todd wrote:
>>
>>> I believe this is one of the reasons that a master/slave  
>>> configuration
>>> comes in handy. Commits to the Master don't slow down queries on the
>>> Slave.
>>>
>>> -Todd
>>>
>>> -----Original Message-----
>>> From: Alok Dhir [mailto:adhir@symplicity.com]
>>> Sent: Monday, November 03, 2008 1:47 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: SOLR Performance
>>>
>>> We've moved past this issue by reducing date precision -- thanks to
>>> all for the help.  Now we're at another problem.
>>>
>>> There is relatively constant updating of the index -- new log  
>>> entries
>>> are pumped in from several applications continuously.  Obviously,  
>>> new
>>> entries do not appear in searches until after a commit occurs.
>>>
>>> The problem is, issuing a commit causes searches to come to a
>>> screeching halt for up to 2 minutes.  We're up to around 80M docs.
>>> Index size is 27G.  The number of docs will soon be 800M, which
>>> doesn't bode well for these "pauses" in search performance.
>>>
>>> I'd appreciate any suggestions.
>>>
>>> ---
>>> Alok K. Dhir
>>> Symplicity Corporation
>>> www.symplicity.com
>>> (703) 351-0200 x 8080
>>> adhir@symplicity.com
>>>
>>> On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:
>>>
>>>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core  
>>>> machine.
>>>>
>>>> Fairly simple schema -- no large text fields, standard request
>>>> handler.  4 small facet fields.
>>>>
>>>> The index is an event log -- a primary search/retrieval requirement
>>>> is date range queries.
>>>>
>>>> A simple query without a date range subquery is ridiculously fast -
>>>> 2ms.  The same query with a date range takes up to 30s (30,000ms).
>>>>
>>>> Concrete example, this query just look 18s:
>>>>
>>>> instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z
>>> TO
>>>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>>>
>>>> The exact same query without the date range took 2ms.
>>>>
>>>> I saw a thread from Apr 2008 which explains the problem being due  
>>>> to
>>>> too much precision on the DateField type, and the range expansion
>>>> leading to far too many elements being checked.  Proposed solution
>>>> appears to be a hack where you index date fields as strings and
>>>> hacking together date functions to generate proper queries/format
>>>> results.
>>>>
>>>> Does this remain the recommended solution to this issue?
>>>>
>>>> Thanks
>>>>
>>>> ---
>>>> Alok K. Dhir
>>>> Symplicity Corporation
>>>> www.symplicity.com
>>>> (703) 351-0200 x 8080
>>>> adhir@symplicity.com
>>>>
>>>
>>>
>>
>


Re: SOLR Performance

Posted by Walter Underwood <wu...@netflix.com>.
The indexing box can be much smaller, especially in terms of CPU.
It just needs one fast thread and enough disk.

wunder

On 11/3/08 2:58 PM, "Alok Dhir" <ad...@symplicity.com> wrote:

> I was afraid of that.  Was hoping not to need another big fat box like
> this one...
> 
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
> 
> On Nov 3, 2008, at 4:53 PM, Feak, Todd wrote:
> 
>> I believe this is one of the reasons that a master/slave configuration
>> comes in handy. Commits to the Master don't slow down queries on the
>> Slave.
>> 
>> -Todd
>> 
>> -----Original Message-----
>> From: Alok Dhir [mailto:adhir@symplicity.com]
>> Sent: Monday, November 03, 2008 1:47 PM
>> To: solr-user@lucene.apache.org
>> Subject: SOLR Performance
>> 
>> We've moved past this issue by reducing date precision -- thanks to
>> all for the help.  Now we're at another problem.
>> 
>> There is relatively constant updating of the index -- new log entries
>> are pumped in from several applications continuously.  Obviously, new
>> entries do not appear in searches until after a commit occurs.
>> 
>> The problem is, issuing a commit causes searches to come to a
>> screeching halt for up to 2 minutes.  We're up to around 80M docs.
>> Index size is 27G.  The number of docs will soon be 800M, which
>> doesn't bode well for these "pauses" in search performance.
>> 
>> I'd appreciate any suggestions.
>> 
>> ---
>> Alok K. Dhir
>> Symplicity Corporation
>> www.symplicity.com
>> (703) 351-0200 x 8080
>> adhir@symplicity.com
>> 
>> On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:
>> 
>>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
>>> 
>>> Fairly simple schema -- no large text fields, standard request
>>> handler.  4 small facet fields.
>>> 
>>> The index is an event log -- a primary search/retrieval requirement
>>> is date range queries.
>>> 
>>> A simple query without a date range subquery is ridiculously fast -
>>> 2ms.  The same query with a date range takes up to 30s (30,000ms).
>>> 
>>> Concrete example, this query just look 18s:
>>> 
>>> instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z
>> TO
>>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>> 
>>> The exact same query without the date range took 2ms.
>>> 
>>> I saw a thread from Apr 2008 which explains the problem being due to
>>> too much precision on the DateField type, and the range expansion
>>> leading to far too many elements being checked.  Proposed solution
>>> appears to be a hack where you index date fields as strings and
>>> hacking together date functions to generate proper queries/format
>>> results.
>>> 
>>> Does this remain the recommended solution to this issue?
>>> 
>>> Thanks
>>> 
>>> ---
>>> Alok K. Dhir
>>> Symplicity Corporation
>>> www.symplicity.com
>>> (703) 351-0200 x 8080
>>> adhir@symplicity.com
>>> 
>> 
>> 
> 


Re: SOLR Performance

Posted by Alok Dhir <ad...@symplicity.com>.
I was afraid of that.  Was hoping not to need another big fat box like  
this one...

---
Alok K. Dhir
Symplicity Corporation
www.symplicity.com
(703) 351-0200 x 8080
adhir@symplicity.com

On Nov 3, 2008, at 4:53 PM, Feak, Todd wrote:

> I believe this is one of the reasons that a master/slave configuration
> comes in handy. Commits to the Master don't slow down queries on the
> Slave.
>
> -Todd
>
> -----Original Message-----
> From: Alok Dhir [mailto:adhir@symplicity.com]
> Sent: Monday, November 03, 2008 1:47 PM
> To: solr-user@lucene.apache.org
> Subject: SOLR Performance
>
> We've moved past this issue by reducing date precision -- thanks to
> all for the help.  Now we're at another problem.
>
> There is relatively constant updating of the index -- new log entries
> are pumped in from several applications continuously.  Obviously, new
> entries do not appear in searches until after a commit occurs.
>
> The problem is, issuing a commit causes searches to come to a
> screeching halt for up to 2 minutes.  We're up to around 80M docs.
> Index size is 27G.  The number of docs will soon be 800M, which
> doesn't bode well for these "pauses" in search performance.
>
> I'd appreciate any suggestions.
>
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
>
> On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:
>
>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
>>
>> Fairly simple schema -- no large text fields, standard request
>> handler.  4 small facet fields.
>>
>> The index is an event log -- a primary search/retrieval requirement
>> is date range queries.
>>
>> A simple query without a date range subquery is ridiculously fast -
>> 2ms.  The same query with a date range takes up to 30s (30,000ms).
>>
>> Concrete example, this query just look 18s:
>>
>> 	instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z
> TO
>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>
>> The exact same query without the date range took 2ms.
>>
>> I saw a thread from Apr 2008 which explains the problem being due to
>> too much precision on the DateField type, and the range expansion
>> leading to far too many elements being checked.  Proposed solution
>> appears to be a hack where you index date fields as strings and
>> hacking together date functions to generate proper queries/format
>> results.
>>
>> Does this remain the recommended solution to this issue?
>>
>> Thanks
>>
>> ---
>> Alok K. Dhir
>> Symplicity Corporation
>> www.symplicity.com
>> (703) 351-0200 x 8080
>> adhir@symplicity.com
>>
>
>


RE: SOLR Performance

Posted by "Feak, Todd" <To...@smss.sony.com>.
I believe this is one of the reasons that a master/slave configuration
comes in handy. Commits to the Master don't slow down queries on the
Slave.

-Todd

-----Original Message-----
From: Alok Dhir [mailto:adhir@symplicity.com] 
Sent: Monday, November 03, 2008 1:47 PM
To: solr-user@lucene.apache.org
Subject: SOLR Performance

We've moved past this issue by reducing date precision -- thanks to  
all for the help.  Now we're at another problem.

There is relatively constant updating of the index -- new log entries  
are pumped in from several applications continuously.  Obviously, new  
entries do not appear in searches until after a commit occurs.

The problem is, issuing a commit causes searches to come to a  
screeching halt for up to 2 minutes.  We're up to around 80M docs.   
Index size is 27G.  The number of docs will soon be 800M, which  
doesn't bode well for these "pauses" in search performance.

I'd appreciate any suggestions.

---
Alok K. Dhir
Symplicity Corporation
www.symplicity.com
(703) 351-0200 x 8080
adhir@symplicity.com

On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:

> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
>
> Fairly simple schema -- no large text fields, standard request  
> handler.  4 small facet fields.
>
> The index is an event log -- a primary search/retrieval requirement  
> is date range queries.
>
> A simple query without a date range subquery is ridiculously fast -  
> 2ms.  The same query with a date range takes up to 30s (30,000ms).
>
> Concrete example, this query just look 18s:
>
> 	instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z
TO  
> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>
> The exact same query without the date range took 2ms.
>
> I saw a thread from Apr 2008 which explains the problem being due to  
> too much precision on the DateField type, and the range expansion  
> leading to far too many elements being checked.  Proposed solution  
> appears to be a hack where you index date fields as strings and  
> hacking together date functions to generate proper queries/format  
> results.
>
> Does this remain the recommended solution to this issue?
>
> Thanks
>
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
>



SOLR Performance

Posted by Alok Dhir <ad...@symplicity.com>.
We've moved past this issue by reducing date precision -- thanks to  
all for the help.  Now we're at another problem.

There is relatively constant updating of the index -- new log entries  
are pumped in from several applications continuously.  Obviously, new  
entries do not appear in searches until after a commit occurs.

The problem is, issuing a commit causes searches to come to a  
screeching halt for up to 2 minutes.  We're up to around 80M docs.   
Index size is 27G.  The number of docs will soon be 800M, which  
doesn't bode well for these "pauses" in search performance.

I'd appreciate any suggestions.

---
Alok K. Dhir
Symplicity Corporation
www.symplicity.com
(703) 351-0200 x 8080
adhir@symplicity.com

On Oct 29, 2008, at 4:30 PM, Alok Dhir wrote:

> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
>
> Fairly simple schema -- no large text fields, standard request  
> handler.  4 small facet fields.
>
> The index is an event log -- a primary search/retrieval requirement  
> is date range queries.
>
> A simple query without a date range subquery is ridiculously fast -  
> 2ms.  The same query with a date range takes up to 30s (30,000ms).
>
> Concrete example, this query just look 18s:
>
> 	instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z TO  
> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>
> The exact same query without the date range took 2ms.
>
> I saw a thread from Apr 2008 which explains the problem being due to  
> too much precision on the DateField type, and the range expansion  
> leading to far too many elements being checked.  Proposed solution  
> appears to be a hack where you index date fields as strings and  
> hacking together date functions to generate proper queries/format  
> results.
>
> Does this remain the recommended solution to this issue?
>
> Thanks
>
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
>


RE: date range query performance

Posted by "Feak, Todd" <To...@smss.sony.com>.
It strikes me that removing just the seconds could very well reduce
overhead to 1/60 of original. 30 second query turns into 500ms query.
Just a swag though.

-Todd

-----Original Message-----
From: Alok Dhir [mailto:adhir@symplicity.com] 
Sent: Wednesday, October 29, 2008 1:48 PM
To: solr-user@lucene.apache.org
Subject: Re: date range query performance

Well, no - we don't care so much about the seconds, but hours &  
minutes are indeed crucial.

---
Alok K. Dhir
Symplicity Corporation
www.symplicity.com
(703) 351-0200 x 8080
adhir@symplicity.com

On Oct 29, 2008, at 4:41 PM, Chris Harris wrote:

> Do you need to search down to the minutes and seconds level? If  
> searching by
> date provides sufficient granularity, for instance, you can  
> normalize all
> the time-of-day portions of the timestamps to midnight while  
> indexing. (So
> index any event happening on Oct 01, 2008 as 2008-10-01T00:00:00Z.)  
> That
> would give Solr many fewer unique timestamp values to go through.
>
> On Wed, Oct 29, 2008 at 1:30 PM, Alok Dhir <ad...@symplicity.com>  
> wrote:
>
>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
>>
>> Fairly simple schema -- no large text fields, standard request  
>> handler.  4
>> small facet fields.
>>
>> The index is an event log -- a primary search/retrieval requirement  
>> is date
>> range queries.
>>
>> A simple query without a date range subquery is ridiculously fast -  
>> 2ms.
>> The same query with a date range takes up to 30s (30,000ms).
>>
>> Concrete example, this query just look 18s:
>>
>>       instance:client\-csm.symplicity.com AND dt: 
>> [2008-10-01T04:00:00Z TO
>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>
>> The exact same query without the date range took 2ms.
>>
>> I saw a thread from Apr 2008 which explains the problem being due  
>> to too
>> much precision on the DateField type, and the range expansion  
>> leading to far
>> too many elements being checked.  Proposed solution appears to be a  
>> hack
>> where you index date fields as strings and hacking together date  
>> functions
>> to generate proper queries/format results.
>>
>> Does this remain the recommended solution to this issue?
>>
>> Thanks
>>
>> ---
>> Alok K. Dhir
>> Symplicity Corporation
>> www.symplicity.com
>> (703) 351-0200 x 8080
>> adhir@symplicity.com
>>
>>



Re: date range query performance

Posted by Erick Erickson <er...@gmail.com>.
I've also seen the suggestion (more from a pure Lucene perspective) of
breaking
apart your dates. Remember that the time/space issues are due to the number
of
terms. So it's possible (although I haven't tried it) to, index many fewer
distinct
terms. e.g. break your dates into some number of fields, say

<year><month><day><hour><minute><second><millisecond>
or maybe
<YearMonthDay><HourMinute><SecondMillisecond>
or
<YearMonthDayHour><MinuteSecond><Millisecond>
or...

and then dance fancy with the queries you generate.

I haven't tried this myself, but if you do I'd really like to get an idea of
how much it improves your results.
Of course I'm not real sure how to accomplish this in SOLR, but free advice
is definitely worth what you pay for it <GG>

Best
Erick

On Wed, Oct 29, 2008 at 4:47 PM, Alok Dhir <ad...@symplicity.com> wrote:

> Well, no - we don't care so much about the seconds, but hours & minutes are
> indeed crucial.
>
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
>
> On Oct 29, 2008, at 4:41 PM, Chris Harris wrote:
>
>  Do you need to search down to the minutes and seconds level? If searching
>> by
>> date provides sufficient granularity, for instance, you can normalize all
>> the time-of-day portions of the timestamps to midnight while indexing. (So
>> index any event happening on Oct 01, 2008 as 2008-10-01T00:00:00Z.) That
>> would give Solr many fewer unique timestamp values to go through.
>>
>> On Wed, Oct 29, 2008 at 1:30 PM, Alok Dhir <ad...@symplicity.com> wrote:
>>
>>  Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
>>>
>>> Fairly simple schema -- no large text fields, standard request handler.
>>>  4
>>> small facet fields.
>>>
>>> The index is an event log -- a primary search/retrieval requirement is
>>> date
>>> range queries.
>>>
>>> A simple query without a date range subquery is ridiculously fast - 2ms.
>>> The same query with a date range takes up to 30s (30,000ms).
>>>
>>> Concrete example, this query just look 18s:
>>>
>>>      instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z TO
>>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>>
>>> The exact same query without the date range took 2ms.
>>>
>>> I saw a thread from Apr 2008 which explains the problem being due to too
>>> much precision on the DateField type, and the range expansion leading to
>>> far
>>> too many elements being checked.  Proposed solution appears to be a hack
>>> where you index date fields as strings and hacking together date
>>> functions
>>> to generate proper queries/format results.
>>>
>>> Does this remain the recommended solution to this issue?
>>>
>>> Thanks
>>>
>>> ---
>>> Alok K. Dhir
>>> Symplicity Corporation
>>> www.symplicity.com
>>> (703) 351-0200 x 8080
>>> adhir@symplicity.com
>>>
>>>
>>>
>

Re: date range query performance

Posted by Alok Dhir <ad...@symplicity.com>.
Well, no - we don't care so much about the seconds, but hours &  
minutes are indeed crucial.

---
Alok K. Dhir
Symplicity Corporation
www.symplicity.com
(703) 351-0200 x 8080
adhir@symplicity.com

On Oct 29, 2008, at 4:41 PM, Chris Harris wrote:

> Do you need to search down to the minutes and seconds level? If  
> searching by
> date provides sufficient granularity, for instance, you can  
> normalize all
> the time-of-day portions of the timestamps to midnight while  
> indexing. (So
> index any event happening on Oct 01, 2008 as 2008-10-01T00:00:00Z.)  
> That
> would give Solr many fewer unique timestamp values to go through.
>
> On Wed, Oct 29, 2008 at 1:30 PM, Alok Dhir <ad...@symplicity.com>  
> wrote:
>
>> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
>>
>> Fairly simple schema -- no large text fields, standard request  
>> handler.  4
>> small facet fields.
>>
>> The index is an event log -- a primary search/retrieval requirement  
>> is date
>> range queries.
>>
>> A simple query without a date range subquery is ridiculously fast -  
>> 2ms.
>> The same query with a date range takes up to 30s (30,000ms).
>>
>> Concrete example, this query just look 18s:
>>
>>       instance:client\-csm.symplicity.com AND dt: 
>> [2008-10-01T04:00:00Z TO
>> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>>
>> The exact same query without the date range took 2ms.
>>
>> I saw a thread from Apr 2008 which explains the problem being due  
>> to too
>> much precision on the DateField type, and the range expansion  
>> leading to far
>> too many elements being checked.  Proposed solution appears to be a  
>> hack
>> where you index date fields as strings and hacking together date  
>> functions
>> to generate proper queries/format results.
>>
>> Does this remain the recommended solution to this issue?
>>
>> Thanks
>>
>> ---
>> Alok K. Dhir
>> Symplicity Corporation
>> www.symplicity.com
>> (703) 351-0200 x 8080
>> adhir@symplicity.com
>>
>>


Re: date range query performance

Posted by Chris Harris <ry...@gmail.com>.
Do you need to search down to the minutes and seconds level? If searching by
date provides sufficient granularity, for instance, you can normalize all
the time-of-day portions of the timestamps to midnight while indexing. (So
index any event happening on Oct 01, 2008 as 2008-10-01T00:00:00Z.) That
would give Solr many fewer unique timestamp values to go through.

On Wed, Oct 29, 2008 at 1:30 PM, Alok Dhir <ad...@symplicity.com> wrote:

> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine.
>
> Fairly simple schema -- no large text fields, standard request handler.  4
> small facet fields.
>
> The index is an event log -- a primary search/retrieval requirement is date
> range queries.
>
> A simple query without a date range subquery is ridiculously fast - 2ms.
>  The same query with a date range takes up to 30s (30,000ms).
>
> Concrete example, this query just look 18s:
>
>        instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z TO
> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>
> The exact same query without the date range took 2ms.
>
> I saw a thread from Apr 2008 which explains the problem being due to too
> much precision on the DateField type, and the range expansion leading to far
> too many elements being checked.  Proposed solution appears to be a hack
> where you index date fields as strings and hacking together date functions
> to generate proper queries/format results.
>
> Does this remain the recommended solution to this issue?
>
> Thanks
>
> ---
> Alok K. Dhir
> Symplicity Corporation
> www.symplicity.com
> (703) 351-0200 x 8080
> adhir@symplicity.com
>
>

Re: date range query performance

Posted by Alok Dhir <ad...@symplicity.com>.
We have implemented the suggested reduction in granularity by dropping  
time altogether and simply disallowing time filtering.  This, in light  
of other search filters we have provided, should prove be sufficient  
for our user base.

We did keep the fine granularity field not for filtering, but for  
sorting.  We definitely need the log entries to be presented in  
chronological order, so the finer resolution date field is useful for  
that at least.

Thanks for the detailed response.

Alok

On Oct 31, 2008, at 2:16 PM, Chris Hostetter wrote:

> : Concrete example, this query just look 18s:
> :
> : 	instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z TO
> : 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"
>
> : I saw a thread from Apr 2008 which explains the problem being due  
> to too much
> : precision on the DateField type, and the range expansion leading  
> to far too
> : many elements being checked.  Proposed solution appears to be a  
> hack where you
> : index date fields as strings and hacking together date functions  
> to generate
> : proper queries/format results.
>
> forteh record, you don't need to index as a "StrField" to get this
> benefit, you can still index using DateField you just need to round  
> your
> dates to some less graunlar level .. if you always want to round  
> down, you
> don't even need to do the rounding yourself, just add "/SECOND"
> or "/MINUTE" or "/HOUR" to each of your dates before sending them to  
> solr.
> (SOLR-741 proposes adding a config option to DateField to let this  
> be done
> server side)
>
> your example query seems to be happy with hour resolution, but in  
> theory
> if sometimes you needed hour resolution when doing "big ranges" but  
> more
> precise resolution when doing "small ranges" you could even in  
> theory have
> a "course" date field that you round to an hour, and redundently index
> the same data in a "fine" date field with minute or second resolution.
>
>
> Also: if you frequently reuse the same ranges over and over (ie: you  
> have
> a form widget people pick from so on any given day there is only N
> discrete ranges being used) putting them in an "fq" param will let  
> them be
> cached uniquely from the main query string
> (instance:client\-csm.symplicity.com) so differnet searches using  
> the same
> date ranges will be faster.  ditto for your label_facet:"Added to
> Position" clause.
>
> -Hoss
>


Re: date range query performance

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Wed, Jan 7, 2009 at 7:47 AM, Jim Adams <ja...@gmail.com> wrote:

> Can someone explain what this means to me?
>
> I'm having a similar performance issue - it's an index with only 1 million
> records or so, but when trying to search on a date range it takes 30
> seconds!  Yes, this date is one with hours, minutes, seconds in them -- do
> I
> need to create an additional field without the time component and reindex
> all my documents so I can get decent search performance?  Or can I tell
> Solr
> "Please ignore the time and do something in a reasonable timeframe" (GRIN)
>
>
Range queries are slow if you have a large number of unique terms. With
dates it is especially a problem because the more precise they are, the more
number of terms you've got in that field.

The easy solution is to round off your dates to minimum precision acceptable
to your use-case. You'll need to re-index.

-- 
Regards,
Shalin Shekhar Mangar.

Re: date range query performance

Posted by Erick Erickson <er...@gmail.com>.
You'll have to search the archives for a more complete explanation, I'm
going from memory here.. (or perhaps it's on the Wiki, I don't remember).

The notion is to break apart your timestamp (if you really, really need the
precision) into several fields rather than one. I.e. index the YYYYMMDD
as one field, then perhaps HHSS a second field and, perhaps, milliseconds
as a third field. This *greatly* reduces the number of unique terms and
should
improve searching on ranges, not to mention sorting. You'll have to
manipulate the timestamp part of the query.

There are variations on the scheme, you could have 6 fields for instance,
YYYY,
MM, DD, HH, SS, MS for instance. Or....

But the very best solution is to do as Erik (no relation) H. suggests, just
reindex
with, say, day granularity if that's fine enough.

Best
Erick

On Wed, Jan 7, 2009 at 6:03 AM, Erik Hatcher <er...@ehatchersolutions.com>wrote:

>
> On Jan 6, 2009, at 9:17 PM, Jim Adams wrote:
>
>> Can someone explain what this means to me?
>>
>
> The below <field> definition sets the timestamp field without time
> granularity, just day.  It's the difference between, say you've indexed a
> document for every millisecond in a day (what is that, 86.4M?), and a single
> term for the single date.
>
>  I'm having a similar performance issue - it's an index with only 1 million
>> records or so, but when trying to search on a date range it takes 30
>> seconds!  Yes, this date is one with hours, minutes, seconds in them -- do
>> I
>> need to create an additional field without the time component and reindex
>> all my documents so I can get decent search performance?  Or can I tell
>> Solr
>> "Please ignore the time and do something in a reasonable timeframe" (GRIN)
>>
>
> Do you care about milliseconds, seconds, minutes, or hours in terms of
> searching?  If not, it's a very good idea to reduce the granularity and thus
> the number of unique terms.
>
>        Erik
>
>
>
>
>>
>> Thanks.
>>
>> On Fri, Oct 31, 2008 at 10:28 PM, Michael Lackhoff <michael@lackhoff.de
>> >wrote:
>>
>>  On 01.11.2008 06:10 Erik Hatcher wrote:
>>>
>>>  Yeah, this should work fine:
>>>>
>>>>   <field name="timestamp" type="date" indexed="true" stored="true"
>>>> default="NOW/DAY" multiValued="false"/>
>>>>
>>>
>>> Wow, that was fast, thanks!
>>>
>>> -Michael
>>>
>>>
>

Re: date range query performance

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jan 6, 2009, at 9:17 PM, Jim Adams wrote:
> Can someone explain what this means to me?

The below <field> definition sets the timestamp field without time  
granularity, just day.  It's the difference between, say you've  
indexed a document for every millisecond in a day (what is that,  
86.4M?), and a single term for the single date.

> I'm having a similar performance issue - it's an index with only 1  
> million
> records or so, but when trying to search on a date range it takes 30
> seconds!  Yes, this date is one with hours, minutes, seconds in them  
> -- do I
> need to create an additional field without the time component and  
> reindex
> all my documents so I can get decent search performance?  Or can I  
> tell Solr
> "Please ignore the time and do something in a reasonable  
> timeframe" (GRIN)

Do you care about milliseconds, seconds, minutes, or hours in terms of  
searching?  If not, it's a very good idea to reduce the granularity  
and thus the number of unique terms.

	Erik


>
>
> Thanks.
>
> On Fri, Oct 31, 2008 at 10:28 PM, Michael Lackhoff <michael@lackhoff.de 
> >wrote:
>
>> On 01.11.2008 06:10 Erik Hatcher wrote:
>>
>>> Yeah, this should work fine:
>>>
>>>    <field name="timestamp" type="date" indexed="true" stored="true"
>>> default="NOW/DAY" multiValued="false"/>
>>
>> Wow, that was fast, thanks!
>>
>> -Michael
>>


Re: date range query performance

Posted by Jim Adams <ja...@gmail.com>.
Can someone explain what this means to me?

I'm having a similar performance issue - it's an index with only 1 million
records or so, but when trying to search on a date range it takes 30
seconds!  Yes, this date is one with hours, minutes, seconds in them -- do I
need to create an additional field without the time component and reindex
all my documents so I can get decent search performance?  Or can I tell Solr
"Please ignore the time and do something in a reasonable timeframe" (GRIN)

Thanks.

On Fri, Oct 31, 2008 at 10:28 PM, Michael Lackhoff <mi...@lackhoff.de>wrote:

> On 01.11.2008 06:10 Erik Hatcher wrote:
>
> > Yeah, this should work fine:
> >
> >     <field name="timestamp" type="date" indexed="true" stored="true"
> > default="NOW/DAY" multiValued="false"/>
>
> Wow, that was fast, thanks!
>
> -Michael
>

Re: date range query performance

Posted by Michael Lackhoff <mi...@lackhoff.de>.
On 01.11.2008 06:10 Erik Hatcher wrote:

> Yeah, this should work fine:
> 
>     <field name="timestamp" type="date" indexed="true" stored="true"  
> default="NOW/DAY" multiValued="false"/>

Wow, that was fast, thanks!

-Michael

Re: date range query performance

Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Nov 1, 2008, at 1:07 AM, Michael Lackhoff wrote:

> On 31.10.2008 19:16 Chris Hostetter wrote:
>
>> forteh record, you don't need to index as a "StrField" to get this
>> benefit, you can still index using DateField you just need to round  
>> your
>> dates to some less graunlar level .. if you always want to round  
>> down, you
>> don't even need to do the rounding yourself, just add "/SECOND"
>> or "/MINUTE" or "/HOUR" to each of your dates before sending them  
>> to solr.
>> (SOLR-741 proposes adding a config option to DateField to let this  
>> be done
>> server side)
>
> Is this also possible for the timestamp that is automatically added to
> all new/updated docs? I would like to be able to search (quickly) for
> everything that was added within the last week or month or whatever.  
> And
> because I update the index only once a day a granuality of /DAY (if  
> that
> exists) would be fine.

Yeah, this should work fine:

    <field name="timestamp" type="date" indexed="true" stored="true"  
default="NOW/DAY" multiValued="false"/>

	Erik


Re: date range query performance

Posted by Michael Lackhoff <mi...@lackhoff.de>.
On 31.10.2008 19:16 Chris Hostetter wrote:

> forteh record, you don't need to index as a "StrField" to get this 
> benefit, you can still index using DateField you just need to round your 
> dates to some less graunlar level .. if you always want to round down, you 
> don't even need to do the rounding yourself, just add "/SECOND" 
> or "/MINUTE" or "/HOUR" to each of your dates before sending them to solr.  
> (SOLR-741 proposes adding a config option to DateField to let this be done 
> server side)

Is this also possible for the timestamp that is automatically added to
all new/updated docs? I would like to be able to search (quickly) for
everything that was added within the last week or month or whatever. And
because I update the index only once a day a granuality of /DAY (if that
exists) would be fine.

- Michael

Re: date range query performance

Posted by Chris Hostetter <ho...@fucit.org>.
: Concrete example, this query just look 18s:
: 
: 	instance:client\-csm.symplicity.com AND dt:[2008-10-01T04:00:00Z TO
: 2008-10-30T03:59:59Z] AND label_facet:"Added to Position"

: I saw a thread from Apr 2008 which explains the problem being due to too much
: precision on the DateField type, and the range expansion leading to far too
: many elements being checked.  Proposed solution appears to be a hack where you
: index date fields as strings and hacking together date functions to generate
: proper queries/format results.

forteh record, you don't need to index as a "StrField" to get this 
benefit, you can still index using DateField you just need to round your 
dates to some less graunlar level .. if you always want to round down, you 
don't even need to do the rounding yourself, just add "/SECOND" 
or "/MINUTE" or "/HOUR" to each of your dates before sending them to solr.  
(SOLR-741 proposes adding a config option to DateField to let this be done 
server side)

your example query seems to be happy with hour resolution, but in theory 
if sometimes you needed hour resolution when doing "big ranges" but more 
precise resolution when doing "small ranges" you could even in theory have 
a "course" date field that you round to an hour, and redundently index 
the same data in a "fine" date field with minute or second resolution.


Also: if you frequently reuse the same ranges over and over (ie: you have 
a form widget people pick from so on any given day there is only N 
discrete ranges being used) putting them in an "fq" param will let them be 
cached uniquely from the main query string 
(instance:client\-csm.symplicity.com) so differnet searches using the same 
date ranges will be faster.  ditto for your label_facet:"Added to 
Position" clause.

-Hoss