You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by adeelmahmood <ad...@gmail.com> on 2010/02/14 23:56:43 UTC
too often delta imports performance effect
we are trying to setup solr for a website where data gets updated pretty
frequently and I want to have those changes reflected in solr indexes sooner
than nighly delta-imports .. so I am thinking we will probably want to set
it up to have delta imports running every 15 mins or so .. and solr search
will obviously be in use while this is going on .. first of all does solr
works well with adding new data or updating existing data while people are
doing searches in it
secondly are these delta imports are gonna cause any significant performance
degradation in solr search
any help is appreciated
--
View this message in context: http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27587778.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: too often delta imports performance effect
Posted by Nick Jenkin <nj...@gmail.com>.
Yes, the old data will show until there has been a commit executed. 50
docs isn't many so you should be fine
-Nick
On Mon, Feb 15, 2010 at 11:41 AM, adeelmahmood <ad...@gmail.com> wrote:
>
> thank you .. that helps .. actually its not that many updates .. close to 10
> fields probably and may be 50 doc updates per 15 .. so i am assuming that by
> handling indexing and searching in parallel you mean that if its updating
> some data .. it will continue to show old data until new data has been
> finalized(committed) or something like that ??
>
>
> Jan Høydahl / Cominvent wrote:
>>
>> Hi,
>>
>> This all depends on actual volumes, HW, architecture etc.
>> What exactly is "pretty frequently", how many document updates/adds per 15
>> minutes?
>>
>> Solr is designed to be able to do indexing and search in parallel, so you
>> don't need to fear this, unless you are already pushing the limits of what
>> your setup can handle. The best way to go is to start out and then
>> optimize when you see bottlenecks.
>>
>> Here is a pointer to Wiki about indexing performance:
>> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
>>
>> --
>> Jan Høydahl - search architect
>> Cominvent AS - www.cominvent.com
>>
>> On 14. feb. 2010, at 23.56, adeelmahmood wrote:
>>
>>>
>>> we are trying to setup solr for a website where data gets updated pretty
>>> frequently and I want to have those changes reflected in solr indexes
>>> sooner
>>> than nighly delta-imports .. so I am thinking we will probably want to
>>> set
>>> it up to have delta imports running every 15 mins or so .. and solr
>>> search
>>> will obviously be in use while this is going on .. first of all does solr
>>> works well with adding new data or updating existing data while people
>>> are
>>> doing searches in it
>>> secondly are these delta imports are gonna cause any significant
>>> performance
>>> degradation in solr search
>>> any help is appreciated
>>> --
>>> View this message in context:
>>> http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27587778.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27588472.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: too often delta imports performance effect
Posted by adeelmahmood <ad...@gmail.com>.
thank you .. that helps .. actually its not that many updates .. close to 10
fields probably and may be 50 doc updates per 15 .. so i am assuming that by
handling indexing and searching in parallel you mean that if its updating
some data .. it will continue to show old data until new data has been
finalized(committed) or something like that ??
Jan Høydahl / Cominvent wrote:
>
> Hi,
>
> This all depends on actual volumes, HW, architecture etc.
> What exactly is "pretty frequently", how many document updates/adds per 15
> minutes?
>
> Solr is designed to be able to do indexing and search in parallel, so you
> don't need to fear this, unless you are already pushing the limits of what
> your setup can handle. The best way to go is to start out and then
> optimize when you see bottlenecks.
>
> Here is a pointer to Wiki about indexing performance:
> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
>
> --
> Jan Høydahl - search architect
> Cominvent AS - www.cominvent.com
>
> On 14. feb. 2010, at 23.56, adeelmahmood wrote:
>
>>
>> we are trying to setup solr for a website where data gets updated pretty
>> frequently and I want to have those changes reflected in solr indexes
>> sooner
>> than nighly delta-imports .. so I am thinking we will probably want to
>> set
>> it up to have delta imports running every 15 mins or so .. and solr
>> search
>> will obviously be in use while this is going on .. first of all does solr
>> works well with adding new data or updating existing data while people
>> are
>> doing searches in it
>> secondly are these delta imports are gonna cause any significant
>> performance
>> degradation in solr search
>> any help is appreciated
>> --
>> View this message in context:
>> http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27587778.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27588472.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: too often delta imports performance effect
Posted by Jan Høydahl / Cominvent <ja...@cominvent.com>.
Hi,
This all depends on actual volumes, HW, architecture etc.
What exactly is "pretty frequently", how many document updates/adds per 15 minutes?
Solr is designed to be able to do indexing and search in parallel, so you don't need to fear this, unless you are already pushing the limits of what your setup can handle. The best way to go is to start out and then optimize when you see bottlenecks.
Here is a pointer to Wiki about indexing performance:
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
--
Jan Høydahl - search architect
Cominvent AS - www.cominvent.com
On 14. feb. 2010, at 23.56, adeelmahmood wrote:
>
> we are trying to setup solr for a website where data gets updated pretty
> frequently and I want to have those changes reflected in solr indexes sooner
> than nighly delta-imports .. so I am thinking we will probably want to set
> it up to have delta imports running every 15 mins or so .. and solr search
> will obviously be in use while this is going on .. first of all does solr
> works well with adding new data or updating existing data while people are
> doing searches in it
> secondly are these delta imports are gonna cause any significant performance
> degradation in solr search
> any help is appreciated
> --
> View this message in context: http://old.nabble.com/too-often-delta-imports-performance-effect-tp27587778p27587778.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>