You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Isan Fulia <is...@germinait.com> on 2011/02/06 16:25:54 UTC
Separating Index Reader and Writer
Hi all,
I have setup two indexes one for reading(R) and other for writing(W).Index R
refers to the same data dir of W (defined in solrconfig via <dataDir>).
To make sure the R index sees the indexed documents of W , i am firing an
empty commit on R.
With this , I am getting performance improvement as compared to using the
same index for reading and writing .
Can anyone help me in knowing why this performance improvement is taking
place even though both the indexeses are pointing to the same data
directory.
--
Thanks & Regards,
Isan Fulia.
Re: Separating Index Reader and Writer
Posted by Isan Fulia <is...@germinait.com>.
Hi peter ,
Can you elaborate a little on how performance gain is in cache warming.I am
getting a good improvement on search time.
On 6 February 2011 23:29, Peter Sturge <pe...@gmail.com> wrote:
> Hi,
>
> We use this scenario in production where we have one write-only Solr
> instance and 1 read-only, pointing to the same data.
> We do this so we can optimize caching/etc. for each instance for
> write/read. The main performance gain is in cache warming and
> associated parameters.
> For your Index W, it's worth turning off cache warming altogether, so
> commits aren't slowed down by warming.
>
> Peter
>
>
> On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia <is...@germinait.com>
> wrote:
> > Hi all,
> > I have setup two indexes one for reading(R) and other for
> writing(W).Index R
> > refers to the same data dir of W (defined in solrconfig via <dataDir>).
> > To make sure the R index sees the indexed documents of W , i am firing an
> > empty commit on R.
> > With this , I am getting performance improvement as compared to using the
> > same index for reading and writing .
> > Can anyone help me in knowing why this performance improvement is taking
> > place even though both the indexeses are pointing to the same data
> > directory.
> >
> > --
> > Thanks & Regards,
> > Isan Fulia.
> >
>
--
Thanks & Regards,
Isan Fulia.
Re: Separating Index Reader and Writer
Posted by Em <ma...@yahoo.de>.
Push again.
Regards
Em wrote:
>
> Just wanted to push that topic.
>
> Regards
>
>
> Em wrote:
>>
>> Hi Peter,
>>
>> I must jump in this discussion: From a logical point of view what you are
>> saying makes only sense if both instances do not run on the same machine
>> or at least not on the same drive.
>>
>> When both run on the same machine and the same drive, the overall used
>> memory should be equal plus I do not understand why this setup should
>> affect cache warming etc., since the process of rewarming should be the
>> same.
>>
>> Well, my knowledge about the internals is not very deep. But from just a
>> logical point of view - to me - the same is happening as if I would do it
>> in a single solr-instance. So what is the difference, what do I overlook?
>>
>> Another thing: While W is committing and writing to the index, is there
>> any inconsistency in R or isn't there any, because W is writing a new
>> Segment and so for R there isn't anything different until the commit
>> finished?
>> Are there problems during optimizing an index?
>>
>> How do you inform R about the finished commit?
>>
>> Thank you for your explanation, it's a really interesting topic!
>>
>> Regards,
>> Em
>>
>> Peter Sturge-2 wrote:
>>>
>>> Hi,
>>>
>>> We use this scenario in production where we have one write-only Solr
>>> instance and 1 read-only, pointing to the same data.
>>> We do this so we can optimize caching/etc. for each instance for
>>> write/read. The main performance gain is in cache warming and
>>> associated parameters.
>>> For your Index W, it's worth turning off cache warming altogether, so
>>> commits aren't slowed down by warming.
>>>
>>> Peter
>>>
>>>
>>> On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia <is...@germinait.com>
>>> wrote:
>>>> Hi all,
>>>> I have setup two indexes one for reading(R) and other for
>>>> writing(W).Index R
>>>> refers to the same data dir of W (defined in solrconfig via <dataDir>).
>>>> To make sure the R index sees the indexed documents of W , i am firing
>>>> an
>>>> empty commit on R.
>>>> With this , I am getting performance improvement as compared to using
>>>> the
>>>> same index for reading and writing .
>>>> Can anyone help me in knowing why this performance improvement is
>>>> taking
>>>> place even though both the indexeses are pointing to the same data
>>>> directory.
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Isan Fulia.
>>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Separating-Index-Reader-and-Writer-tp2437666p2516736.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Separating Index Reader and Writer
Posted by Em <ma...@yahoo.de>.
Just wanted to push that topic.
Regards
Em wrote:
>
> Hi Peter,
>
> I must jump in this discussion: From a logical point of view what you are
> saying makes only sense if both instances do not run on the same machine
> or at least not on the same drive.
>
> When both run on the same machine and the same drive, the overall used
> memory should be equal plus I do not understand why this setup should
> affect cache warming etc., since the process of rewarming should be the
> same.
>
> Well, my knowledge about the internals is not very deep. But from just a
> logical point of view - to me - the same is happening as if I would do it
> in a single solr-instance. So what is the difference, what do I overlook?
>
> Another thing: While W is committing and writing to the index, is there
> any inconsistency in R or isn't there any, because W is writing a new
> Segment and so for R there isn't anything different until the commit
> finished?
> Are there problems during optimizing an index?
>
> How do you inform R about the finished commit?
>
> Thank you for your explanation, it's a really interesting topic!
>
> Regards,
> Em
>
> Peter Sturge-2 wrote:
>>
>> Hi,
>>
>> We use this scenario in production where we have one write-only Solr
>> instance and 1 read-only, pointing to the same data.
>> We do this so we can optimize caching/etc. for each instance for
>> write/read. The main performance gain is in cache warming and
>> associated parameters.
>> For your Index W, it's worth turning off cache warming altogether, so
>> commits aren't slowed down by warming.
>>
>> Peter
>>
>>
>> On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia <is...@germinait.com>
>> wrote:
>>> Hi all,
>>> I have setup two indexes one for reading(R) and other for
>>> writing(W).Index R
>>> refers to the same data dir of W (defined in solrconfig via <dataDir>).
>>> To make sure the R index sees the indexed documents of W , i am firing
>>> an
>>> empty commit on R.
>>> With this , I am getting performance improvement as compared to using
>>> the
>>> same index for reading and writing .
>>> Can anyone help me in knowing why this performance improvement is taking
>>> place even though both the indexeses are pointing to the same data
>>> directory.
>>>
>>> --
>>> Thanks & Regards,
>>> Isan Fulia.
>>>
>>
>>
>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Separating-Index-Reader-and-Writer-tp2437666p2452238.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Separating Index Reader and Writer
Posted by Em <ma...@yahoo.de>.
Hi Peter,
I must jump in this discussion: From a logical point of view what you are
saying makes only sense if both instances do not run on the same machine or
at least not on the same drive.
When both run on the same machine and the same drive, the overall used
memory should be equal plus I do not understand why this setup should affect
cache warming etc., since the process of rewarming should be the same.
Well, my knowledge about the internals is not very deep. But from just a
logical point of view - to me - the same is happening as if I would do it in
a single solr-instance. So what is the difference, what do I overlook?
Another thing: While W is committing and writing to the index, is there any
inconsistency in R or isn't there any, because W is writing a new Segment
and so for R there isn't anything different until the commit finished?
Are there problems during optimizing an index?
How do you inform R about the finished commit?
Thank you for your explanation, it's a really interesting topic!
Regards,
Em
Peter Sturge-2 wrote:
>
> Hi,
>
> We use this scenario in production where we have one write-only Solr
> instance and 1 read-only, pointing to the same data.
> We do this so we can optimize caching/etc. for each instance for
> write/read. The main performance gain is in cache warming and
> associated parameters.
> For your Index W, it's worth turning off cache warming altogether, so
> commits aren't slowed down by warming.
>
> Peter
>
>
> On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia <is...@germinait.com>
> wrote:
>> Hi all,
>> I have setup two indexes one for reading(R) and other for
>> writing(W).Index R
>> refers to the same data dir of W (defined in solrconfig via <dataDir>).
>> To make sure the R index sees the indexed documents of W , i am firing an
>> empty commit on R.
>> With this , I am getting performance improvement as compared to using the
>> same index for reading and writing .
>> Can anyone help me in knowing why this performance improvement is taking
>> place even though both the indexeses are pointing to the same data
>> directory.
>>
>> --
>> Thanks & Regards,
>> Isan Fulia.
>>
>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Separating-Index-Reader-and-Writer-tp2437666p2438730.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Separating Index Reader and Writer
Posted by Peter Sturge <pe...@gmail.com>.
Hi,
We use this scenario in production where we have one write-only Solr
instance and 1 read-only, pointing to the same data.
We do this so we can optimize caching/etc. for each instance for
write/read. The main performance gain is in cache warming and
associated parameters.
For your Index W, it's worth turning off cache warming altogether, so
commits aren't slowed down by warming.
Peter
On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia <is...@germinait.com> wrote:
> Hi all,
> I have setup two indexes one for reading(R) and other for writing(W).Index R
> refers to the same data dir of W (defined in solrconfig via <dataDir>).
> To make sure the R index sees the indexed documents of W , i am firing an
> empty commit on R.
> With this , I am getting performance improvement as compared to using the
> same index for reading and writing .
> Can anyone help me in knowing why this performance improvement is taking
> place even though both the indexeses are pointing to the same data
> directory.
>
> --
> Thanks & Regards,
> Isan Fulia.
>