You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Arunkumar Ayyavu <ar...@gmail.com> on 2010/10/03 20:55:22 UTC
Multiple masters and replication between masters?
I'm looking at setting up multiple masters for redundancy (for index
updates). I found the thread in this link
(http://www.lucidimagination.com/search/document/68ac303ce8425506/multiple_masters_solr_replication_1_4)
discussed this subject more than a year back. Does Solr support such
configuration today?
And does Solr support replication between masters? Otherwise, I'll
have to post the updates to all masters to keep the indexes of masters
in sync. Does SolrCloud address this case? (Please note it is too
early for me to read about SolrCloud as I'm still learning Solr)
Thanks in advance.
--
Arun
Re: Multiple masters and replication between masters?
Posted by Arunkumar Ayyavu <ar...@gmail.com>.
Thanks Otis. That was helpful.
On Mon, Oct 11, 2010 at 9:19 AM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> Arun,
>
> Yes, changing the solrconfig.xml to point to the new master could require a
> restart.
> However, if you use logical addresses (VIPs in the Load Balancer or even local
> hostname aliases if you don't have a LB) then you just need to point those
> VIPs/aliases to new IPs and the Solr slave won't have to be restarted.
>
>
> Otis
> ----Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
>> From: Arunkumar Ayyavu <ar...@gmail.com>
>> To: solr-user@lucene.apache.org
>> Sent: Sun, October 10, 2010 1:57:34 PM
>> Subject: Re: Multiple masters and replication between masters?
>>
>> On Mon, Oct 4, 2010 at 4:58 PM, Upayavira <uv...@odoko.co.uk> wrote:
>> > On Mon, 2010-10-04 at 00:25 +0530, Arunkumar Ayyavu wrote:
>> >> I'm looking at setting up multiple masters for redundancy (for index
>> >> updates). I found the thread in this link
>> >>
>>(http://www.lucidimagination.com/search/document/68ac303ce8425506/multiple_masters_solr_replication_1_4)
>>
>> >> discussed this subject more than a year back. Does Solr support such
>> >> configuration today?
>> >
>> > Solr does not support master/master replication. When you commit
>> > documents to SOLR, it adds a segment to the underlying Lucene index.
>> > Replication then syncs that segment to your slaves. To do master/master
>> > replication, you would have to pull changes from each master, then merge
>> > those changed segments into a single updated index. This is more complex
>> > than what is happening in the current Solr replication (which is not
>> > much more than an rsync of the index files).
>> >
>> > Note, if you commit your changes to two masters, you cannot switch a
>> > slave between them, as it is unlikely that the two masters will have
>> > matching index files. If you did so, you would probably trigger a pull
>> > of the entire index across the network, which (while it would likely
>> > work) would not be the most efficient action.
>> >
>> > What you can do is think cleverly about how you organise your
>> > master/slave setup. E.g. have a slave that doesn't get queried, but
>> > exists to take over the role of the master in case it fails. The index
>> > on a slave is the same as that in a master, and can immediately take on
>> > the role of the master (receiving commits), and upon failure of your
>> > master, you could point your other slaves at this new master, and things
>> > should just carry on as before.
>> Wouldn't this require restart of Solr instances?
>>
>> Sorry, I couldn't respond to you earlier as I wasn't checking my mails
>> for sometime.
>>
>> >
>> > Also, if you have a lot of slaves (such that they are placing too big a
>> > load on your master), insert intermediate hosts that are both slaves off
>> > the master, and masters to your query slaves. That way, you could have,
>> > say, two boxes slaving off the master, then 20 or 30 slaving off them.
>> >
>> >> And does Solr support replication between masters? Otherwise, I'll
>> >> have to post the updates to all masters to keep the indexes of masters
>> >> in sync. Does SolrCloud address this case? (Please note it is too
>> >> early for me to read about SolrCloud as I'm still learning Solr)
>> >
>> > I don't believe SolrCloud is aiming to support master/master
>> > replication.
>> >
>> > HTH
>> >
>> > Upayavira
>> >
>> >
>> >
>>
>>
>>
>> --
>> Arun
>>
>
--
Arun
Re: Multiple masters and replication between masters?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Arun,
Yes, changing the solrconfig.xml to point to the new master could require a
restart.
However, if you use logical addresses (VIPs in the Load Balancer or even local
hostname aliases if you don't have a LB) then you just need to point those
VIPs/aliases to new IPs and the Solr slave won't have to be restarted.
Otis
----Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Arunkumar Ayyavu <ar...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Sun, October 10, 2010 1:57:34 PM
> Subject: Re: Multiple masters and replication between masters?
>
> On Mon, Oct 4, 2010 at 4:58 PM, Upayavira <uv...@odoko.co.uk> wrote:
> > On Mon, 2010-10-04 at 00:25 +0530, Arunkumar Ayyavu wrote:
> >> I'm looking at setting up multiple masters for redundancy (for index
> >> updates). I found the thread in this link
> >>
>(http://www.lucidimagination.com/search/document/68ac303ce8425506/multiple_masters_solr_replication_1_4)
>
> >> discussed this subject more than a year back. Does Solr support such
> >> configuration today?
> >
> > Solr does not support master/master replication. When you commit
> > documents to SOLR, it adds a segment to the underlying Lucene index.
> > Replication then syncs that segment to your slaves. To do master/master
> > replication, you would have to pull changes from each master, then merge
> > those changed segments into a single updated index. This is more complex
> > than what is happening in the current Solr replication (which is not
> > much more than an rsync of the index files).
> >
> > Note, if you commit your changes to two masters, you cannot switch a
> > slave between them, as it is unlikely that the two masters will have
> > matching index files. If you did so, you would probably trigger a pull
> > of the entire index across the network, which (while it would likely
> > work) would not be the most efficient action.
> >
> > What you can do is think cleverly about how you organise your
> > master/slave setup. E.g. have a slave that doesn't get queried, but
> > exists to take over the role of the master in case it fails. The index
> > on a slave is the same as that in a master, and can immediately take on
> > the role of the master (receiving commits), and upon failure of your
> > master, you could point your other slaves at this new master, and things
> > should just carry on as before.
> Wouldn't this require restart of Solr instances?
>
> Sorry, I couldn't respond to you earlier as I wasn't checking my mails
> for sometime.
>
> >
> > Also, if you have a lot of slaves (such that they are placing too big a
> > load on your master), insert intermediate hosts that are both slaves off
> > the master, and masters to your query slaves. That way, you could have,
> > say, two boxes slaving off the master, then 20 or 30 slaving off them.
> >
> >> And does Solr support replication between masters? Otherwise, I'll
> >> have to post the updates to all masters to keep the indexes of masters
> >> in sync. Does SolrCloud address this case? (Please note it is too
> >> early for me to read about SolrCloud as I'm still learning Solr)
> >
> > I don't believe SolrCloud is aiming to support master/master
> > replication.
> >
> > HTH
> >
> > Upayavira
> >
> >
> >
>
>
>
> --
> Arun
>
Re: Multiple masters and replication between masters?
Posted by Arunkumar Ayyavu <ar...@gmail.com>.
On Mon, Oct 4, 2010 at 4:58 PM, Upayavira <uv...@odoko.co.uk> wrote:
> On Mon, 2010-10-04 at 00:25 +0530, Arunkumar Ayyavu wrote:
>> I'm looking at setting up multiple masters for redundancy (for index
>> updates). I found the thread in this link
>> (http://www.lucidimagination.com/search/document/68ac303ce8425506/multiple_masters_solr_replication_1_4)
>> discussed this subject more than a year back. Does Solr support such
>> configuration today?
>
> Solr does not support master/master replication. When you commit
> documents to SOLR, it adds a segment to the underlying Lucene index.
> Replication then syncs that segment to your slaves. To do master/master
> replication, you would have to pull changes from each master, then merge
> those changed segments into a single updated index. This is more complex
> than what is happening in the current Solr replication (which is not
> much more than an rsync of the index files).
>
> Note, if you commit your changes to two masters, you cannot switch a
> slave between them, as it is unlikely that the two masters will have
> matching index files. If you did so, you would probably trigger a pull
> of the entire index across the network, which (while it would likely
> work) would not be the most efficient action.
>
> What you can do is think cleverly about how you organise your
> master/slave setup. E.g. have a slave that doesn't get queried, but
> exists to take over the role of the master in case it fails. The index
> on a slave is the same as that in a master, and can immediately take on
> the role of the master (receiving commits), and upon failure of your
> master, you could point your other slaves at this new master, and things
> should just carry on as before.
Wouldn't this require restart of Solr instances?
Sorry, I couldn't respond to you earlier as I wasn't checking my mails
for sometime.
>
> Also, if you have a lot of slaves (such that they are placing too big a
> load on your master), insert intermediate hosts that are both slaves off
> the master, and masters to your query slaves. That way, you could have,
> say, two boxes slaving off the master, then 20 or 30 slaving off them.
>
>> And does Solr support replication between masters? Otherwise, I'll
>> have to post the updates to all masters to keep the indexes of masters
>> in sync. Does SolrCloud address this case? (Please note it is too
>> early for me to read about SolrCloud as I'm still learning Solr)
>
> I don't believe SolrCloud is aiming to support master/master
> replication.
>
> HTH
>
> Upayavira
>
>
>
--
Arun
Re: Multiple masters and replication between masters?
Posted by Upayavira <uv...@odoko.co.uk>.
On Mon, 2010-10-04 at 00:25 +0530, Arunkumar Ayyavu wrote:
> I'm looking at setting up multiple masters for redundancy (for index
> updates). I found the thread in this link
> (http://www.lucidimagination.com/search/document/68ac303ce8425506/multiple_masters_solr_replication_1_4)
> discussed this subject more than a year back. Does Solr support such
> configuration today?
Solr does not support master/master replication. When you commit
documents to SOLR, it adds a segment to the underlying Lucene index.
Replication then syncs that segment to your slaves. To do master/master
replication, you would have to pull changes from each master, then merge
those changed segments into a single updated index. This is more complex
than what is happening in the current Solr replication (which is not
much more than an rsync of the index files).
Note, if you commit your changes to two masters, you cannot switch a
slave between them, as it is unlikely that the two masters will have
matching index files. If you did so, you would probably trigger a pull
of the entire index across the network, which (while it would likely
work) would not be the most efficient action.
What you can do is think cleverly about how you organise your
master/slave setup. E.g. have a slave that doesn't get queried, but
exists to take over the role of the master in case it fails. The index
on a slave is the same as that in a master, and can immediately take on
the role of the master (receiving commits), and upon failure of your
master, you could point your other slaves at this new master, and things
should just carry on as before.
Also, if you have a lot of slaves (such that they are placing too big a
load on your master), insert intermediate hosts that are both slaves off
the master, and masters to your query slaves. That way, you could have,
say, two boxes slaving off the master, then 20 or 30 slaving off them.
> And does Solr support replication between masters? Otherwise, I'll
> have to post the updates to all masters to keep the indexes of masters
> in sync. Does SolrCloud address this case? (Please note it is too
> early for me to read about SolrCloud as I'm still learning Solr)
I don't believe SolrCloud is aiming to support master/master
replication.
HTH
Upayavira
Re: Multiple masters and replication between masters?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
Would this help you:
http://wiki.apache.org/solr/SolrReplication#Setting_up_a_Repeater
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Arunkumar Ayyavu <ar...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Sun, October 3, 2010 2:55:22 PM
> Subject: Multiple masters and replication between masters?
>
> I'm looking at setting up multiple masters for redundancy (for index
> updates). I found the thread in this link
>(http://www.lucidimagination.com/search/document/68ac303ce8425506/multiple_masters_solr_replication_1_4)
>)
> discussed this subject more than a year back. Does Solr support such
> configuration today?
>
> And does Solr support replication between masters? Otherwise, I'll
> have to post the updates to all masters to keep the indexes of masters
> in sync. Does SolrCloud address this case? (Please note it is too
> early for me to read about SolrCloud as I'm still learning Solr)
>
> Thanks in advance.
>
> --
> Arun
>