You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Brandon Ramirez <Br...@elementk.com> on 2011/10/07 15:53:18 UTC
Replication with an HA master
We are getting ready to start a project using Solr as our backend search engine and I am trying to devise a deployment architecture that works for us. We definitely need a master/slave replication strategy, that's for sure, but my concern is the master becomes a single point of failure.
Fortunately, real-time search is not a requirement for us. If search results are a few minutes out of sync with our database, it's not a big deal.
So what I would like to do is have a set of query servers (slaves) that are only used for querying, no indexing and have them use Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA indexing, I'd like to have 2 masters: a primary and a standby. All indexing requests go to the primary unless it's taken out of service. To keep the standby ready to takeover if it needs to, it needs to be more up to date than the slaves. I'd like to have it replicate every 30 seconds or so.
The reason I'm asking about it on this list is that I haven't seen any Solr documentation or even anything that talks about this. I can't be the only one concerned about having a single point of failure, so I'm reaching out to see what others have done in this case before I go with my own solution.
Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
Software Engineer II | Element K | www.elementk.com<http://www.elementk.com/>
Re: Replication with an HA master
Posted by pravesh <su...@yahoo.com>.
Have you looked at Zookeeper, a sub-project of apache Hadoop?
Thanx
Pravesh
--
View this message in context: http://lucene.472066.n3.nabble.com/Replication-with-an-HA-master-tp3402852p3409069.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Replication with an HA master
Posted by Brandon Ramirez <Br...@elementk.com>.
I have the luxury of JMS in my environment, so that may be a simple way to solve this...
Sent from my iPhone
On Oct 13, 2011, at 4:02 PM, "Robert Stewart" <bs...@gmail.com> wrote:
> Yes that is a good point. Thanks.
>
> I think I will avoid using NAS/SAN and use two masters, one setup as a repeater (slave and master). In case of very rare master failure, some minor manual intervention will be required to re-configure remaining master or bring other one back up.
>
> My only concern in that case is losing new documents from solrj client since there is no broker/buffer/queue between solrj client and SOLR master. It would be nice if there was some open source broker/queue which could sit between solrj and SOLR and queue up messages (publish/subscribe).
>
> Bob
>
> On Oct 13, 2011, at 3:56 PM, Jaeger, Jay - DOT wrote:
>
>> One thing to consider is the case where the JVM is up, but the system is otherwise unavailable (say, a NIC failure, firewall failure, load balancer failure) - especially if you use a SAN (whose connection is different from the normal network).
>>
>> In such a case the old master might have uncommitted updates.
>>
>> JRJ
>>
>> -----Original Message-----
>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>> Sent: Tuesday, October 11, 2011 3:17 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Replication with an HA master
>>
>> Hello,
>> ----- Original Message -----
>>
>>> From: Robert Stewart <bs...@gmail.com>
>>> To: solr-user@lucene.apache.org
>>> Cc:
>>> Sent: Tuesday, October 11, 2011 3:37 PM
>>> Subject: Re: Replication with an HA master
>>>
>>> In the case of using a shared (SAN) index between 2 masters, what happens if the
>>> live master fails in such a way that the index remains "locked" (such
>>> as if some hardware failure and it did not unlock/close index). Will the other
>>> master be able to open/write to the index as new documents are added?
>>
>>
>> You'd use native locks, which should disappear if the JVM dies. If it does not, then I'm not 100% sure what happens, but in the worst case there would be a need for a quick manual (or scripted) intervention. But your index would be up to date!
>>
>>> Also, if that can work ok, would it work if you have a LB (VIP) from both
>>> indexing and replication sides of the 2 masters, such that some VIP used by
>>> solrj for indexing new documents via HTTP, and the same VIP used by slave
>>> searchers for replication? That sounds like it would work.
>>
>>
>> Precisely what you should do. e.g. "master-vip" is the "hostname" that both SolrJ would post new docs to and the master "server" slaves would poll for index changes.
>>
>> Otis
>> ----
>>
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> Lucene ecosystem search :: http://search-lucene.com/
>>
>>
>>
>>
>>> On Oct 11, 2011, at 3:16 PM, Otis Gospodnetic wrote:
>>>
>>>> Hello,
>>>>
>>>> Yes, you've read about NFS, which is why I gave the example of a SAN
>>> (which can have multiple power supplies, controllers, etc.)
>>>>
>>>> Yes, should be OK to have multiple Solr instances have the same index open,
>>> since only one of them will actually be writing to it, thanks to LB.
>>>>
>>>> Otis
>>>> ----
>>>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>>>> Lucene ecosystem search :: http://search-lucene.com/
>>>>
>>>>
>>>>> ________________________________
>>>>> From: Brandon Ramirez <Br...@elementk.com>
>>>>> To: "solr-user@lucene.apache.org"
>>> <so...@lucene.apache.org>
>>>>> Sent: Tuesday, October 11, 2011 2:55 PM
>>>>> Subject: RE: Replication with an HA master
>>>>>
>>>>> Using a shared volume crossed my mind too, but I discarded the idea
>>> because of literature I have read about Lucene performing poorly against remote
>>> file systems. But then I suppose a SAN wouldn't be a remote file system in
>>> the same sense as an NFS-mounted NAS or similar.
>>>>>
>>>>> Should I be concerned about two solr instances on two machines having
>>> the same SAN-based index open, as long as only one of them is receiving
>>> requests? I would think in theory it would work, but I don't have any
>>> production-level experience with Solr yet, only textbook knowledge.
>>>>>
>>>>>
>>>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
>>>>> Software Engineer II | Element K | www.elementk.com
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>>>>> Sent: Tuesday, October 11, 2011 2:28 PM
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: Re: Replication with an HA master
>>>>>
>>>>> A few alternatives:
>>>>> * Have the master keep the index on a shared disk (e.g. SAN)
>>>>> * Use LB to easily switch to between masters, potentially even
>>> automatically if LB can detect the primary is down
>>>>>
>>>>> Otis
>>>>> ----
>>>>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene
>>> ecosystem search :: http://search-lucene.com/
>>>>>
>>>>>
>>>>>> ________________________________
>>>>>> From: Robert Stewart <bs...@gmail.com>
>>>>>> To: solr-user@lucene.apache.org
>>>>>> Sent: Friday, October 7, 2011 10:22 AM
>>>>>> Subject: Re: Replication with an HA master
>>>>>>
>>>>>> Your idea sounds like the correct path. Setup 2 masters, one
>>> running
>>>>>> in "slave" mode which pulls replicas from the live
>>> master. When/if live master goes down, you just reconfigure and restart the
>>> backup master to be the live master. You'd also need to then start data
>>> import on the backup master (enable/start cron job?), and redirect slave
>>> searchers to pull replicas from the new live master. All that could be done
>>> using scripts or something like puppet possibly.
>>>>>>
>>>>>> Another solution maybe is to run 2 "live" masters, which
>>> both index the same content from the same data source. If one goes down, then
>>> you just need to redirect slave searchers to the backup master for replication.
>>>>>>
>>>>>> I am also starting a similar project which needs some disaster
>>> recovery processes in place, so any other info would be useful to me as well.
>>>>>>
>>>>>> Bob
>>>>>>
>>>>>> On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
>>>>>>
>>>>>>> We are getting ready to start a project using Solr as our
>>> backend search engine and I am trying to devise a deployment architecture that
>>> works for us. We definitely need a master/slave replication strategy,
>>> that's for sure, but my concern is the master becomes a single point of
>>> failure.
>>>>>>>
>>>>>>> Fortunately, real-time search is not a requirement for us. If
>>> search results are a few minutes out of sync with our database, it's not a
>>> big deal.
>>>>>>>
>>>>>>> So what I would like to do is have a set of query servers
>>> (slaves) that are only used for querying, no indexing and have them use
>>> Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA
>>> indexing, I'd like to have 2 masters: a primary and a standby. All indexing
>>> requests go to the primary unless it's taken out of service. To keep the
>>> standby ready to takeover if it needs to, it needs to be more up to date than
>>> the slaves. I'd like to have it replicate every 30 seconds or so.
>>>>>>>
>>>>>>> The reason I'm asking about it on this list is that I
>>> haven't seen any Solr documentation or even anything that talks about this.
>>> I can't be the only one concerned about having a single point of failure, so
>>> I'm reaching out to see what others have done in this case before I go with
>>> my own solution.
>>>>>>>
>>>>>>>
>>>>>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
>>> Software
>>>>>>> Engineer II | Element K |
>>> www.elementk.com<http://www.elementk.com/>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>
>
Re: Replication with an HA master
Posted by Robert Stewart <bs...@gmail.com>.
Yes that is a good point. Thanks.
I think I will avoid using NAS/SAN and use two masters, one setup as a repeater (slave and master). In case of very rare master failure, some minor manual intervention will be required to re-configure remaining master or bring other one back up.
My only concern in that case is losing new documents from solrj client since there is no broker/buffer/queue between solrj client and SOLR master. It would be nice if there was some open source broker/queue which could sit between solrj and SOLR and queue up messages (publish/subscribe).
Bob
On Oct 13, 2011, at 3:56 PM, Jaeger, Jay - DOT wrote:
> One thing to consider is the case where the JVM is up, but the system is otherwise unavailable (say, a NIC failure, firewall failure, load balancer failure) - especially if you use a SAN (whose connection is different from the normal network).
>
> In such a case the old master might have uncommitted updates.
>
> JRJ
>
> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> Sent: Tuesday, October 11, 2011 3:17 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Replication with an HA master
>
> Hello,
> ----- Original Message -----
>
>> From: Robert Stewart <bs...@gmail.com>
>> To: solr-user@lucene.apache.org
>> Cc:
>> Sent: Tuesday, October 11, 2011 3:37 PM
>> Subject: Re: Replication with an HA master
>>
>> In the case of using a shared (SAN) index between 2 masters, what happens if the
>> live master fails in such a way that the index remains "locked" (such
>> as if some hardware failure and it did not unlock/close index). Will the other
>> master be able to open/write to the index as new documents are added?
>
>
> You'd use native locks, which should disappear if the JVM dies. If it does not, then I'm not 100% sure what happens, but in the worst case there would be a need for a quick manual (or scripted) intervention. But your index would be up to date!
>
>> Also, if that can work ok, would it work if you have a LB (VIP) from both
>> indexing and replication sides of the 2 masters, such that some VIP used by
>> solrj for indexing new documents via HTTP, and the same VIP used by slave
>> searchers for replication? That sounds like it would work.
>
>
> Precisely what you should do. e.g. "master-vip" is the "hostname" that both SolrJ would post new docs to and the master "server" slaves would poll for index changes.
>
> Otis
> ----
>
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
>
>> On Oct 11, 2011, at 3:16 PM, Otis Gospodnetic wrote:
>>
>>> Hello,
>>>
>>> Yes, you've read about NFS, which is why I gave the example of a SAN
>> (which can have multiple power supplies, controllers, etc.)
>>>
>>> Yes, should be OK to have multiple Solr instances have the same index open,
>> since only one of them will actually be writing to it, thanks to LB.
>>>
>>> Otis
>>> ----
>>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>>> Lucene ecosystem search :: http://search-lucene.com/
>>>
>>>
>>>> ________________________________
>>>> From: Brandon Ramirez <Br...@elementk.com>
>>>> To: "solr-user@lucene.apache.org"
>> <so...@lucene.apache.org>
>>>> Sent: Tuesday, October 11, 2011 2:55 PM
>>>> Subject: RE: Replication with an HA master
>>>>
>>>> Using a shared volume crossed my mind too, but I discarded the idea
>> because of literature I have read about Lucene performing poorly against remote
>> file systems. But then I suppose a SAN wouldn't be a remote file system in
>> the same sense as an NFS-mounted NAS or similar.
>>>>
>>>> Should I be concerned about two solr instances on two machines having
>> the same SAN-based index open, as long as only one of them is receiving
>> requests? I would think in theory it would work, but I don't have any
>> production-level experience with Solr yet, only textbook knowledge.
>>>>
>>>>
>>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
>>>> Software Engineer II | Element K | www.elementk.com
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>>>> Sent: Tuesday, October 11, 2011 2:28 PM
>>>> To: solr-user@lucene.apache.org
>>>> Subject: Re: Replication with an HA master
>>>>
>>>> A few alternatives:
>>>> * Have the master keep the index on a shared disk (e.g. SAN)
>>>> * Use LB to easily switch to between masters, potentially even
>> automatically if LB can detect the primary is down
>>>>
>>>> Otis
>>>> ----
>>>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene
>> ecosystem search :: http://search-lucene.com/
>>>>
>>>>
>>>>> ________________________________
>>>>> From: Robert Stewart <bs...@gmail.com>
>>>>> To: solr-user@lucene.apache.org
>>>>> Sent: Friday, October 7, 2011 10:22 AM
>>>>> Subject: Re: Replication with an HA master
>>>>>
>>>>> Your idea sounds like the correct path. Setup 2 masters, one
>> running
>>>>> in "slave" mode which pulls replicas from the live
>> master. When/if live master goes down, you just reconfigure and restart the
>> backup master to be the live master. You'd also need to then start data
>> import on the backup master (enable/start cron job?), and redirect slave
>> searchers to pull replicas from the new live master. All that could be done
>> using scripts or something like puppet possibly.
>>>>>
>>>>> Another solution maybe is to run 2 "live" masters, which
>> both index the same content from the same data source. If one goes down, then
>> you just need to redirect slave searchers to the backup master for replication.
>>>>>
>>>>> I am also starting a similar project which needs some disaster
>> recovery processes in place, so any other info would be useful to me as well.
>>>>>
>>>>> Bob
>>>>>
>>>>> On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
>>>>>
>>>>>> We are getting ready to start a project using Solr as our
>> backend search engine and I am trying to devise a deployment architecture that
>> works for us. We definitely need a master/slave replication strategy,
>> that's for sure, but my concern is the master becomes a single point of
>> failure.
>>>>>>
>>>>>> Fortunately, real-time search is not a requirement for us. If
>> search results are a few minutes out of sync with our database, it's not a
>> big deal.
>>>>>>
>>>>>> So what I would like to do is have a set of query servers
>> (slaves) that are only used for querying, no indexing and have them use
>> Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA
>> indexing, I'd like to have 2 masters: a primary and a standby. All indexing
>> requests go to the primary unless it's taken out of service. To keep the
>> standby ready to takeover if it needs to, it needs to be more up to date than
>> the slaves. I'd like to have it replicate every 30 seconds or so.
>>>>>>
>>>>>> The reason I'm asking about it on this list is that I
>> haven't seen any Solr documentation or even anything that talks about this.
>> I can't be the only one concerned about having a single point of failure, so
>> I'm reaching out to see what others have done in this case before I go with
>> my own solution.
>>>>>>
>>>>>>
>>>>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
>> Software
>>>>>> Engineer II | Element K |
>> www.elementk.com<http://www.elementk.com/>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>
RE: Replication with an HA master
Posted by "Jaeger, Jay - DOT" <Ja...@dot.wi.gov>.
One thing to consider is the case where the JVM is up, but the system is otherwise unavailable (say, a NIC failure, firewall failure, load balancer failure) - especially if you use a SAN (whose connection is different from the normal network).
In such a case the old master might have uncommitted updates.
JRJ
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Tuesday, October 11, 2011 3:17 PM
To: solr-user@lucene.apache.org
Subject: Re: Replication with an HA master
Hello,
----- Original Message -----
> From: Robert Stewart <bs...@gmail.com>
> To: solr-user@lucene.apache.org
> Cc:
> Sent: Tuesday, October 11, 2011 3:37 PM
> Subject: Re: Replication with an HA master
>
> In the case of using a shared (SAN) index between 2 masters, what happens if the
> live master fails in such a way that the index remains "locked" (such
> as if some hardware failure and it did not unlock/close index). Will the other
> master be able to open/write to the index as new documents are added?
You'd use native locks, which should disappear if the JVM dies. If it does not, then I'm not 100% sure what happens, but in the worst case there would be a need for a quick manual (or scripted) intervention. But your index would be up to date!
> Also, if that can work ok, would it work if you have a LB (VIP) from both
> indexing and replication sides of the 2 masters, such that some VIP used by
> solrj for indexing new documents via HTTP, and the same VIP used by slave
> searchers for replication? That sounds like it would work.
Precisely what you should do. e.g. "master-vip" is the "hostname" that both SolrJ would post new docs to and the master "server" slaves would poll for index changes.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
> On Oct 11, 2011, at 3:16 PM, Otis Gospodnetic wrote:
>
>> Hello,
>>
>> Yes, you've read about NFS, which is why I gave the example of a SAN
> (which can have multiple power supplies, controllers, etc.)
>>
>> Yes, should be OK to have multiple Solr instances have the same index open,
> since only one of them will actually be writing to it, thanks to LB.
>>
>> Otis
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> Lucene ecosystem search :: http://search-lucene.com/
>>
>>
>>> ________________________________
>>> From: Brandon Ramirez <Br...@elementk.com>
>>> To: "solr-user@lucene.apache.org"
> <so...@lucene.apache.org>
>>> Sent: Tuesday, October 11, 2011 2:55 PM
>>> Subject: RE: Replication with an HA master
>>>
>>> Using a shared volume crossed my mind too, but I discarded the idea
> because of literature I have read about Lucene performing poorly against remote
> file systems. But then I suppose a SAN wouldn't be a remote file system in
> the same sense as an NFS-mounted NAS or similar.
>>>
>>> Should I be concerned about two solr instances on two machines having
> the same SAN-based index open, as long as only one of them is receiving
> requests? I would think in theory it would work, but I don't have any
> production-level experience with Solr yet, only textbook knowledge.
>>>
>>>
>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
>>> Software Engineer II | Element K | www.elementk.com
>>>
>>>
>>> -----Original Message-----
>>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>>> Sent: Tuesday, October 11, 2011 2:28 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Replication with an HA master
>>>
>>> A few alternatives:
>>> * Have the master keep the index on a shared disk (e.g. SAN)
>>> * Use LB to easily switch to between masters, potentially even
> automatically if LB can detect the primary is down
>>>
>>> Otis
>>> ----
>>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene
> ecosystem search :: http://search-lucene.com/
>>>
>>>
>>>> ________________________________
>>>> From: Robert Stewart <bs...@gmail.com>
>>>> To: solr-user@lucene.apache.org
>>>> Sent: Friday, October 7, 2011 10:22 AM
>>>> Subject: Re: Replication with an HA master
>>>>
>>>> Your idea sounds like the correct path. Setup 2 masters, one
> running
>>>> in "slave" mode which pulls replicas from the live
> master. When/if live master goes down, you just reconfigure and restart the
> backup master to be the live master. You'd also need to then start data
> import on the backup master (enable/start cron job?), and redirect slave
> searchers to pull replicas from the new live master. All that could be done
> using scripts or something like puppet possibly.
>>>>
>>>> Another solution maybe is to run 2 "live" masters, which
> both index the same content from the same data source. If one goes down, then
> you just need to redirect slave searchers to the backup master for replication.
>>>>
>>>> I am also starting a similar project which needs some disaster
> recovery processes in place, so any other info would be useful to me as well.
>>>>
>>>> Bob
>>>>
>>>> On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
>>>>
>>>>> We are getting ready to start a project using Solr as our
> backend search engine and I am trying to devise a deployment architecture that
> works for us. We definitely need a master/slave replication strategy,
> that's for sure, but my concern is the master becomes a single point of
> failure.
>>>>>
>>>>> Fortunately, real-time search is not a requirement for us. If
> search results are a few minutes out of sync with our database, it's not a
> big deal.
>>>>>
>>>>> So what I would like to do is have a set of query servers
> (slaves) that are only used for querying, no indexing and have them use
> Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA
> indexing, I'd like to have 2 masters: a primary and a standby. All indexing
> requests go to the primary unless it's taken out of service. To keep the
> standby ready to takeover if it needs to, it needs to be more up to date than
> the slaves. I'd like to have it replicate every 30 seconds or so.
>>>>>
>>>>> The reason I'm asking about it on this list is that I
> haven't seen any Solr documentation or even anything that talks about this.
> I can't be the only one concerned about having a single point of failure, so
> I'm reaching out to see what others have done in this case before I go with
> my own solution.
>>>>>
>>>>>
>>>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
> Software
>>>>> Engineer II | Element K |
> www.elementk.com<http://www.elementk.com/>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>
Re: Replication with an HA master
Posted by Ted Dunning <te...@gmail.com>.
On Tue, Oct 11, 2011 at 8:17 PM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
> > In the case of using a shared (SAN) index between 2 masters, what happens
> if the
> > live master fails in such a way that the index remains "locked" (such
> > as if some hardware failure and it did not unlock/close index). Will the
> other
> > master be able to open/write to the index as new documents are added?
>
>
> You'd use native locks, which should disappear if the JVM dies. If it does
> not, then I'm not 100% sure what happens, but in the worst case there would
> be a need for a quick manual (or scripted) intervention. But your index
> would be up to date!
>
Or use ZK which was designed for this. The quality of implementation of
locks varies a lot and I would much prefer ZK.
> Also, if that can work ok, would it work if you have a LB (VIP) from both
> > indexing and replication sides of the 2 masters, such that some VIP used
> by
> > solrj for indexing new documents via HTTP, and the same VIP used by slave
> > searchers for replication? That sounds like it would work.
>
>
> Precisely what you should do. e.g. "master-vip" is the "hostname" that
> both SolrJ would post new docs to and the master "server" slaves would poll
> for index changes.
Most load balancer appliances can be scripted enough to keep an eye on ZK.
It would also be the work of a moment to write a program to watch a ZK
leader election for changes. That would allow you to have faster response
to failures if you like.
Re: Replication with an HA master
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello,
----- Original Message -----
> From: Robert Stewart <bs...@gmail.com>
> To: solr-user@lucene.apache.org
> Cc:
> Sent: Tuesday, October 11, 2011 3:37 PM
> Subject: Re: Replication with an HA master
>
> In the case of using a shared (SAN) index between 2 masters, what happens if the
> live master fails in such a way that the index remains "locked" (such
> as if some hardware failure and it did not unlock/close index). Will the other
> master be able to open/write to the index as new documents are added?
You'd use native locks, which should disappear if the JVM dies. If it does not, then I'm not 100% sure what happens, but in the worst case there would be a need for a quick manual (or scripted) intervention. But your index would be up to date!
> Also, if that can work ok, would it work if you have a LB (VIP) from both
> indexing and replication sides of the 2 masters, such that some VIP used by
> solrj for indexing new documents via HTTP, and the same VIP used by slave
> searchers for replication? That sounds like it would work.
Precisely what you should do. e.g. "master-vip" is the "hostname" that both SolrJ would post new docs to and the master "server" slaves would poll for index changes.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
> On Oct 11, 2011, at 3:16 PM, Otis Gospodnetic wrote:
>
>> Hello,
>>
>> Yes, you've read about NFS, which is why I gave the example of a SAN
> (which can have multiple power supplies, controllers, etc.)
>>
>> Yes, should be OK to have multiple Solr instances have the same index open,
> since only one of them will actually be writing to it, thanks to LB.
>>
>> Otis
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> Lucene ecosystem search :: http://search-lucene.com/
>>
>>
>>> ________________________________
>>> From: Brandon Ramirez <Br...@elementk.com>
>>> To: "solr-user@lucene.apache.org"
> <so...@lucene.apache.org>
>>> Sent: Tuesday, October 11, 2011 2:55 PM
>>> Subject: RE: Replication with an HA master
>>>
>>> Using a shared volume crossed my mind too, but I discarded the idea
> because of literature I have read about Lucene performing poorly against remote
> file systems. But then I suppose a SAN wouldn't be a remote file system in
> the same sense as an NFS-mounted NAS or similar.
>>>
>>> Should I be concerned about two solr instances on two machines having
> the same SAN-based index open, as long as only one of them is receiving
> requests? I would think in theory it would work, but I don't have any
> production-level experience with Solr yet, only textbook knowledge.
>>>
>>>
>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
>>> Software Engineer II | Element K | www.elementk.com
>>>
>>>
>>> -----Original Message-----
>>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>>> Sent: Tuesday, October 11, 2011 2:28 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Replication with an HA master
>>>
>>> A few alternatives:
>>> * Have the master keep the index on a shared disk (e.g. SAN)
>>> * Use LB to easily switch to between masters, potentially even
> automatically if LB can detect the primary is down
>>>
>>> Otis
>>> ----
>>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene
> ecosystem search :: http://search-lucene.com/
>>>
>>>
>>>> ________________________________
>>>> From: Robert Stewart <bs...@gmail.com>
>>>> To: solr-user@lucene.apache.org
>>>> Sent: Friday, October 7, 2011 10:22 AM
>>>> Subject: Re: Replication with an HA master
>>>>
>>>> Your idea sounds like the correct path. Setup 2 masters, one
> running
>>>> in "slave" mode which pulls replicas from the live
> master. When/if live master goes down, you just reconfigure and restart the
> backup master to be the live master. You'd also need to then start data
> import on the backup master (enable/start cron job?), and redirect slave
> searchers to pull replicas from the new live master. All that could be done
> using scripts or something like puppet possibly.
>>>>
>>>> Another solution maybe is to run 2 "live" masters, which
> both index the same content from the same data source. If one goes down, then
> you just need to redirect slave searchers to the backup master for replication.
>>>>
>>>> I am also starting a similar project which needs some disaster
> recovery processes in place, so any other info would be useful to me as well.
>>>>
>>>> Bob
>>>>
>>>> On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
>>>>
>>>>> We are getting ready to start a project using Solr as our
> backend search engine and I am trying to devise a deployment architecture that
> works for us. We definitely need a master/slave replication strategy,
> that's for sure, but my concern is the master becomes a single point of
> failure.
>>>>>
>>>>> Fortunately, real-time search is not a requirement for us. If
> search results are a few minutes out of sync with our database, it's not a
> big deal.
>>>>>
>>>>> So what I would like to do is have a set of query servers
> (slaves) that are only used for querying, no indexing and have them use
> Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA
> indexing, I'd like to have 2 masters: a primary and a standby. All indexing
> requests go to the primary unless it's taken out of service. To keep the
> standby ready to takeover if it needs to, it needs to be more up to date than
> the slaves. I'd like to have it replicate every 30 seconds or so.
>>>>>
>>>>> The reason I'm asking about it on this list is that I
> haven't seen any Solr documentation or even anything that talks about this.
> I can't be the only one concerned about having a single point of failure, so
> I'm reaching out to see what others have done in this case before I go with
> my own solution.
>>>>>
>>>>>
>>>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
> Software
>>>>> Engineer II | Element K |
> www.elementk.com<http://www.elementk.com/>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>
Re: Replication with an HA master
Posted by Robert Stewart <bs...@gmail.com>.
In the case of using a shared (SAN) index between 2 masters, what happens if the live master fails in such a way that the index remains "locked" (such as if some hardware failure and it did not unlock/close index). Will the other master be able to open/write to the index as new documents are added?
Also, if that can work ok, would it work if you have a LB (VIP) from both indexing and replication sides of the 2 masters, such that some VIP used by solrj for indexing new documents via HTTP, and the same VIP used by slave searchers for replication? That sounds like it would work.
On Oct 11, 2011, at 3:16 PM, Otis Gospodnetic wrote:
> Hello,
>
> Yes, you've read about NFS, which is why I gave the example of a SAN (which can have multiple power supplies, controllers, etc.)
>
> Yes, should be OK to have multiple Solr instances have the same index open, since only one of them will actually be writing to it, thanks to LB.
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>> ________________________________
>> From: Brandon Ramirez <Br...@elementk.com>
>> To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
>> Sent: Tuesday, October 11, 2011 2:55 PM
>> Subject: RE: Replication with an HA master
>>
>> Using a shared volume crossed my mind too, but I discarded the idea because of literature I have read about Lucene performing poorly against remote file systems. But then I suppose a SAN wouldn't be a remote file system in the same sense as an NFS-mounted NAS or similar.
>>
>> Should I be concerned about two solr instances on two machines having the same SAN-based index open, as long as only one of them is receiving requests? I would think in theory it would work, but I don't have any production-level experience with Solr yet, only textbook knowledge.
>>
>>
>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
>> Software Engineer II | Element K | www.elementk.com
>>
>>
>> -----Original Message-----
>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>> Sent: Tuesday, October 11, 2011 2:28 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Replication with an HA master
>>
>> A few alternatives:
>> * Have the master keep the index on a shared disk (e.g. SAN)
>> * Use LB to easily switch to between masters, potentially even automatically if LB can detect the primary is down
>>
>> Otis
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/
>>
>>
>>> ________________________________
>>> From: Robert Stewart <bs...@gmail.com>
>>> To: solr-user@lucene.apache.org
>>> Sent: Friday, October 7, 2011 10:22 AM
>>> Subject: Re: Replication with an HA master
>>>
>>> Your idea sounds like the correct path. Setup 2 masters, one running
>>> in "slave" mode which pulls replicas from the live master. When/if live master goes down, you just reconfigure and restart the backup master to be the live master. You'd also need to then start data import on the backup master (enable/start cron job?), and redirect slave searchers to pull replicas from the new live master. All that could be done using scripts or something like puppet possibly.
>>>
>>> Another solution maybe is to run 2 "live" masters, which both index the same content from the same data source. If one goes down, then you just need to redirect slave searchers to the backup master for replication.
>>>
>>> I am also starting a similar project which needs some disaster recovery processes in place, so any other info would be useful to me as well.
>>>
>>> Bob
>>>
>>> On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
>>>
>>>> We are getting ready to start a project using Solr as our backend search engine and I am trying to devise a deployment architecture that works for us. We definitely need a master/slave replication strategy, that's for sure, but my concern is the master becomes a single point of failure.
>>>>
>>>> Fortunately, real-time search is not a requirement for us. If search results are a few minutes out of sync with our database, it's not a big deal.
>>>>
>>>> So what I would like to do is have a set of query servers (slaves) that are only used for querying, no indexing and have them use Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA indexing, I'd like to have 2 masters: a primary and a standby. All indexing requests go to the primary unless it's taken out of service. To keep the standby ready to takeover if it needs to, it needs to be more up to date than the slaves. I'd like to have it replicate every 30 seconds or so.
>>>>
>>>> The reason I'm asking about it on this list is that I haven't seen any Solr documentation or even anything that talks about this. I can't be the only one concerned about having a single point of failure, so I'm reaching out to see what others have done in this case before I go with my own solution.
>>>>
>>>>
>>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848 Software
>>>> Engineer II | Element K | www.elementk.com<http://www.elementk.com/>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>>
Re: Replication with an HA master
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hello,
Yes, you've read about NFS, which is why I gave the example of a SAN (which can have multiple power supplies, controllers, etc.)
Yes, should be OK to have multiple Solr instances have the same index open, since only one of them will actually be writing to it, thanks to LB.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
>________________________________
>From: Brandon Ramirez <Br...@elementk.com>
>To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
>Sent: Tuesday, October 11, 2011 2:55 PM
>Subject: RE: Replication with an HA master
>
>Using a shared volume crossed my mind too, but I discarded the idea because of literature I have read about Lucene performing poorly against remote file systems. But then I suppose a SAN wouldn't be a remote file system in the same sense as an NFS-mounted NAS or similar.
>
>Should I be concerned about two solr instances on two machines having the same SAN-based index open, as long as only one of them is receiving requests? I would think in theory it would work, but I don't have any production-level experience with Solr yet, only textbook knowledge.
>
>
>Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
>Software Engineer II | Element K | www.elementk.com
>
>
>-----Original Message-----
>From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>Sent: Tuesday, October 11, 2011 2:28 PM
>To: solr-user@lucene.apache.org
>Subject: Re: Replication with an HA master
>
>A few alternatives:
>* Have the master keep the index on a shared disk (e.g. SAN)
>* Use LB to easily switch to between masters, potentially even automatically if LB can detect the primary is down
>
>Otis
>----
>Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/
>
>
>>________________________________
>>From: Robert Stewart <bs...@gmail.com>
>>To: solr-user@lucene.apache.org
>>Sent: Friday, October 7, 2011 10:22 AM
>>Subject: Re: Replication with an HA master
>>
>>Your idea sounds like the correct path. Setup 2 masters, one running
>>in "slave" mode which pulls replicas from the live master. When/if live master goes down, you just reconfigure and restart the backup master to be the live master. You'd also need to then start data import on the backup master (enable/start cron job?), and redirect slave searchers to pull replicas from the new live master. All that could be done using scripts or something like puppet possibly.
>>
>>Another solution maybe is to run 2 "live" masters, which both index the same content from the same data source. If one goes down, then you just need to redirect slave searchers to the backup master for replication.
>>
>>I am also starting a similar project which needs some disaster recovery processes in place, so any other info would be useful to me as well.
>>
>>Bob
>>
>>On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
>>
>>> We are getting ready to start a project using Solr as our backend search engine and I am trying to devise a deployment architecture that works for us. We definitely need a master/slave replication strategy, that's for sure, but my concern is the master becomes a single point of failure.
>>>
>>> Fortunately, real-time search is not a requirement for us. If search results are a few minutes out of sync with our database, it's not a big deal.
>>>
>>> So what I would like to do is have a set of query servers (slaves) that are only used for querying, no indexing and have them use Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA indexing, I'd like to have 2 masters: a primary and a standby. All indexing requests go to the primary unless it's taken out of service. To keep the standby ready to takeover if it needs to, it needs to be more up to date than the slaves. I'd like to have it replicate every 30 seconds or so.
>>>
>>> The reason I'm asking about it on this list is that I haven't seen any Solr documentation or even anything that talks about this. I can't be the only one concerned about having a single point of failure, so I'm reaching out to see what others have done in this case before I go with my own solution.
>>>
>>>
>>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848 Software
>>> Engineer II | Element K | www.elementk.com<http://www.elementk.com/>
>>>
>>
>>
>>
>>
>
>
>
>
Re: Replication with an HA master
Posted by Ted Dunning <te...@gmail.com>.
On Tue, Oct 11, 2011 at 6:55 PM, Brandon Ramirez <
Brandon_Ramirez@elementk.com> wrote:
> Using a shared volume crossed my mind too, but I discarded the idea because
> of literature I have read about Lucene performing poorly against remote file
> systems. But then I suppose a SAN wouldn't be a remote file system in the
> same sense as an NFS-mounted NAS or similar.
>
I have had one major customer who has tested Solr on MapR's via an NFS
export and they reported very good results.
I think that the common wisdom of "don't use Lucene on NFS" comes from two
sources, but now somehwhat dated:
- the old problem with Lucene assuming that an open file would not be
deleted. This has been fixed approximately forever.
- crummy NFS servers give results that are, well, crummy.
Should I be concerned about two solr instances on two machines having the
> same SAN-based index open, as long as only one of them is receiving
> requests?
>
I think that you would want to reopen the index whenever there is a master
switch. If you aren't doing updates, then both can open the same index. If
one is doing updates, then you can use the second for searches as long as
you make sure that the indexer has the parameters set correctly to not
delete old files for a period long enough to allow the second to reopen the
new version of the index.
If you are using a system like MapR (or even a NetApp), then you can use
snapshots to avoid deletions. With a setup like that, you would have the
reader switch to a new snapshot each time one appears and set the writer
directory to snapshot every minute or few. Whether that fits your need is
a different question, of course.
I would strongly recommend coordinating who is writer and who is not using a
system like a leader election in Zookeeper.
RE: Replication with an HA master
Posted by Brandon Ramirez <Br...@elementk.com>.
Using a shared volume crossed my mind too, but I discarded the idea because of literature I have read about Lucene performing poorly against remote file systems. But then I suppose a SAN wouldn't be a remote file system in the same sense as an NFS-mounted NAS or similar.
Should I be concerned about two solr instances on two machines having the same SAN-based index open, as long as only one of them is receiving requests? I would think in theory it would work, but I don't have any production-level experience with Solr yet, only textbook knowledge.
Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
Software Engineer II | Element K | www.elementk.com
-----Original Message-----
From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
Sent: Tuesday, October 11, 2011 2:28 PM
To: solr-user@lucene.apache.org
Subject: Re: Replication with an HA master
A few alternatives:
* Have the master keep the index on a shared disk (e.g. SAN)
* Use LB to easily switch to between masters, potentially even automatically if LB can detect the primary is down
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/
>________________________________
>From: Robert Stewart <bs...@gmail.com>
>To: solr-user@lucene.apache.org
>Sent: Friday, October 7, 2011 10:22 AM
>Subject: Re: Replication with an HA master
>
>Your idea sounds like the correct path. Setup 2 masters, one running
>in "slave" mode which pulls replicas from the live master. When/if live master goes down, you just reconfigure and restart the backup master to be the live master. You'd also need to then start data import on the backup master (enable/start cron job?), and redirect slave searchers to pull replicas from the new live master. All that could be done using scripts or something like puppet possibly.
>
>Another solution maybe is to run 2 "live" masters, which both index the same content from the same data source. If one goes down, then you just need to redirect slave searchers to the backup master for replication.
>
>I am also starting a similar project which needs some disaster recovery processes in place, so any other info would be useful to me as well.
>
>Bob
>
>On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
>
>> We are getting ready to start a project using Solr as our backend search engine and I am trying to devise a deployment architecture that works for us. We definitely need a master/slave replication strategy, that's for sure, but my concern is the master becomes a single point of failure.
>>
>> Fortunately, real-time search is not a requirement for us. If search results are a few minutes out of sync with our database, it's not a big deal.
>>
>> So what I would like to do is have a set of query servers (slaves) that are only used for querying, no indexing and have them use Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA indexing, I'd like to have 2 masters: a primary and a standby. All indexing requests go to the primary unless it's taken out of service. To keep the standby ready to takeover if it needs to, it needs to be more up to date than the slaves. I'd like to have it replicate every 30 seconds or so.
>>
>> The reason I'm asking about it on this list is that I haven't seen any Solr documentation or even anything that talks about this. I can't be the only one concerned about having a single point of failure, so I'm reaching out to see what others have done in this case before I go with my own solution.
>>
>>
>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848 Software
>> Engineer II | Element K | www.elementk.com<http://www.elementk.com/>
>>
>
>
>
>
Re: Replication with an HA master
Posted by Otis Gospodnetic <ot...@yahoo.com>.
A few alternatives:
* Have the master keep the index on a shared disk (e.g. SAN)
* Use LB to easily switch to between masters, potentially even automatically if LB can detect the primary is down
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
>________________________________
>From: Robert Stewart <bs...@gmail.com>
>To: solr-user@lucene.apache.org
>Sent: Friday, October 7, 2011 10:22 AM
>Subject: Re: Replication with an HA master
>
>Your idea sounds like the correct path. Setup 2 masters, one running in "slave" mode which pulls replicas from the live master. When/if live master goes down, you just reconfigure and restart the backup master to be the live master. You'd also need to then start data import on the backup master (enable/start cron job?), and redirect slave searchers to pull replicas from the new live master. All that could be done using scripts or something like puppet possibly.
>
>Another solution maybe is to run 2 "live" masters, which both index the same content from the same data source. If one goes down, then you just need to redirect slave searchers to the backup master for replication.
>
>I am also starting a similar project which needs some disaster recovery processes in place, so any other info would be useful to me as well.
>
>Bob
>
>On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
>
>> We are getting ready to start a project using Solr as our backend search engine and I am trying to devise a deployment architecture that works for us. We definitely need a master/slave replication strategy, that's for sure, but my concern is the master becomes a single point of failure.
>>
>> Fortunately, real-time search is not a requirement for us. If search results are a few minutes out of sync with our database, it's not a big deal.
>>
>> So what I would like to do is have a set of query servers (slaves) that are only used for querying, no indexing and have them use Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA indexing, I'd like to have 2 masters: a primary and a standby. All indexing requests go to the primary unless it's taken out of service. To keep the standby ready to takeover if it needs to, it needs to be more up to date than the slaves. I'd like to have it replicate every 30 seconds or so.
>>
>> The reason I'm asking about it on this list is that I haven't seen any Solr documentation or even anything that talks about this. I can't be the only one concerned about having a single point of failure, so I'm reaching out to see what others have done in this case before I go with my own solution.
>>
>>
>> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
>> Software Engineer II | Element K | www.elementk.com<http://www.elementk.com/>
>>
>
>
>
>
Re: Replication with an HA master
Posted by Robert Stewart <bs...@gmail.com>.
Your idea sounds like the correct path. Setup 2 masters, one running in "slave" mode which pulls replicas from the live master. When/if live master goes down, you just reconfigure and restart the backup master to be the live master. You'd also need to then start data import on the backup master (enable/start cron job?), and redirect slave searchers to pull replicas from the new live master. All that could be done using scripts or something like puppet possibly.
Another solution maybe is to run 2 "live" masters, which both index the same content from the same data source. If one goes down, then you just need to redirect slave searchers to the backup master for replication.
I am also starting a similar project which needs some disaster recovery processes in place, so any other info would be useful to me as well.
Bob
On Oct 7, 2011, at 9:53 AM, Brandon Ramirez wrote:
> We are getting ready to start a project using Solr as our backend search engine and I am trying to devise a deployment architecture that works for us. We definitely need a master/slave replication strategy, that's for sure, but my concern is the master becomes a single point of failure.
>
> Fortunately, real-time search is not a requirement for us. If search results are a few minutes out of sync with our database, it's not a big deal.
>
> So what I would like to do is have a set of query servers (slaves) that are only used for querying, no indexing and have them use Solr's HTTP replication mechanism on a 2 or 3 minute interval. To get HA indexing, I'd like to have 2 masters: a primary and a standby. All indexing requests go to the primary unless it's taken out of service. To keep the standby ready to takeover if it needs to, it needs to be more up to date than the slaves. I'd like to have it replicate every 30 seconds or so.
>
> The reason I'm asking about it on this list is that I haven't seen any Solr documentation or even anything that talks about this. I can't be the only one concerned about having a single point of failure, so I'm reaching out to see what others have done in this case before I go with my own solution.
>
>
> Brandon Ramirez | Office: 585.214.5413 | Fax: 585.295.4848
> Software Engineer II | Element K | www.elementk.com<http://www.elementk.com/>
>