You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rakesh Godhani <rg...@sermo.com> on 2008/07/28 20:25:00 UTC
Multiple Update servers
Hi, we are currently evaluating Solr and have been browsing the archives for
one particular issue but can¹t seem to find the answer, so please forgive me
if I¹m asking a repetitive question. We like the idea of having multiple
slave servers serving up queries and a master performing updates. However
the the issue for us there is no redundancy for the master. So a couple of
questions:
1. Can there be multiple masters (or update servers) sharing the same index
files, performing updates at the same time (ie. Hosting the index on a SAN)?
2. Is there a recommended architecture utilizing a SAN. (For example 2
slaves and 2 masters sharing a SAN). We current don¹t have that many
records prob about a million and growing. We are mainly concerned about
redundancy, then performance.
Thanks
-Rakesh
Re: Multiple Update servers
Posted by Alexander Ramos Jardim <al...@gmail.com>.
I use the snapshooter with rsync
2008/7/29 Rakesh Godhani <rg...@sermo.com>
> After Matthew's comment I was thinking about putting them both behind a
> load
> balancer, with the LB directing all traffic to one until it fails and then
> kick over to the other one.
>
> In your architectures I'm guessing the masters share the same physical
> index, but do the slaves share the same index as the masters or do you use
> rsync or some other mechanism to distribute copies.
>
> Thanks
> -Rakesh
>
>
>
>
> On 7/29/08 5:07 PM, "Alexander Ramos Jardim"
> <al...@gmail.com> wrote:
>
> > You could implement a script that woiuld control which master server is
> > indexing and put them behind something like a NAT.
> >
> > I use that that control my master redundancy.
> >
> > 2008/7/29 Rakesh Godhani <rg...@sermo.com>
> >
> >> Thanks for the input, much appreciated.
> >> -Rakesh
> >>
> >>
> >>
> >> On 7/29/08 12:18 PM, "Matthew Runo" <mr...@zappos.com> wrote:
> >>
> >>> As far as I know only one machine can write to an index at a time.
> >>> More than that and I got corrupted indexes.
> >>>
> >>> Thanks!
> >>>
> >>> Matthew Runo
> >>> Software Developer
> >>> Zappos.com
> >>> 702.943.7833
> >>>
> >>> On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:
> >>>
> >>>>
> >>>> Hi, we are currently evaluating Solr and have been browsing the
> >>>> archives for
> >>>> one particular issue but can¹t seem to find the answer, so please
> >>>> forgive me
> >>>> if I¹m asking a repetitive question. We like the idea of having
> >>>> multiple
> >>>> slave servers serving up queries and a master performing updates.
> >>>> However
> >>>> the the issue for us there is no redundancy for the master. So a
> >>>> couple of
> >>>> questions:
> >>>>
> >>>> 1. Can there be multiple masters (or update servers) sharing the
> >>>> same index
> >>>> files, performing updates at the same time (ie. Hosting the index on
> >>>> a SAN)?
> >>>>
> >>>> 2. Is there a recommended architecture utilizing a SAN. (For
> >>>> example 2
> >>>> slaves and 2 masters sharing a SAN). We current don¹t have that many
> >>>> records prob about a million and growing. We are mainly concerned
> >>>> about
> >>>> redundancy, then performance.
> >>>>
> >>>> Thanks
> >>>> -Rakesh
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >
>
>
>
--
Alexander Ramos Jardim
Re: Multiple Update servers
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi Rakesh,
Solr ships with a set of collection distribution scripts (rsync based) which
can be used to setup master-slave replication. I believe most people use
them for transferring index from master to slaves and use separate local
filesystems on their servers.
Take a look at http://wiki.apache.org/solr/CollectionDistribution
On Wed, Jul 30, 2008 at 3:10 AM, Rakesh Godhani <rg...@sermo.com> wrote:
> After Matthew's comment I was thinking about putting them both behind a
> load
> balancer, with the LB directing all traffic to one until it fails and then
> kick over to the other one.
If both masters use the same index on the SAN then this may not work because
if a failure happens during indexing then the other master will try to open
an IndexWriter on the shared index and may fail with LockObtainedException
because the other master will have already obtained a lock on it. Note that
lock is only obtained by masters and not by slaves. Probably someone who
knows more about this part can confirm.
>
>
> In your architectures I'm guessing the masters share the same physical
> index, but do the slaves share the same index as the masters or do you use
> rsync or some other mechanism to distribute copies.
>
If you want to use a SAN
--
Regards,
Shalin Shekhar Mangar.
Re: Multiple Update servers
Posted by Rakesh Godhani <rg...@sermo.com>.
After Matthew's comment I was thinking about putting them both behind a load
balancer, with the LB directing all traffic to one until it fails and then
kick over to the other one.
In your architectures I'm guessing the masters share the same physical
index, but do the slaves share the same index as the masters or do you use
rsync or some other mechanism to distribute copies.
Thanks
-Rakesh
On 7/29/08 5:07 PM, "Alexander Ramos Jardim"
<al...@gmail.com> wrote:
> You could implement a script that woiuld control which master server is
> indexing and put them behind something like a NAT.
>
> I use that that control my master redundancy.
>
> 2008/7/29 Rakesh Godhani <rg...@sermo.com>
>
>> Thanks for the input, much appreciated.
>> -Rakesh
>>
>>
>>
>> On 7/29/08 12:18 PM, "Matthew Runo" <mr...@zappos.com> wrote:
>>
>>> As far as I know only one machine can write to an index at a time.
>>> More than that and I got corrupted indexes.
>>>
>>> Thanks!
>>>
>>> Matthew Runo
>>> Software Developer
>>> Zappos.com
>>> 702.943.7833
>>>
>>> On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:
>>>
>>>>
>>>> Hi, we are currently evaluating Solr and have been browsing the
>>>> archives for
>>>> one particular issue but can¹t seem to find the answer, so please
>>>> forgive me
>>>> if I¹m asking a repetitive question. We like the idea of having
>>>> multiple
>>>> slave servers serving up queries and a master performing updates.
>>>> However
>>>> the the issue for us there is no redundancy for the master. So a
>>>> couple of
>>>> questions:
>>>>
>>>> 1. Can there be multiple masters (or update servers) sharing the
>>>> same index
>>>> files, performing updates at the same time (ie. Hosting the index on
>>>> a SAN)?
>>>>
>>>> 2. Is there a recommended architecture utilizing a SAN. (For
>>>> example 2
>>>> slaves and 2 masters sharing a SAN). We current don¹t have that many
>>>> records prob about a million and growing. We are mainly concerned
>>>> about
>>>> redundancy, then performance.
>>>>
>>>> Thanks
>>>> -Rakesh
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
Re: Multiple Update servers
Posted by Alexander Ramos Jardim <al...@gmail.com>.
You could implement a script that woiuld control which master server is
indexing and put them behind something like a NAT.
I use that that control my master redundancy.
2008/7/29 Rakesh Godhani <rg...@sermo.com>
> Thanks for the input, much appreciated.
> -Rakesh
>
>
>
> On 7/29/08 12:18 PM, "Matthew Runo" <mr...@zappos.com> wrote:
>
> > As far as I know only one machine can write to an index at a time.
> > More than that and I got corrupted indexes.
> >
> > Thanks!
> >
> > Matthew Runo
> > Software Developer
> > Zappos.com
> > 702.943.7833
> >
> > On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:
> >
> >>
> >> Hi, we are currently evaluating Solr and have been browsing the
> >> archives for
> >> one particular issue but can¹t seem to find the answer, so please
> >> forgive me
> >> if I¹m asking a repetitive question. We like the idea of having
> >> multiple
> >> slave servers serving up queries and a master performing updates.
> >> However
> >> the the issue for us there is no redundancy for the master. So a
> >> couple of
> >> questions:
> >>
> >> 1. Can there be multiple masters (or update servers) sharing the
> >> same index
> >> files, performing updates at the same time (ie. Hosting the index on
> >> a SAN)?
> >>
> >> 2. Is there a recommended architecture utilizing a SAN. (For
> >> example 2
> >> slaves and 2 masters sharing a SAN). We current don¹t have that many
> >> records prob about a million and growing. We are mainly concerned
> >> about
> >> redundancy, then performance.
> >>
> >> Thanks
> >> -Rakesh
> >>
> >>
> >>
> >
> >
>
>
>
--
Alexander Ramos Jardim
Re: Multiple Update servers
Posted by Rakesh Godhani <rg...@sermo.com>.
Thanks for the input, much appreciated.
-Rakesh
On 7/29/08 12:18 PM, "Matthew Runo" <mr...@zappos.com> wrote:
> As far as I know only one machine can write to an index at a time.
> More than that and I got corrupted indexes.
>
> Thanks!
>
> Matthew Runo
> Software Developer
> Zappos.com
> 702.943.7833
>
> On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:
>
>>
>> Hi, we are currently evaluating Solr and have been browsing the
>> archives for
>> one particular issue but can¹t seem to find the answer, so please
>> forgive me
>> if I¹m asking a repetitive question. We like the idea of having
>> multiple
>> slave servers serving up queries and a master performing updates.
>> However
>> the the issue for us there is no redundancy for the master. So a
>> couple of
>> questions:
>>
>> 1. Can there be multiple masters (or update servers) sharing the
>> same index
>> files, performing updates at the same time (ie. Hosting the index on
>> a SAN)?
>>
>> 2. Is there a recommended architecture utilizing a SAN. (For
>> example 2
>> slaves and 2 masters sharing a SAN). We current don¹t have that many
>> records prob about a million and growing. We are mainly concerned
>> about
>> redundancy, then performance.
>>
>> Thanks
>> -Rakesh
>>
>>
>>
>
>
Re: Multiple Update servers
Posted by Matthew Runo <mr...@zappos.com>.
As far as I know only one machine can write to an index at a time.
More than that and I got corrupted indexes.
Thanks!
Matthew Runo
Software Developer
Zappos.com
702.943.7833
On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:
>
> Hi, we are currently evaluating Solr and have been browsing the
> archives for
> one particular issue but can’t seem to find the answer, so please
> forgive me
> if I’m asking a repetitive question. We like the idea of having
> multiple
> slave servers serving up queries and a master performing updates.
> However
> the the issue for us there is no redundancy for the master. So a
> couple of
> questions:
>
> 1. Can there be multiple masters (or update servers) sharing the
> same index
> files, performing updates at the same time (ie. Hosting the index on
> a SAN)?
>
> 2. Is there a recommended architecture utilizing a SAN. (For
> example 2
> slaves and 2 masters sharing a SAN). We current don’t have that many
> records – prob about a million and growing. We are mainly concerned
> about
> redundancy, then performance.
>
> Thanks
> -Rakesh
>
>
>