You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Damien Tournoud <da...@tournoud.net> on 2009/05/21 18:37:10 UTC
No sanity checks before replicating files?
Hi list,
We have deployed an experimental Solr 1.4 cluster (a master/slave
setup, with automatic promotion of the slave as a master in case of
failure) on drupal.org, to manage our medium size index (3GB, about
400K documents).
One of the problem we are facing is that there seems to be no sanity
checks before downloading files. Take the following scenario:
- initial situation: s1 is master, s2 is slave
- s1 fails, the virtual IP falls back to s2
- some updates happen on s2
- suppose now that s1 gets back online, s2 tries to replicate from
s1, but after replicating all the files (3GB), the commit fails
because the local index has been locally updated, the replication
fails, but the process restarts at the next poll (redownload all the
index files, fails again...) and so on
We are considering configuring each server to replicate from the
virtual IP, which should solve that issue for us, but couldn't the
slave do some sanity checks before trying to download all the files
from the master?
Thanks in advance for any help you could provide,
Damien Tournoud
Re: No sanity checks before replicating files?
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
I think this problem might happen when there are uncommitted changes in S2
and the master S1 comes back online. In that case, slave's generation is
still less than master's and installation of index diff from master may
fail.
However, I do not understand a few points. Damien, if S1 comes back online
and S2 starts replicating from S1, any changes to S2 will be discarded when
a successful replication happens. How do you intend to protect against that?
A better way is to detect when S1 comes back online and make it a slave of
S2.
2009/5/22 Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>
> Let us see what is the desired behavior.
>
> When s1 comes back up online , s2 must download a fresh copy of index
> from s1 because s1 is the slave and s2 has a newer version of index
> than s1.
>
> Are you suggesting that s2 downloads the index files and then commit
> fails? The code is written as follows
>
> boolean freshDownloadneeded = myIndexGeneration >= mastersIndexgeneration;
>
> then it should be a problem
>
> can u post the stacktrace?
>
> On Thu, May 21, 2009 at 11:45 PM, Otis Gospodnetic
> <ot...@yahoo.com> wrote:
> >
> > Aha, I see. Perhaps you can post the error message/stack trace?
> >
> > As for the sanity check, I bet a call to http://host:port/solr/replication?command=indexversion
> could be used ensure only newer versions of the index are being pulled.
> We'll see what Paul says when he wakes up. :)
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: Damien Tournoud <da...@tournoud.net>
> >> To: solr-user@lucene.apache.org
> >> Sent: Thursday, May 21, 2009 1:26:30 PM
> >> Subject: Re: No sanity checks before replicating files?
> >>
> >> Hi Otis,
> >>
> >> Thanks for your answer.
> >>
> >> On Thu, May 21, 2009 at 7:14 PM, Otis Gospodnetic
> >> wrote:
> >> > Interesting, this is similar to my suggestion to another person I just
> replied
> >> to here on solr-user.
> >> > Have you actually run into this problem? I haven't tried it, but I'd
> think
> >> the first next replication (copying index from s1 to s2) would not
> necessarily
> >> fail, but would simply overwrite any changes that were made on s2 while
> it was
> >> serving as the master. Is that not what happens?
> >>
> >> No it doesn't. For some reason, Solr download all the files of the
> >> index, but fails to commit the changes locally. At the next poll, the
> >> process restarts. Not only does this clogs the network, but it also
> >> unnecessarily uses resources on the newly promoted slave, until we
> >> change its configuration.
> >>
> >> > If that's what happens, then I think what you'd simply have to do is
> to:
> >> >
> >> > 1) bring s1 back up, but don't make it a master immediately
> >> > 2) take away the master role from s2
> >> > 3) make s1 copy the index from s2, since s2 might have a more up to
> date index
> >> now
> >> > 4) make s1 the master
> >>
> >> Once s2 is the master, we want it to stay this way. We will reassign
> >> s1 as the slave at a later stage, when resources allows. What worries
> >> me is that strange behavior of Solr 1.4 replication when the "slave"
> >> index is fresher then the "master" one.
> >>
> >> Damien
> >
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>
--
Regards,
Shalin Shekhar Mangar.
Re: No sanity checks before replicating files?
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
Let us see what is the desired behavior.
When s1 comes back up online , s2 must download a fresh copy of index
from s1 because s1 is the slave and s2 has a newer version of index
than s1.
Are you suggesting that s2 downloads the index files and then commit
fails? The code is written as follows
boolean freshDownloadneeded = myIndexGeneration >= mastersIndexgeneration;
then it should be a problem
can u post the stacktrace?
On Thu, May 21, 2009 at 11:45 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
>
> Aha, I see. Perhaps you can post the error message/stack trace?
>
> As for the sanity check, I bet a call to http://host:port/solr/replication?command=indexversion could be used ensure only newer versions of the index are being pulled. We'll see what Paul says when he wakes up. :)
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: Damien Tournoud <da...@tournoud.net>
>> To: solr-user@lucene.apache.org
>> Sent: Thursday, May 21, 2009 1:26:30 PM
>> Subject: Re: No sanity checks before replicating files?
>>
>> Hi Otis,
>>
>> Thanks for your answer.
>>
>> On Thu, May 21, 2009 at 7:14 PM, Otis Gospodnetic
>> wrote:
>> > Interesting, this is similar to my suggestion to another person I just replied
>> to here on solr-user.
>> > Have you actually run into this problem? I haven't tried it, but I'd think
>> the first next replication (copying index from s1 to s2) would not necessarily
>> fail, but would simply overwrite any changes that were made on s2 while it was
>> serving as the master. Is that not what happens?
>>
>> No it doesn't. For some reason, Solr download all the files of the
>> index, but fails to commit the changes locally. At the next poll, the
>> process restarts. Not only does this clogs the network, but it also
>> unnecessarily uses resources on the newly promoted slave, until we
>> change its configuration.
>>
>> > If that's what happens, then I think what you'd simply have to do is to:
>> >
>> > 1) bring s1 back up, but don't make it a master immediately
>> > 2) take away the master role from s2
>> > 3) make s1 copy the index from s2, since s2 might have a more up to date index
>> now
>> > 4) make s1 the master
>>
>> Once s2 is the master, we want it to stay this way. We will reassign
>> s1 as the slave at a later stage, when resources allows. What worries
>> me is that strange behavior of Solr 1.4 replication when the "slave"
>> index is fresher then the "master" one.
>>
>> Damien
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: No sanity checks before replicating files?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Aha, I see. Perhaps you can post the error message/stack trace?
As for the sanity check, I bet a call to http://host:port/solr/replication?command=indexversion could be used ensure only newer versions of the index are being pulled. We'll see what Paul says when he wakes up. :)
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Damien Tournoud <da...@tournoud.net>
> To: solr-user@lucene.apache.org
> Sent: Thursday, May 21, 2009 1:26:30 PM
> Subject: Re: No sanity checks before replicating files?
>
> Hi Otis,
>
> Thanks for your answer.
>
> On Thu, May 21, 2009 at 7:14 PM, Otis Gospodnetic
> wrote:
> > Interesting, this is similar to my suggestion to another person I just replied
> to here on solr-user.
> > Have you actually run into this problem? I haven't tried it, but I'd think
> the first next replication (copying index from s1 to s2) would not necessarily
> fail, but would simply overwrite any changes that were made on s2 while it was
> serving as the master. Is that not what happens?
>
> No it doesn't. For some reason, Solr download all the files of the
> index, but fails to commit the changes locally. At the next poll, the
> process restarts. Not only does this clogs the network, but it also
> unnecessarily uses resources on the newly promoted slave, until we
> change its configuration.
>
> > If that's what happens, then I think what you'd simply have to do is to:
> >
> > 1) bring s1 back up, but don't make it a master immediately
> > 2) take away the master role from s2
> > 3) make s1 copy the index from s2, since s2 might have a more up to date index
> now
> > 4) make s1 the master
>
> Once s2 is the master, we want it to stay this way. We will reassign
> s1 as the slave at a later stage, when resources allows. What worries
> me is that strange behavior of Solr 1.4 replication when the "slave"
> index is fresher then the "master" one.
>
> Damien
Re: No sanity checks before replicating files?
Posted by Damien Tournoud <da...@tournoud.net>.
Hi Otis,
Thanks for your answer.
On Thu, May 21, 2009 at 7:14 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> Interesting, this is similar to my suggestion to another person I just replied to here on solr-user.
> Have you actually run into this problem? I haven't tried it, but I'd think the first next replication (copying index from s1 to s2) would not necessarily fail, but would simply overwrite any changes that were made on s2 while it was serving as the master. Is that not what happens?
No it doesn't. For some reason, Solr download all the files of the
index, but fails to commit the changes locally. At the next poll, the
process restarts. Not only does this clogs the network, but it also
unnecessarily uses resources on the newly promoted slave, until we
change its configuration.
> If that's what happens, then I think what you'd simply have to do is to:
>
> 1) bring s1 back up, but don't make it a master immediately
> 2) take away the master role from s2
> 3) make s1 copy the index from s2, since s2 might have a more up to date index now
> 4) make s1 the master
Once s2 is the master, we want it to stay this way. We will reassign
s1 as the slave at a later stage, when resources allows. What worries
me is that strange behavior of Solr 1.4 replication when the "slave"
index is fresher then the "master" one.
Damien
Re: No sanity checks before replicating files?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Damien,
Interesting, this is similar to my suggestion to another person I just replied to here on solr-user.
Have you actually run into this problem? I haven't tried it, but I'd think the first next replication (copying index from s1 to s2) would not necessarily fail, but would simply overwrite any changes that were made on s2 while it was serving as the master. Is that not what happens? If that's what happens, then I think what you'd simply have to do is to:
1) bring s1 back up, but don't make it a master immediately
2) take away the master role from s2
3) make s1 copy the index from s2, since s2 might have a more up to date index now
4) make s1 the master
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Damien Tournoud <da...@tournoud.net>
> To: solr-user@lucene.apache.org
> Sent: Thursday, May 21, 2009 12:37:10 PM
> Subject: No sanity checks before replicating files?
>
> Hi list,
>
> We have deployed an experimental Solr 1.4 cluster (a master/slave
> setup, with automatic promotion of the slave as a master in case of
> failure) on drupal.org, to manage our medium size index (3GB, about
> 400K documents).
>
> One of the problem we are facing is that there seems to be no sanity
> checks before downloading files. Take the following scenario:
>
> - initial situation: s1 is master, s2 is slave
> - s1 fails, the virtual IP falls back to s2
> - some updates happen on s2
> - suppose now that s1 gets back online, s2 tries to replicate from
> s1, but after replicating all the files (3GB), the commit fails
> because the local index has been locally updated, the replication
> fails, but the process restarts at the next poll (redownload all the
> index files, fails again...) and so on
>
> We are considering configuring each server to replicate from the
> virtual IP, which should solve that issue for us, but couldn't the
> slave do some sanity checks before trying to download all the files
> from the master?
>
> Thanks in advance for any help you could provide,
>
> Damien Tournoud