You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Raúl Grande Durán <ra...@hotmail.com> on 2013/02/19 10:48:22 UTC

Slaves always replicate entire index & Index versions

Hello.
We have recently updated our Solr from 3.5 to 4.1 and everything is running perfect except the replication between nodes. We have a master-repeater-2slaves architecture and we have seen some things that weren't happening before:
When a Slave (repeater or slaves) starts to replicate it needs to download the entire index. Even when some little changes has been made to the index at master. This takes such a long time since our index is more than 20 Gb.After replication cycle we have different index generations in master, repeater and slaves. For example:Master: gen. 64590Repeater: gen. 64591Both slaves: gen. 64592
My replicationHandler configuration is like this:<requestHandler name="/replication" class="solr.ReplicationHandler" >     <lst name="master">       <str name="enable">${enable.master:false}</str>       <str name="replicateAfter">commit</str>       <str name="replicateAfter">startup</str>       <str name="confFiles">schema.xml,stopwords.txt</str>     </lst>     <lst name="slave">       <str name="enable">${enable.slave:false}</str>       <str name="masterUrl">${solr.master.url:"http://localhost/solr"}</str>       <str name="pollInterval">00:03:00</str>     </lst> </requestHandler>
Our problems are very similar to those explained here: http://lucene.472066.n3.nabble.com/Problem-with-replication-td2294313.html
Any ideas?? Thanks 		 	   		  

Re: Slaves always replicate entire index & Index versions

Posted by Amit Nithian <an...@gmail.com>.
A few others have posted about this too apparently and SOLR-4413 is the
root problem. Basically what I am seeing is that if your index directory is
not index/ but rather index.<timestamp> set in the index.properties a new
index will be downloaded all the time because the download is expecting
your index to be in <solr_data_dir>/index. Sounds like a quick solution
might be to rename your index directory to just "index" and see if the
problem goes away.

To confirm, look at line 728 in the SnapPuller.java file (in
downloadIndexFiles)

I am hoping that the patch and a more unified "getIndexDir" can be added to
the next release of Solr as this is a fairly significant bug to me.

Cheers
Amit

On Thu, Feb 21, 2013 at 12:56 AM, Amit Nithian <an...@gmail.com> wrote:

> So the diff in generation numbers are due to the commits I believe that
> Solr does when it has the new index files but the fact that it's
> downloading a new index each time is baffling and I just noticed that too
> (hit the replicate button and noticed a full index download). I'm going to
> pop in to the source and see what's going on to see why unless there's a
> known bug filed about this?
>
>
> On Tue, Feb 19, 2013 at 1:48 AM, Raúl Grande Durán <
> raulgrande83@hotmail.com> wrote:
>
>>
>> Hello.
>> We have recently updated our Solr from 3.5 to 4.1 and everything is
>> running perfect except the replication between nodes. We have a
>> master-repeater-2slaves architecture and we have seen some things that
>> weren't happening before:
>> When a Slave (repeater or slaves) starts to replicate it needs to
>> download the entire index. Even when some little changes has been made to
>> the index at master. This takes such a long time since our index is more
>> than 20 Gb.After replication cycle we have different index generations in
>> master, repeater and slaves. For example:Master: gen. 64590Repeater: gen.
>> 64591Both slaves: gen. 64592
>> My replicationHandler configuration is like this:<requestHandler
>> name="/replication" class="solr.ReplicationHandler" >     <lst
>> name="master">       <str name="enable">${enable.master:false}</str>
>> <str name="replicateAfter">commit</str>       <str
>> name="replicateAfter">startup</str>       <str
>> name="confFiles">schema.xml,stopwords.txt</str>     </lst>     <lst
>> name="slave">       <str name="enable">${enable.slave:false}</str>
>> <str name="masterUrl">${solr.master.url:"http://localhost/solr"}</str>
>>     <str name="pollInterval">00:03:00</str>     </lst> </requestHandler>
>> Our problems are very similar to those explained here:
>> http://lucene.472066.n3.nabble.com/Problem-with-replication-td2294313.html
>> Any ideas?? Thanks
>
>
>

Re: Slaves always replicate entire index & Index versions

Posted by raulgrande83 <ra...@hotmail.com>.
Amit Nithian wrote
> For your issue above in your last post, is it possible that there was a
> commit on the master in that slight window after solr checks for the
> latest
> generation of the master but before it downloads the actual files? How
> frequent are the commits on your master?

No, I don't think there was any commit because our indexing process was
stopped during my first replication tests. We force commits to Solr from our
documents processing scripts, because we need to know exactly when the
document is in fact indexed.




--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4042172.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slaves always replicate entire index & Index versions

Posted by Amit Nithian <an...@gmail.com>.
Sounds good I am trying the combination of my patch and 4413 now to see how
it works and will have to see if I can put unit tests around them as some
of what I thought may not be true with respect to the commit generation
numbers.

For your issue above in your last post, is it possible that there was a
commit on the master in that slight window after solr checks for the latest
generation of the master but before it downloads the actual files? How
frequent are the commits on your master?


On Thu, Feb 21, 2013 at 2:00 AM, raulgrande83 <ra...@hotmail.com>wrote:

> Thanks for the patch, we'll try to install these fixes and post if
> replication works or not.
>
> I renamed 'index.<timestamp>' folders to just 'index' but it didn't work.
> These lines appeared in the log:
> INFO: Master's generation: 64594
> 21-feb-2013 10:42:00 org.apache.solr.handler.SnapPuller fetchLatestIndex
> INFO: Slave's generation: 64593
> 21-feb-2013 10:42:00 org.apache.solr.handler.SnapPuller fetchLatestIndex
> INFO: Starting replication process
> 21-feb-2013 10:42:00 org.apache.solr.handler.SnapPuller fetchFileList
> SEVERE: No files to download for index generation: 64594
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4041827.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Slaves always replicate entire index & Index versions

Posted by raulgrande83 <ra...@hotmail.com>.
Thanks for the patch, we'll try to install these fixes and post if
replication works or not.

I renamed 'index.<timestamp>' folders to just 'index' but it didn't work.
These lines appeared in the log:
INFO: Master's generation: 64594 
21-feb-2013 10:42:00 org.apache.solr.handler.SnapPuller fetchLatestIndex 
INFO: Slave's generation: 64593 
21-feb-2013 10:42:00 org.apache.solr.handler.SnapPuller fetchLatestIndex 
INFO: Starting replication process 
21-feb-2013 10:42:00 org.apache.solr.handler.SnapPuller fetchFileList 
SEVERE: No files to download for index generation: 64594 



--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4041827.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slaves always replicate entire index & Index versions

Posted by Amit Nithian <an...@gmail.com>.
Thanks for the links... I have updated SOLR-4471 with a proposed solution
that I hope can be incorporated or amended so we can get a clean fix into
the next version so our operations and network staff will be happier with
not having gigs of data flying around the network :-)


On Thu, Feb 21, 2013 at 1:24 AM, raulgrande83 <ra...@hotmail.com>wrote:

> Hi Amit,
>
> I have came across some JIRAs that may be useful in this issue:
> https://issues.apache.org/jira/browse/SOLR-4471
> https://issues.apache.org/jira/browse/SOLR-4354
> https://issues.apache.org/jira/browse/SOLR-4303
> https://issues.apache.org/jira/browse/SOLR-4413
> https://issues.apache.org/jira/browse/SOLR-2326
>
> Please, let us know if you find any solution.
>
> Regards.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4041817.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Slaves always replicate entire index & Index versions

Posted by raulgrande83 <ra...@hotmail.com>.
Hi Amit,

I have came across some JIRAs that may be useful in this issue:
https://issues.apache.org/jira/browse/SOLR-4471
https://issues.apache.org/jira/browse/SOLR-4354
https://issues.apache.org/jira/browse/SOLR-4303
https://issues.apache.org/jira/browse/SOLR-4413
https://issues.apache.org/jira/browse/SOLR-2326

Please, let us know if you find any solution.

Regards.



--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4041817.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slaves always replicate entire index & Index versions

Posted by Amit Nithian <an...@gmail.com>.
So the diff in generation numbers are due to the commits I believe that
Solr does when it has the new index files but the fact that it's
downloading a new index each time is baffling and I just noticed that too
(hit the replicate button and noticed a full index download). I'm going to
pop in to the source and see what's going on to see why unless there's a
known bug filed about this?


On Tue, Feb 19, 2013 at 1:48 AM, Raúl Grande Durán <raulgrande83@hotmail.com
> wrote:

>
> Hello.
> We have recently updated our Solr from 3.5 to 4.1 and everything is
> running perfect except the replication between nodes. We have a
> master-repeater-2slaves architecture and we have seen some things that
> weren't happening before:
> When a Slave (repeater or slaves) starts to replicate it needs to download
> the entire index. Even when some little changes has been made to the index
> at master. This takes such a long time since our index is more than 20
> Gb.After replication cycle we have different index generations in master,
> repeater and slaves. For example:Master: gen. 64590Repeater: gen. 64591Both
> slaves: gen. 64592
> My replicationHandler configuration is like this:<requestHandler
> name="/replication" class="solr.ReplicationHandler" >     <lst
> name="master">       <str name="enable">${enable.master:false}</str>
> <str name="replicateAfter">commit</str>       <str
> name="replicateAfter">startup</str>       <str
> name="confFiles">schema.xml,stopwords.txt</str>     </lst>     <lst
> name="slave">       <str name="enable">${enable.slave:false}</str>
> <str name="masterUrl">${solr.master.url:"http://localhost/solr"}</str>
>     <str name="pollInterval">00:03:00</str>     </lst> </requestHandler>
> Our problems are very similar to those explained here:
> http://lucene.472066.n3.nabble.com/Problem-with-replication-td2294313.html
> Any ideas?? Thanks

Re: Slaves always replicate entire index & Index versions

Posted by raulgrande83 <ra...@hotmail.com>.
I always have a "clean" index folder, I mean I don't get the
index.<timestamp> folder at any time. Besides, I haven't noticed that slaves
pull entire index in replication.

I restarted my repeater node and replication started to work fine again, the
slaves updated all the changes that their master had. But after a while the
same problem appeared, I have a lower version in slaves and they doesn't
replicate.

Definitely with these issues I don't think this Solr version is suitable for
a production environment. We'll have to wait a bit more...



--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4043379.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slaves always replicate entire index & Index versions

Posted by Arkadi Colson <ar...@smartbit.be>.
The index folder is indeed gone but it seems to work. Maybe just a 
structural change...

Met vriendelijke groeten

Arkadi Colson

Smartbit bvba • Hoogstraat 13 • 3670 Meeuwen
T +32 11 64 08 80 • F +32 11 64 08 81

On 04/02/2013 04:08 PM, yayati wrote:
> I moved solr 4.1 to solr 4.2 on one of slave server earlier my index
> directory has index.timestamp, but now, it has only index folder no
> timestamp. Is this is bug.?? Though size of index is same as on master . It
> shows replication running on dasboard with both master and slave version.
> what happened to timestamp in index directory
>
>
> index.timestamp  -- earlier with 4.1
>
> index  -- this is new folder
>
> Please reply asap.
>
> thanks
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4053179.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


Re: Slaves always replicate entire index & Index versions

Posted by yayati <ya...@gmail.com>.
I moved solr 4.1 to solr 4.2 on one of slave server earlier my index
directory has index.timestamp, but now, it has only index folder no
timestamp. Is this is bug.?? Though size of index is same as on master . It
shows replication running on dasboard with both master and slave version.
what happened to timestamp in index directory


index.timestamp  -- earlier with 4.1

index  -- this is new folder

Please reply asap.

thanks



--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4053179.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slaves always replicate entire index & Index versions

Posted by raulgrande83 <ra...@hotmail.com>.
I have created a new JIRA issue here
https://issues.apache.org/jira/browse/SOLR-4511



--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4043595.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slaves always replicate entire index & Index versions

Posted by Mark Miller <ma...@gmail.com>.
On Feb 27, 2013, at 4:40 AM, Bernd Fehling <be...@uni-bielefeld.de> wrote:

> Under this circumstances with replication I would not even dream about using SolrCloud.

The funny part about that is that the master->slave replication issue(s) in 4.1 don't apply to SolrCloud, so you would have had better luck :)

- Mark


Re: Slaves always replicate entire index & Index versions

Posted by Bernd Fehling <be...@uni-bielefeld.de>.
May be the info about index version is pulled from the repeaters
"data/replication.properties" file and the content of that file
is wrong.
Had something similar and only solution for me was deleting the
replication.properties file. But no guarantee about this.

Actually the replication is pretty much messed up in solr4.1.
Have seen about 6 or 7 erroneous combinations with replication that were
leading to some kind of problems. My problem is I can't reproduce it
continously to use a debugger :-(
Positive point is, if something goes wrong it goes wrong on all slaves.
Some kind of continuity :-)

While writing this I just found a new combination.
- master had clean index (everything committed, and optimized) and was
  successfully replicated to all slaves
- master has added a few docs and committed
- master was replicated to all slaves
- all slaves have same generation and version as master
but:
- all slaves have now no index directory anymore. They only have an index.xxxxxxxxx
  directory and an additional index.properties file.

I already knew that something would go wrong when I started replication
and saw that the slaves pulled the whole index (again) from the master and
not only the files with added docs.

Under this circumstances with replication I would not even dream about using SolrCloud.

Regards
Bernd


Am 27.02.2013 08:50, schrieb raulgrande83:
> I'm now having a different problem. In my master-repeater-2slaves
> architecture I have these generations versions:
> Master: 29147
> Repeater: 29147
> Slaves: 29037
> 
> When I go to slaves logs it shows "Slave in sync with master". That is
> apparently because if I do
> http://localhost:17045/solr/replication?command=indexversion (my repeater's
> replication URL) the response is:
> <long name="generation">29037</long>
> 
> Why this URL is returning an old index version? Any solutions to this?
> 
> 
> 
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4043314.html
> Sent from the Solr - User mailing list archive at Nabble.com.
> 

Re: Slaves always replicate entire index & Index versions

Posted by raulgrande83 <ra...@hotmail.com>.
I'm now having a different problem. In my master-repeater-2slaves
architecture I have these generations versions:
Master: 29147
Repeater: 29147
Slaves: 29037

When I go to slaves logs it shows "Slave in sync with master". That is
apparently because if I do
http://localhost:17045/solr/replication?command=indexversion (my repeater's
replication URL) the response is:
<long name="generation">29037</long>

Why this URL is returning an old index version? Any solutions to this?



--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4043314.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slaves always replicate entire index & Index versions

Posted by Mark Miller <ma...@gmail.com>.
On Feb 25, 2013, at 5:54 AM, raulgrande83 <ra...@hotmail.com> wrote:

> Mark, is going to be an official 4.2 release soon?

I've suggested on the dev mailing list that I will create a Lucene/Solr 4.2 release within the next few weeks unless someone beats me to it.

I can't do it this week, I likely can't do it next week, but I should be up for it the week after that.

- Mark

Re: Slaves always replicate entire index & Index versions

Posted by raulgrande83 <ra...@hotmail.com>.
Hello everybody.

I have downloaded the 4.2-SNAPSHOT version that Mark linked at the JIRA and
our first tests have been OK. Slaves now doesn't need to replicate the
entire index and index versions between nodes are the same when replication
process is completed.

This 4.2 version is here:
https://issues.apache.org/jira/browse/SOLR-4471?focusedCommentId=13584470&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13584470

Mark, is going to be an official 4.2 release soon?

Thanks all for your help!!



--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4042690.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slaves always replicate entire index & Index versions

Posted by Artyom <ic...@mail.ru>.
Interesting, that there is no such a bug if I disable index compression,
discussed here:
https://issues.apache.org/jira/browse/SOLR-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566364#comment-13566364



--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4042967.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Slaves always replicate entire index & Index versions

Posted by Mark Miller <ma...@gmail.com>.
We are fixing this bug here: https://issues.apache.org/jira/browse/SOLR-4471

- Mark


On Feb 22, 2013, at 7:07 AM, Artyom <ic...@mail.ru> wrote:

> I have the same problem. This bug appeared in 4.0 rarely, but 4.1 downloads
> the full index every time.
> 


Re: Slaves always replicate entire index & Index versions

Posted by Artyom <ic...@mail.ru>.
I have the same problem. This bug appeared in 4.0 rarely, but 4.1 downloads
the full index every time.



--
View this message in context: http://lucene.472066.n3.nabble.com/Slaves-always-replicate-entire-index-Index-versions-tp4041256p4042209.html
Sent from the Solr - User mailing list archive at Nabble.com.