You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Petersen, Robert" <ro...@mail.rakuten.com> on 2013/07/09 22:22:15 UTC
replication getting stuck on a file
Hi
My solr 3.6.1 slave farm is suddenly getting stuck during replication. It seems to stop on a random file on various slaves (not all) and not continue. I've tried stoping and restarting tomcat etc but some slaves just can't get the index pulled down. Note there is plenty of space on the hard drive. I don't get it. Everything else seems fine. Does this ring a bell for anyone? I have the slaves set for five minute polling intervals.
Here is what I see in admin page, it just stays on that one file and won't get past it while the speed steadily averages down to 0kbs:
Master http://ssbuyma01:8983/solr/10000/replication
Latest Index Version:null, Generation: null
Replicatable Index Version:1276893670111, Generation: 127205
Poll Interval 00:05:00
Local Index Index Version: 1276893670084, Generation: 127202
Location: /var/LucidWorks/lucidworks/solr/10000/data/index
Size: 23.06 GB
Times Replicated Since Startup: 48903
Previous Replication Done At: Tue Jul 09 12:55:01 EDT 2013
Config Files Replicated At: null
Config Files Replicated: null
Times Config Files Replicated Since Startup: null
Next Replication Cycle At: Tue Jul 09 13:00:00 EDT 2013
Current Replication Status Start Time: Tue Jul 09 12:55:00 EDT 2013
Files Downloaded: 59 / 486
Downloaded: 88.73 MB / 23.06 GB [0.0%]
Downloading File: _34mt.fnm, Downloaded: 1.35 MB / 1.35 MB [100.0%]
Time Elapsed: 691s, Estimated Time Remaining: 183204s, Speed: 131.49 KB/s
Robert (Robi) Petersen
Senior Software Engineer
Search Department
RE: replication getting stuck on a file
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
I have seen this happen before in our 3.6.1 deployment. It seemed related to high JVM memory consumption on the server when our index got too big (ie we were close to getting OOMs). That is probably why restarting solr sort of fixes it, assuming the file it is stuck on is the final file and it got 100% of it.
Thanks
Robi
-----Original Message-----
From: Rohit Harchandani [mailto:rharchu@gmail.com]
Sent: Thursday, August 01, 2013 1:55 PM
To: solr-user@lucene.apache.org
Subject: Re: replication getting stuck on a file
I am facing this problem in solr 4.0 too. Its definitely not related to autowarming. It just gets stuck while downloading a file and there is no way to abort the replication except restarting solr.
On Wed, Jul 10, 2013 at 6:10 PM, adityab <ad...@yahoo.com> wrote:
> I have seen this in 4.2.1 too.
> Once replication is finished, on Admin UI we see 100% and time and
> dlspeed information goes out of wack Same is reflected in mbeans. But
> whats actually happening in the background is auto-warmup of caches
> (in my case) May be some minor stats bug
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/replication-getting-stuck-on-a-file
> -tp4076707p4077112.html Sent from the Solr - User mailing list archive
> at Nabble.com.
>
Re: replication getting stuck on a file
Posted by Rohit Harchandani <rh...@gmail.com>.
I am facing this problem in solr 4.0 too. Its definitely not related to
autowarming. It just gets stuck while downloading a file and there is no
way to abort the replication except restarting solr.
On Wed, Jul 10, 2013 at 6:10 PM, adityab <ad...@yahoo.com> wrote:
> I have seen this in 4.2.1 too.
> Once replication is finished, on Admin UI we see 100% and time and dlspeed
> information goes out of wack Same is reflected in mbeans. But whats
> actually
> happening in the background is auto-warmup of caches (in my case)
> May be some minor stats bug
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/replication-getting-stuck-on-a-file-tp4076707p4077112.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: replication getting stuck on a file
Posted by adityab <ad...@yahoo.com>.
I have seen this in 4.2.1 too.
Once replication is finished, on Admin UI we see 100% and time and dlspeed
information goes out of wack Same is reflected in mbeans. But whats actually
happening in the background is auto-warmup of caches (in my case)
May be some minor stats bug
--
View this message in context: http://lucene.472066.n3.nabble.com/replication-getting-stuck-on-a-file-tp4076707p4077112.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: replication getting stuck on a file
Posted by Erick Erickson <er...@gmail.com>.
Hmmm, that is kind of funny. I know this is ugly, but what happens if you
1> stop the slave
2> completely delete the data/index directory (directory too, not just contents)
3> fire it back up?
inelegant at best, but if it cures your problem....
Erick
On Tue, Jul 9, 2013 at 5:57 PM, Petersen, Robert
<ro...@mail.rakuten.com> wrote:
> Look at the speed and time remaining on this one, pretty funny:
>
>
> Master http://ssbuyma01:8983/solr/10000/replication
> Latest Index Version:null, Generation: null
> Replicatable Index Version:1276893670202, Generation: 127213
> Poll Interval 00:05:00
> Local Index Index Version: 1276893670108, Generation: 127204
> Location: /var/LucidWorks/lucidworks/solr/10000/data/index
> Size: 23.13 GB
> Times Replicated Since Startup: 48874
> Previous Replication Done At: Tue Jul 09 13:12:05 PDT 2013
> Config Files Replicated At: null
> Config Files Replicated: null
> Times Config Files Replicated Since Startup: null
> Next Replication Cycle At: Tue Jul 09 13:17:04 PDT 2013
> Current Replication Status Start Time: Tue Jul 09 13:12:04 PDT 2013
> Files Downloaded: 10 / 538
> Downloaded: 1.67 MB / 23.13 GB [0.0%]
> Downloading File: _34n2.prx, Downloaded: 140 bytes / 140 bytes [100.0%]
> Time Elapsed: 6203s, Estimated Time Remaining: 88091277s, Speed: 281 bytes/s
>
>
> -----Original Message-----
> From: Petersen, Robert [mailto:robert.petersen@mail.rakuten.com]
> Sent: Tuesday, July 09, 2013 1:22 PM
> To: solr-user@lucene.apache.org
> Subject: replication getting stuck on a file
>
> Hi
>
> My solr 3.6.1 slave farm is suddenly getting stuck during replication. It seems to stop on a random file on various slaves (not all) and not continue. I've tried stoping and restarting tomcat etc but some slaves just can't get the index pulled down. Note there is plenty of space on the hard drive. I don't get it. Everything else seems fine. Does this ring a bell for anyone? I have the slaves set for five minute polling intervals.
>
> Here is what I see in admin page, it just stays on that one file and won't get past it while the speed steadily averages down to 0kbs:
>
> Master http://ssbuyma01:8983/solr/10000/replication
> Latest Index Version:null, Generation: null Replicatable Index Version:1276893670111, Generation: 127205
> Poll Interval 00:05:00
> Local Index Index Version: 1276893670084, Generation: 127202
> Location: /var/LucidWorks/lucidworks/solr/10000/data/index
> Size: 23.06 GB
> Times Replicated Since Startup: 48903
> Previous Replication Done At: Tue Jul 09 12:55:01 EDT 2013 Config Files Replicated At: null Config Files Replicated: null Times Config Files Replicated Since Startup: null Next Replication Cycle At: Tue Jul 09 13:00:00 EDT 2013
> Current Replication Status Start Time: Tue Jul 09 12:55:00 EDT 2013
> Files Downloaded: 59 / 486
> Downloaded: 88.73 MB / 23.06 GB [0.0%]
> Downloading File: _34mt.fnm, Downloaded: 1.35 MB / 1.35 MB [100.0%] Time Elapsed: 691s, Estimated Time Remaining: 183204s, Speed: 131.49 KB/s
>
>
> Robert (Robi) Petersen
> Senior Software Engineer
> Search Department
>
>
>
>
>
>
>
>
>
RE: replication getting stuck on a file
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
Look at the speed and time remaining on this one, pretty funny:
Master http://ssbuyma01:8983/solr/10000/replication
Latest Index Version:null, Generation: null
Replicatable Index Version:1276893670202, Generation: 127213
Poll Interval 00:05:00
Local Index Index Version: 1276893670108, Generation: 127204
Location: /var/LucidWorks/lucidworks/solr/10000/data/index
Size: 23.13 GB
Times Replicated Since Startup: 48874
Previous Replication Done At: Tue Jul 09 13:12:05 PDT 2013
Config Files Replicated At: null
Config Files Replicated: null
Times Config Files Replicated Since Startup: null
Next Replication Cycle At: Tue Jul 09 13:17:04 PDT 2013
Current Replication Status Start Time: Tue Jul 09 13:12:04 PDT 2013
Files Downloaded: 10 / 538
Downloaded: 1.67 MB / 23.13 GB [0.0%]
Downloading File: _34n2.prx, Downloaded: 140 bytes / 140 bytes [100.0%]
Time Elapsed: 6203s, Estimated Time Remaining: 88091277s, Speed: 281 bytes/s
-----Original Message-----
From: Petersen, Robert [mailto:robert.petersen@mail.rakuten.com]
Sent: Tuesday, July 09, 2013 1:22 PM
To: solr-user@lucene.apache.org
Subject: replication getting stuck on a file
Hi
My solr 3.6.1 slave farm is suddenly getting stuck during replication. It seems to stop on a random file on various slaves (not all) and not continue. I've tried stoping and restarting tomcat etc but some slaves just can't get the index pulled down. Note there is plenty of space on the hard drive. I don't get it. Everything else seems fine. Does this ring a bell for anyone? I have the slaves set for five minute polling intervals.
Here is what I see in admin page, it just stays on that one file and won't get past it while the speed steadily averages down to 0kbs:
Master http://ssbuyma01:8983/solr/10000/replication
Latest Index Version:null, Generation: null Replicatable Index Version:1276893670111, Generation: 127205
Poll Interval 00:05:00
Local Index Index Version: 1276893670084, Generation: 127202
Location: /var/LucidWorks/lucidworks/solr/10000/data/index
Size: 23.06 GB
Times Replicated Since Startup: 48903
Previous Replication Done At: Tue Jul 09 12:55:01 EDT 2013 Config Files Replicated At: null Config Files Replicated: null Times Config Files Replicated Since Startup: null Next Replication Cycle At: Tue Jul 09 13:00:00 EDT 2013
Current Replication Status Start Time: Tue Jul 09 12:55:00 EDT 2013
Files Downloaded: 59 / 486
Downloaded: 88.73 MB / 23.06 GB [0.0%]
Downloading File: _34mt.fnm, Downloaded: 1.35 MB / 1.35 MB [100.0%] Time Elapsed: 691s, Estimated Time Remaining: 183204s, Speed: 131.49 KB/s
Robert (Robi) Petersen
Senior Software Engineer
Search Department