You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nathaniel Domingo <ni...@gmail.com> on 2012/12/06 03:48:30 UTC
solr cloud node stuck at recovering for days
Hi,
I'm very new to solr, less than a month. I just set-up a solrcloud cluster
last week and has encountered a problem. i have four nodes with two shards.
one node is stuck at recovering for days now. how do i go about fixing this?
Thanks
Re: solr cloud node stuck at recovering for days
Posted by Mark Miller <ma...@gmail.com>.
Looks like the connecting to ZooKeeper is "flapping". So as it tries to recover it keeps losing the connection to zookeeper and then trying again, and I don't have enough of the log to tell, but that probably just repeats and repeats.
I guess the network is probably not so fast and or the load and network are not working out so well…
Try raising the zkClientTimeout - see the FAQ: http://wiki.apache.org/solr/SolrCloud#FAQ
It defaults to like 15 seconds. You might try 30 or 40 seconds. If that doesn't help, you have to figure out why Solr is having so much trouble communicating with ZooKeeper.
Are you using an external ensemble, a single ZooKeeper node, embedded ZooKeeper?
- Mark
On Dec 5, 2012, at 7:59 PM, Nathaniel Domingo <ni...@gmail.com> wrote:
> here's a link to a portion of the log in pastebin.
>
> http://pastebin.com/UDBMDdMv
>
> Thanks
>
>
> On Thu, Dec 6, 2012 at 11:53 AM, Mark Miller <ma...@gmail.com> wrote:
>
>> I think the list strips most attachments or something - can you try
>> something like pastebin.com?
>>
>> Thanks,
>>
>> mark
>>
>> On Dec 5, 2012, at 7:46 PM, Nathaniel Domingo <ni...@gmail.com>
>> wrote:
>>
>>> attached is a log relevant to the recovery.
>>>
>>> Thanks
>>>
>>>
>>> On Thu, Dec 6, 2012 at 11:23 AM, Mark Miller <ma...@gmail.com>
>> wrote:
>>> Okay - logs from that node would help a lot then (or just the parts
>> around when it's trying to recover).
>>>
>>> - Mark
>>>
>>> On Dec 5, 2012, at 7:11 PM, Nathaniel Domingo <ni...@gmail.com>
>> wrote:
>>>
>>>> yes, i tried restarting the node twice already and both times it just
>> got
>>>> stucked in recovering. one node also had some problems a few days ago,
>>>> after a restart, it eventually moved from recovering to active after an
>>>> hour. i'm using solr 4.0.0.
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Thu, Dec 6, 2012 at 11:03 AM, Mark Miller <ma...@gmail.com>
>> wrote:
>>>>
>>>>> Did you try restarting that node? Have you seen a successful recovery
>>>>> before? What exact version are you using?
>>>>>
>>>>> Can you share any related info in the logs for that node?
>>>>>
>>>>> - Mark
>>>>>
>>>>> On Dec 5, 2012, at 6:48 PM, Nathaniel Domingo <niel.domingo@gmail.com
>>>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm very new to solr, less than a month. I just set-up a solrcloud
>>>>> cluster
>>>>>> last week and has encountered a problem. i have four nodes with two
>>>>> shards.
>>>>>> one node is stuck at recovering for days now. how do i go about
>> fixing
>>>>> this?
>>>>>>
>>>>>> Thanks
>>>>>
>>>>>
>>>
>>>
>>
>>
Re: solr cloud node stuck at recovering for days
Posted by Nathaniel Domingo <ni...@gmail.com>.
here's a link to a portion of the log in pastebin.
http://pastebin.com/UDBMDdMv
Thanks
On Thu, Dec 6, 2012 at 11:53 AM, Mark Miller <ma...@gmail.com> wrote:
> I think the list strips most attachments or something - can you try
> something like pastebin.com?
>
> Thanks,
>
> mark
>
> On Dec 5, 2012, at 7:46 PM, Nathaniel Domingo <ni...@gmail.com>
> wrote:
>
> > attached is a log relevant to the recovery.
> >
> > Thanks
> >
> >
> > On Thu, Dec 6, 2012 at 11:23 AM, Mark Miller <ma...@gmail.com>
> wrote:
> > Okay - logs from that node would help a lot then (or just the parts
> around when it's trying to recover).
> >
> > - Mark
> >
> > On Dec 5, 2012, at 7:11 PM, Nathaniel Domingo <ni...@gmail.com>
> wrote:
> >
> > > yes, i tried restarting the node twice already and both times it just
> got
> > > stucked in recovering. one node also had some problems a few days ago,
> > > after a restart, it eventually moved from recovering to active after an
> > > hour. i'm using solr 4.0.0.
> > >
> > > Thanks
> > >
> > >
> > > On Thu, Dec 6, 2012 at 11:03 AM, Mark Miller <ma...@gmail.com>
> wrote:
> > >
> > >> Did you try restarting that node? Have you seen a successful recovery
> > >> before? What exact version are you using?
> > >>
> > >> Can you share any related info in the logs for that node?
> > >>
> > >> - Mark
> > >>
> > >> On Dec 5, 2012, at 6:48 PM, Nathaniel Domingo <niel.domingo@gmail.com
> >
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> I'm very new to solr, less than a month. I just set-up a solrcloud
> > >> cluster
> > >>> last week and has encountered a problem. i have four nodes with two
> > >> shards.
> > >>> one node is stuck at recovering for days now. how do i go about
> fixing
> > >> this?
> > >>>
> > >>> Thanks
> > >>
> > >>
> >
> >
>
>
Re: solr cloud node stuck at recovering for days
Posted by Mark Miller <ma...@gmail.com>.
I think the list strips most attachments or something - can you try something like pastebin.com?
Thanks,
mark
On Dec 5, 2012, at 7:46 PM, Nathaniel Domingo <ni...@gmail.com> wrote:
> attached is a log relevant to the recovery.
>
> Thanks
>
>
> On Thu, Dec 6, 2012 at 11:23 AM, Mark Miller <ma...@gmail.com> wrote:
> Okay - logs from that node would help a lot then (or just the parts around when it's trying to recover).
>
> - Mark
>
> On Dec 5, 2012, at 7:11 PM, Nathaniel Domingo <ni...@gmail.com> wrote:
>
> > yes, i tried restarting the node twice already and both times it just got
> > stucked in recovering. one node also had some problems a few days ago,
> > after a restart, it eventually moved from recovering to active after an
> > hour. i'm using solr 4.0.0.
> >
> > Thanks
> >
> >
> > On Thu, Dec 6, 2012 at 11:03 AM, Mark Miller <ma...@gmail.com> wrote:
> >
> >> Did you try restarting that node? Have you seen a successful recovery
> >> before? What exact version are you using?
> >>
> >> Can you share any related info in the logs for that node?
> >>
> >> - Mark
> >>
> >> On Dec 5, 2012, at 6:48 PM, Nathaniel Domingo <ni...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I'm very new to solr, less than a month. I just set-up a solrcloud
> >> cluster
> >>> last week and has encountered a problem. i have four nodes with two
> >> shards.
> >>> one node is stuck at recovering for days now. how do i go about fixing
> >> this?
> >>>
> >>> Thanks
> >>
> >>
>
>
Re: solr cloud node stuck at recovering for days
Posted by Nathaniel Domingo <ni...@gmail.com>.
attached is a log relevant to the recovery.
Thanks
On Thu, Dec 6, 2012 at 11:23 AM, Mark Miller <ma...@gmail.com> wrote:
> Okay - logs from that node would help a lot then (or just the parts around
> when it's trying to recover).
>
> - Mark
>
> On Dec 5, 2012, at 7:11 PM, Nathaniel Domingo <ni...@gmail.com>
> wrote:
>
> > yes, i tried restarting the node twice already and both times it just got
> > stucked in recovering. one node also had some problems a few days ago,
> > after a restart, it eventually moved from recovering to active after an
> > hour. i'm using solr 4.0.0.
> >
> > Thanks
> >
> >
> > On Thu, Dec 6, 2012 at 11:03 AM, Mark Miller <ma...@gmail.com>
> wrote:
> >
> >> Did you try restarting that node? Have you seen a successful recovery
> >> before? What exact version are you using?
> >>
> >> Can you share any related info in the logs for that node?
> >>
> >> - Mark
> >>
> >> On Dec 5, 2012, at 6:48 PM, Nathaniel Domingo <ni...@gmail.com>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I'm very new to solr, less than a month. I just set-up a solrcloud
> >> cluster
> >>> last week and has encountered a problem. i have four nodes with two
> >> shards.
> >>> one node is stuck at recovering for days now. how do i go about fixing
> >> this?
> >>>
> >>> Thanks
> >>
> >>
>
>
Re: solr cloud node stuck at recovering for days
Posted by Mark Miller <ma...@gmail.com>.
Okay - logs from that node would help a lot then (or just the parts around when it's trying to recover).
- Mark
On Dec 5, 2012, at 7:11 PM, Nathaniel Domingo <ni...@gmail.com> wrote:
> yes, i tried restarting the node twice already and both times it just got
> stucked in recovering. one node also had some problems a few days ago,
> after a restart, it eventually moved from recovering to active after an
> hour. i'm using solr 4.0.0.
>
> Thanks
>
>
> On Thu, Dec 6, 2012 at 11:03 AM, Mark Miller <ma...@gmail.com> wrote:
>
>> Did you try restarting that node? Have you seen a successful recovery
>> before? What exact version are you using?
>>
>> Can you share any related info in the logs for that node?
>>
>> - Mark
>>
>> On Dec 5, 2012, at 6:48 PM, Nathaniel Domingo <ni...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I'm very new to solr, less than a month. I just set-up a solrcloud
>> cluster
>>> last week and has encountered a problem. i have four nodes with two
>> shards.
>>> one node is stuck at recovering for days now. how do i go about fixing
>> this?
>>>
>>> Thanks
>>
>>
Re: solr cloud node stuck at recovering for days
Posted by Nathaniel Domingo <ni...@gmail.com>.
yes, i tried restarting the node twice already and both times it just got
stucked in recovering. one node also had some problems a few days ago,
after a restart, it eventually moved from recovering to active after an
hour. i'm using solr 4.0.0.
Thanks
On Thu, Dec 6, 2012 at 11:03 AM, Mark Miller <ma...@gmail.com> wrote:
> Did you try restarting that node? Have you seen a successful recovery
> before? What exact version are you using?
>
> Can you share any related info in the logs for that node?
>
> - Mark
>
> On Dec 5, 2012, at 6:48 PM, Nathaniel Domingo <ni...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'm very new to solr, less than a month. I just set-up a solrcloud
> cluster
> > last week and has encountered a problem. i have four nodes with two
> shards.
> > one node is stuck at recovering for days now. how do i go about fixing
> this?
> >
> > Thanks
>
>
Re: solr cloud node stuck at recovering for days
Posted by Mark Miller <ma...@gmail.com>.
Did you try restarting that node? Have you seen a successful recovery before? What exact version are you using?
Can you share any related info in the logs for that node?
- Mark
On Dec 5, 2012, at 6:48 PM, Nathaniel Domingo <ni...@gmail.com> wrote:
> Hi,
>
> I'm very new to solr, less than a month. I just set-up a solrcloud cluster
> last week and has encountered a problem. i have four nodes with two shards.
> one node is stuck at recovering for days now. how do i go about fixing this?
>
> Thanks