Data from deleted from Solr (Solr cloud)

[shreejay <sh...@gmail.com>]

Hi All, 

I have a solrlcoud instance with 3 shards. Each shard has 2 instance (2
servers each running a instance of solr)

Lets say I had Instance1 and instance2 in shard1 … At some point, instance2
went down due to OOM (out of memory) . instance1 for some reason was not
replicating the data properly and when it became the leader, it had only
around 1% of the data that instance2 had. I restarted instance2, and hoped
that instance1 will replicate from 2, but instead instanace2 replicated from
instance1 . and ended up deleting the original index folder it had. There
were around 2 million documents in that instance.

Can any one of solrlcoud users give any hints if I can recover this data?




--Shreejay



--
View this message in context: http://lucene.472066.n3.nabble.com/Data-from-deleted-from-Solr-Solr-cloud-tp4028055.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Data from deleted from Solr (Solr cloud)

[shreejay <sh...@gmail.com>]

Thanks Mark. Unfortunately I only took log file backup of the Solr instance
which failed. 

Even if we stopped and started all instances in a shard at same time, isn't
there a possibility that the Solr Instance with less data (or no data) might
declare itself leader and the one with the data will start copying the empty
index folder from the leader? 

We tried using the latest branch_4x few weeks back and hit the  SOLR-4144
<https://issues.apache.org/jira/browse/SOLR-4144>  issue. Although it was
fixed in a day, we decided to use the released version only (4.0) since we
cannot have such extended periods of outage. 

We are looking forward to 4.1 and will start using it as soon as its
released. 

--Shreejay



--
View this message in context: http://lucene.472066.n3.nabble.com/Data-from-deleted-from-Solr-Solr-cloud-tp4028055p4029212.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Data from deleted from Solr (Solr cloud)

[Mark Miller <ma...@gmail.com>]

Hmm…have not seen a report of this before.

For the future, FWIW, if you are in a bad state, the best thing to do is stop all the replicas in the shard and then start it again. That way everyone will work together to trade data and determine the best leader.

When you restarted instance 2, instance 1 was already the leader - so 2 just replicates from him. If you stopped the replicas and started them, that is when the best leader is chosen.

I'd also encourage you to try out a nightly build if you can - would be good to be sure whatever this problem is is fixed.

It would be interesting if you could post the logs of the node that became leader without all of the docs.

- Mark

On Dec 19, 2012, at 11:08 AM, shreejay <sh...@gmail.com> wrote:

> Hi All, 
> 
> I have a solrlcoud instance with 3 shards. Each shard has 2 instance (2
> servers each running a instance of solr)
> 
> Lets say I had Instance1 and instance2 in shard1 … At some point, instance2
> went down due to OOM (out of memory) . instance1 for some reason was not
> replicating the data properly and when it became the leader, it had only
> around 1% of the data that instance2 had. I restarted instance2, and hoped
> that instance1 will replicate from 2, but instead instanace2 replicated from
> instance1 . and ended up deleting the original index folder it had. There
> were around 2 million documents in that instance.
> 
> Can any one of solrlcoud users give any hints if I can recover this data?
> 
> 
> 
> 
> --Shreejay
> 
> 
> 
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Data-from-deleted-from-Solr-Solr-cloud-tp4028055.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Data from deleted from Solr (Solr cloud)

[Erick Erickson <er...@gmail.com>]

Note that you don't have to re-compile, the latest successful build is
always available, you can find the from here:
http://wiki.apache.org/solr/NightlyBuilds

FWIW,
Erick


On Thu, Dec 20, 2012 at 3:36 AM, shreejay <sh...@gmail.com> wrote:

> Thanks John.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Data-from-deleted-from-Solr-Solr-cloud-tp4028055p4028252.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Data from deleted from Solr (Solr cloud)

[shreejay <sh...@gmail.com>]

Thanks John. 



--
View this message in context: http://lucene.472066.n3.nabble.com/Data-from-deleted-from-Solr-Solr-cloud-tp4028055p4028252.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Data from deleted from Solr (Solr cloud)

[John Nielsen <jn...@mcb.dk>]

Yeah, I ran into this issue myself with solr-4.0.0.

To fix it, I had to compile my own version from the solr-4x branch. That
is, I assume it's fixed as I have been unable to replicate it after the
switch.

I'm afraid you will have to reindex your data.


-- 
Med venlig hilsen / Best regards

*John Nielsen*
Programmer



*MCB A/S*
Enghaven 15
DK-7500 Holstebro

Kundeservice: +45 9610 2824
post@mcb.dk
www.mcb.dk


On Wed, Dec 19, 2012 at 5:08 PM, shreejay <sh...@gmail.com> wrote:

> Hi All,
>
> I have a solrlcoud instance with 3 shards. Each shard has 2 instance (2
> servers each running a instance of solr)
>
> Lets say I had Instance1 and instance2 in shard1 … At some point, instance2
> went down due to OOM (out of memory) . instance1 for some reason was not
> replicating the data properly and when it became the leader, it had only
> around 1% of the data that instance2 had. I restarted instance2, and hoped
> that instance1 will replicate from 2, but instead instanace2 replicated
> from
> instance1 . and ended up deleting the original index folder it had. There
> were around 2 million documents in that instance.
>
> Can any one of solrlcoud users give any hints if I can recover this data?
>
>
>
>
> --Shreejay
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Data-from-deleted-from-Solr-Solr-cloud-tp4028055.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>