Re: solrcloud shards backup/restoration

[adfel70 <ad...@gmail.com>]

did you solve this eventually?


Aditya Sakhuja wrote
> How does one recover from an index corruption ? That's what I am trying to
> eventually tackle here.
> 
> Thanks
> Aditya
> 
> On Thursday, September 19, 2013, Aditya Sakhuja wrote:
> 
>> Hi,
>>
>> Sorry for the late followup on this. Let me put in more details here.
>>
>> *The problem:*
>>
>> Cannot successfully restore back the index backed up with
>> '/replication?command=backup'. The backup was generated as *
>> snapshot.yyyymmdd*
>>
>> *My setup and steps:*
>> *
>> *
>> 6 solrcloud instances
>> 7 zookeepers instances
>>
>> Steps:
>>
>> 1.> Take snapshot using
>> *http://host1:8893/solr/replication?command=backup
>> *, on one host only. move *snapshot.yyyymmdd *to some reliable storage.
>>
>> 2.> Stop all 6 solr instances, all 7 zk instances.
>>
>> 3.> Delete ../collectionname/data/* on all solrcloud nodes. ie. deleting
>> the index data completely.
>>
>> 4.> Delete zookeeper/data/version*/* on all zookeeper nodes.
>>
>> 5.> Copy back index from backup to one of the nodes.
>>      \> cp *snapshot.yyyymmdd/*  *../collectionname/data/index/*
>>
>> 6.> Restart all zk instances. Restart all solrcloud instances.
>>
>>
>> *Outcome:*
>> *
>> *
>> All solr instances are up. However, *num of docs = 0 *for all nodes.
>> Looking at the node where the index was restored, there is a new
>> index.yymmddhhmmss directory being created and index.properties pointing
>> to
>> it. That explains why no documents are reported.
>>
>>
>> How do I have solrcloud pickup data from the index directory on a restart
>> ?
>>
>> Thanks in advance,
>> Aditya
>>
>>
>>
>> On Fri, Sep 6, 2013 at 3:41 PM, Aditya Sakhuja <

> aditya.sakhuja@

> >wrote:
>>
>> Thanks Shalin and Mark for your responses. I am on the same page about
>> the
>> conventions for taking the backup. However, I am less sure about the
>> restoration of the index. Lets say we have 3 shards across 3 solrcloud
>> servers.
>>
>> 1.> I am assuming we should take a backup from each of the shard leaders
>> to get a complete collection. do you think that will get the complete
>> index
>> ( not worrying about what is not hard committed at the time of backup ).
>> ?
>>
>> 2.> How do we go about restoring the index in a fresh solrcloud cluster ?
>> From the structure of the snapshot I took, I did not see any
>> replication.properties or index.properties  which I see normally on a
>> healthy solrcloud cluster nodes.
>> if I have the snapshot named snapshot.20130905 does the
>> snapshot.20130905/* go into data/index ?
>>
>> Thanks
>> Aditya
>>
>>
>>
>> On Fri, Sep 6, 2013 at 7:28 AM, Mark Miller <

> markrmiller@

> > wrote:
>>
>> Phone typing. The end should not say "don't hard commit" - it should say
>> "do a hard commit and take a snapshot".
>>
>> Mark
>>
>> Sent from my iPhone
>>
>> On Sep 6, 2013, at 7:26 AM, Mark Miller <

> markrmiller@

> > wrote:
>>
>> > I don't know that it's too bad though - its always been the case that
>> if
>> you do a backup while indexing, it's just going to get up to the last
>> hard
>> commit. With SolrCloud that will still be the case. So just make sure you
>> do a hard commit right before taking the backup - yes, it might miss a
>> few
>> docs in the tran log, but if you are taking a back up while indexing, you
>> don't have great precision in any case - you will roughly get a snapshot
>> for around that time - even without SolrCloud, if you are worried about
>> precision and getting every update into that backup, you want to stop
>> indexing and commit first. But if you just want a rough snapshot for
>> around
>> that time, in both cases you can still just don't hard commit and take a
>> snapshot.
>> >
>> > Mark
>> >
>> > Sent from my iPhone
>> >
>> > On Sep 6, 2013, at 1:13 AM, Shalin Shekhar Mangar <
>> 

> shalinmangar@

>> wrote:
>> >
>> >> The replication handler's backup command was built for pre-SolrCloud.
>> >> It takes a snapshot of the index but it is unaware of the transaction
>> >> log which is a key component in SolrCloud. Hence unless you stop
>> >> updates, commit your changes and then take a backup, you will likely
>> >> miss some updates.
>> >>
>> >> That being said, I'm curious to see how peer sync behaves when you try
>> >> to restore from a snapshot. When you say that you haven't been
>> >> successful in restoring, what exactly is the behaviour you observed?
>> >>
>> >> On Fri, Sep 6, 2013 at 5:14 AM, Aditya Sakhuja <
>> 

> aditya.sakhuja@

>> wrote:
>> >>> Hello,
>> >>>
>> >>> I was looking for a good backup / recovery solution for the solrcloud
>> >>> indexes. I am more looking for restoring the indexes from the index
>> >>> snapshot, which can be taken using the replicationHandler's backup
>> command.
>> >>>
>> >>> I am looking for something that works with solrcloud 4.3 eventually,
>> but
>> >>> still relevant if you tested with a previous version.
>> >>>
>> >>> I haven't been successful in have the restored index replicate across
>> the
>> >>> new replicas, after I restart all the nodes, with one node having the
>> >>> restored index.
>> >>>
>> >>> Is restoring the indexes on all the nodes the best way to do it ?
>> >>> --
>> >>> Regards,
>> >>> -Aditya Sakhuja
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Shalin Shekhar Mangar.
>>
>>
>>
>>
>> --
>> Regards,
>> -Aditya Sakhuja
>>
>> --
>> Regards,
>> -Aditya Sakhuja
>>
> 
> 
> -- 
> Regards,
> -Aditya Sakhuja





--
View this message in context: http://lucene.472066.n3.nabble.com/solrcloud-shards-backup-restoration-tp4088447p4099789.html
Sent from the Solr - User mailing list archive at Nabble.com.