You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Furkan KAMACI <fu...@gmail.com> on 2013/04/30 14:06:54 UTC
How to Recovery Backup Snapshot at SolrCloud?
Hi Folks;
I can backup my indexes at SolrCloud via
http://_master_host_:_port_/solr/replication?command=backup
and it creates a file called snapshot. I know that I should pull that
directory any other safe place (a backup store) However what should I do to
make a recovery from that backup file?
Re: How to Recovery Backup Snapshot at SolrCloud?
Posted by Furkan KAMACI <fu...@gmail.com>.
So does backup command is used for just to get more consistent index folder
compared to just using index folder for backup?
2013/5/1 Michael Della Bitta <mi...@appinions.com>
> Yeah, it's a consistency problem. Copying all those files takes time, and
> without something with some knowledge of how Lucene works managing the
> atomicity of the work, you might end up with a segments file that doesn't
> match the segments you actually copied.
>
>
> Michael Della Bitta
>
> ------------------------------------------------
> Appinions
> 18 East 41st Street, 2nd Floor
> New York, NY 10017-6271
>
> www.appinions.com
>
> Where Influence Isn’t a Game
>
>
> On Wed, May 1, 2013 at 8:59 AM, Furkan KAMACI <furkankamaci@gmail.com
> >wrote:
>
> > Sorry but what will I do? Will I copy everything under snapshot folder
> into
> > under index folder? If I don't run backup command and just copy index
> > folder anywhere else what is the difference between them (is it something
> > like consistency for if any writing operation for segment files did not
> > finished, or does backup command retrieves indexes data at RAM)
> >
> >
> >
> > 2013/5/1 Timothy Potter <th...@gmail.com>
> >
> > > I agree with Michael that you'll only ever need your backup if you
> > > lose all nodes hosting a shard (leader + all other replicas), so the
> > > tlog doesn't really factor in when recovering from backup.
> > >
> > > The snapshot created by the replication handler is the index only and
> > > it makes most sense in my mind to remove the tlog before firing up a
> > > new node using the snapshot.
> > >
> > > The way I see it, the backup would be needed if you lost all nodes
> > > hosting a shard, then you need to 1) recover one node (the leader)
> > > from backup, 2) re-index any documents that were indexed between the
> > > time of your last snapshot and the time of the failure, then 3) bring
> > > additional replicas online as needed. The additional replicas will
> > > sync with the leader, snap pulling the entire index. You could do it
> > > 1,2,3 or 1,3,2.
> > >
> > > Cheers,
> > > Tim
> > >
> > > On Tue, Apr 30, 2013 at 12:45 PM, Furkan KAMACI <
> furkankamaci@gmail.com>
> > > wrote:
> > > > I had index and tlog folder under my data folder. I have a snapshot
> > > folder
> > > > too when I make backup. However what will I do next if I want to use
> > > > backup, will I remove index and tlog folders and put just my snapshot
> > > > folder? What folks do?
> > > >
> > > > 2013/4/30 Michael Della Bitta <mi...@appinions.com>
> > > >
> > > >> Presumably you'd only be restoring a backup in the face of a
> > > catastrophe.
> > > >>
> > > >> Yes, you'd need to stop the node. And the transaction logs may not
> be
> > > >> useful in this case. You'd have trouble reconciling them with the
> > > version
> > > >> of the index in your backup I would think.
> > > >>
> > > >> Anybody who knows more about this want to chime in?
> > > >>
> > > >>
> > > >> Michael Della Bitta
> > > >>
> > > >> ------------------------------------------------
> > > >> Appinions
> > > >> 18 East 41st Street, 2nd Floor
> > > >> New York, NY 10017-6271
> > > >>
> > > >> www.appinions.com
> > > >>
> > > >> Where Influence Isn’t a Game
> > > >>
> > > >>
> > > >> On Tue, Apr 30, 2013 at 11:03 AM, Furkan KAMACI <
> > furkankamaci@gmail.com
> > > >> >wrote:
> > > >>
> > > >> > Should I stop the node first? And what will happen to transaction
> > > logs?
> > > >> > Should I backup it too?
> > > >> >
> > > >> > 2013/4/30 Michael Della Bitta <mi...@appinions.com>
> > > >> >
> > > >> > > That directory is the data directory for the core... you'd just
> > > swap it
> > > >> > in.
> > > >> > >
> > > >> > >
> > > >> > > Michael Della Bitta
> > > >> > >
> > > >> > > ------------------------------------------------
> > > >> > > Appinions
> > > >> > > 18 East 41st Street, 2nd Floor
> > > >> > > New York, NY 10017-6271
> > > >> > >
> > > >> > > www.appinions.com
> > > >> > >
> > > >> > > Where Influence Isn’t a Game
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Apr 30, 2013 at 8:06 AM, Furkan KAMACI <
> > > furkankamaci@gmail.com
> > > >> > > >wrote:
> > > >> > >
> > > >> > > > Hi Folks;
> > > >> > > >
> > > >> > > > I can backup my indexes at SolrCloud via
> > > >> > > > http://_master_host_:_port_/solr/replication?command=backup
> > > >> > > > and it creates a file called snapshot. I know that I should
> pull
> > > that
> > > >> > > > directory any other safe place (a backup store) However what
> > > should I
> > > >> > do
> > > >> > > to
> > > >> > > > make a recovery from that backup file?
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
Re: How to Recovery Backup Snapshot at SolrCloud?
Posted by Michael Della Bitta <mi...@appinions.com>.
Yeah, it's a consistency problem. Copying all those files takes time, and
without something with some knowledge of how Lucene works managing the
atomicity of the work, you might end up with a segments file that doesn't
match the segments you actually copied.
Michael Della Bitta
------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
www.appinions.com
Where Influence Isn’t a Game
On Wed, May 1, 2013 at 8:59 AM, Furkan KAMACI <fu...@gmail.com>wrote:
> Sorry but what will I do? Will I copy everything under snapshot folder into
> under index folder? If I don't run backup command and just copy index
> folder anywhere else what is the difference between them (is it something
> like consistency for if any writing operation for segment files did not
> finished, or does backup command retrieves indexes data at RAM)
>
>
>
> 2013/5/1 Timothy Potter <th...@gmail.com>
>
> > I agree with Michael that you'll only ever need your backup if you
> > lose all nodes hosting a shard (leader + all other replicas), so the
> > tlog doesn't really factor in when recovering from backup.
> >
> > The snapshot created by the replication handler is the index only and
> > it makes most sense in my mind to remove the tlog before firing up a
> > new node using the snapshot.
> >
> > The way I see it, the backup would be needed if you lost all nodes
> > hosting a shard, then you need to 1) recover one node (the leader)
> > from backup, 2) re-index any documents that were indexed between the
> > time of your last snapshot and the time of the failure, then 3) bring
> > additional replicas online as needed. The additional replicas will
> > sync with the leader, snap pulling the entire index. You could do it
> > 1,2,3 or 1,3,2.
> >
> > Cheers,
> > Tim
> >
> > On Tue, Apr 30, 2013 at 12:45 PM, Furkan KAMACI <fu...@gmail.com>
> > wrote:
> > > I had index and tlog folder under my data folder. I have a snapshot
> > folder
> > > too when I make backup. However what will I do next if I want to use
> > > backup, will I remove index and tlog folders and put just my snapshot
> > > folder? What folks do?
> > >
> > > 2013/4/30 Michael Della Bitta <mi...@appinions.com>
> > >
> > >> Presumably you'd only be restoring a backup in the face of a
> > catastrophe.
> > >>
> > >> Yes, you'd need to stop the node. And the transaction logs may not be
> > >> useful in this case. You'd have trouble reconciling them with the
> > version
> > >> of the index in your backup I would think.
> > >>
> > >> Anybody who knows more about this want to chime in?
> > >>
> > >>
> > >> Michael Della Bitta
> > >>
> > >> ------------------------------------------------
> > >> Appinions
> > >> 18 East 41st Street, 2nd Floor
> > >> New York, NY 10017-6271
> > >>
> > >> www.appinions.com
> > >>
> > >> Where Influence Isn’t a Game
> > >>
> > >>
> > >> On Tue, Apr 30, 2013 at 11:03 AM, Furkan KAMACI <
> furkankamaci@gmail.com
> > >> >wrote:
> > >>
> > >> > Should I stop the node first? And what will happen to transaction
> > logs?
> > >> > Should I backup it too?
> > >> >
> > >> > 2013/4/30 Michael Della Bitta <mi...@appinions.com>
> > >> >
> > >> > > That directory is the data directory for the core... you'd just
> > swap it
> > >> > in.
> > >> > >
> > >> > >
> > >> > > Michael Della Bitta
> > >> > >
> > >> > > ------------------------------------------------
> > >> > > Appinions
> > >> > > 18 East 41st Street, 2nd Floor
> > >> > > New York, NY 10017-6271
> > >> > >
> > >> > > www.appinions.com
> > >> > >
> > >> > > Where Influence Isn’t a Game
> > >> > >
> > >> > >
> > >> > > On Tue, Apr 30, 2013 at 8:06 AM, Furkan KAMACI <
> > furkankamaci@gmail.com
> > >> > > >wrote:
> > >> > >
> > >> > > > Hi Folks;
> > >> > > >
> > >> > > > I can backup my indexes at SolrCloud via
> > >> > > > http://_master_host_:_port_/solr/replication?command=backup
> > >> > > > and it creates a file called snapshot. I know that I should pull
> > that
> > >> > > > directory any other safe place (a backup store) However what
> > should I
> > >> > do
> > >> > > to
> > >> > > > make a recovery from that backup file?
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
Re: How to Recovery Backup Snapshot at SolrCloud?
Posted by Furkan KAMACI <fu...@gmail.com>.
Sorry but what will I do? Will I copy everything under snapshot folder into
under index folder? If I don't run backup command and just copy index
folder anywhere else what is the difference between them (is it something
like consistency for if any writing operation for segment files did not
finished, or does backup command retrieves indexes data at RAM)
2013/5/1 Timothy Potter <th...@gmail.com>
> I agree with Michael that you'll only ever need your backup if you
> lose all nodes hosting a shard (leader + all other replicas), so the
> tlog doesn't really factor in when recovering from backup.
>
> The snapshot created by the replication handler is the index only and
> it makes most sense in my mind to remove the tlog before firing up a
> new node using the snapshot.
>
> The way I see it, the backup would be needed if you lost all nodes
> hosting a shard, then you need to 1) recover one node (the leader)
> from backup, 2) re-index any documents that were indexed between the
> time of your last snapshot and the time of the failure, then 3) bring
> additional replicas online as needed. The additional replicas will
> sync with the leader, snap pulling the entire index. You could do it
> 1,2,3 or 1,3,2.
>
> Cheers,
> Tim
>
> On Tue, Apr 30, 2013 at 12:45 PM, Furkan KAMACI <fu...@gmail.com>
> wrote:
> > I had index and tlog folder under my data folder. I have a snapshot
> folder
> > too when I make backup. However what will I do next if I want to use
> > backup, will I remove index and tlog folders and put just my snapshot
> > folder? What folks do?
> >
> > 2013/4/30 Michael Della Bitta <mi...@appinions.com>
> >
> >> Presumably you'd only be restoring a backup in the face of a
> catastrophe.
> >>
> >> Yes, you'd need to stop the node. And the transaction logs may not be
> >> useful in this case. You'd have trouble reconciling them with the
> version
> >> of the index in your backup I would think.
> >>
> >> Anybody who knows more about this want to chime in?
> >>
> >>
> >> Michael Della Bitta
> >>
> >> ------------------------------------------------
> >> Appinions
> >> 18 East 41st Street, 2nd Floor
> >> New York, NY 10017-6271
> >>
> >> www.appinions.com
> >>
> >> Where Influence Isn’t a Game
> >>
> >>
> >> On Tue, Apr 30, 2013 at 11:03 AM, Furkan KAMACI <furkankamaci@gmail.com
> >> >wrote:
> >>
> >> > Should I stop the node first? And what will happen to transaction
> logs?
> >> > Should I backup it too?
> >> >
> >> > 2013/4/30 Michael Della Bitta <mi...@appinions.com>
> >> >
> >> > > That directory is the data directory for the core... you'd just
> swap it
> >> > in.
> >> > >
> >> > >
> >> > > Michael Della Bitta
> >> > >
> >> > > ------------------------------------------------
> >> > > Appinions
> >> > > 18 East 41st Street, 2nd Floor
> >> > > New York, NY 10017-6271
> >> > >
> >> > > www.appinions.com
> >> > >
> >> > > Where Influence Isn’t a Game
> >> > >
> >> > >
> >> > > On Tue, Apr 30, 2013 at 8:06 AM, Furkan KAMACI <
> furkankamaci@gmail.com
> >> > > >wrote:
> >> > >
> >> > > > Hi Folks;
> >> > > >
> >> > > > I can backup my indexes at SolrCloud via
> >> > > > http://_master_host_:_port_/solr/replication?command=backup
> >> > > > and it creates a file called snapshot. I know that I should pull
> that
> >> > > > directory any other safe place (a backup store) However what
> should I
> >> > do
> >> > > to
> >> > > > make a recovery from that backup file?
> >> > > >
> >> > >
> >> >
> >>
>
Re: How to Recovery Backup Snapshot at SolrCloud?
Posted by Timothy Potter <th...@gmail.com>.
I agree with Michael that you'll only ever need your backup if you
lose all nodes hosting a shard (leader + all other replicas), so the
tlog doesn't really factor in when recovering from backup.
The snapshot created by the replication handler is the index only and
it makes most sense in my mind to remove the tlog before firing up a
new node using the snapshot.
The way I see it, the backup would be needed if you lost all nodes
hosting a shard, then you need to 1) recover one node (the leader)
from backup, 2) re-index any documents that were indexed between the
time of your last snapshot and the time of the failure, then 3) bring
additional replicas online as needed. The additional replicas will
sync with the leader, snap pulling the entire index. You could do it
1,2,3 or 1,3,2.
Cheers,
Tim
On Tue, Apr 30, 2013 at 12:45 PM, Furkan KAMACI <fu...@gmail.com> wrote:
> I had index and tlog folder under my data folder. I have a snapshot folder
> too when I make backup. However what will I do next if I want to use
> backup, will I remove index and tlog folders and put just my snapshot
> folder? What folks do?
>
> 2013/4/30 Michael Della Bitta <mi...@appinions.com>
>
>> Presumably you'd only be restoring a backup in the face of a catastrophe.
>>
>> Yes, you'd need to stop the node. And the transaction logs may not be
>> useful in this case. You'd have trouble reconciling them with the version
>> of the index in your backup I would think.
>>
>> Anybody who knows more about this want to chime in?
>>
>>
>> Michael Della Bitta
>>
>> ------------------------------------------------
>> Appinions
>> 18 East 41st Street, 2nd Floor
>> New York, NY 10017-6271
>>
>> www.appinions.com
>>
>> Where Influence Isn’t a Game
>>
>>
>> On Tue, Apr 30, 2013 at 11:03 AM, Furkan KAMACI <furkankamaci@gmail.com
>> >wrote:
>>
>> > Should I stop the node first? And what will happen to transaction logs?
>> > Should I backup it too?
>> >
>> > 2013/4/30 Michael Della Bitta <mi...@appinions.com>
>> >
>> > > That directory is the data directory for the core... you'd just swap it
>> > in.
>> > >
>> > >
>> > > Michael Della Bitta
>> > >
>> > > ------------------------------------------------
>> > > Appinions
>> > > 18 East 41st Street, 2nd Floor
>> > > New York, NY 10017-6271
>> > >
>> > > www.appinions.com
>> > >
>> > > Where Influence Isn’t a Game
>> > >
>> > >
>> > > On Tue, Apr 30, 2013 at 8:06 AM, Furkan KAMACI <furkankamaci@gmail.com
>> > > >wrote:
>> > >
>> > > > Hi Folks;
>> > > >
>> > > > I can backup my indexes at SolrCloud via
>> > > > http://_master_host_:_port_/solr/replication?command=backup
>> > > > and it creates a file called snapshot. I know that I should pull that
>> > > > directory any other safe place (a backup store) However what should I
>> > do
>> > > to
>> > > > make a recovery from that backup file?
>> > > >
>> > >
>> >
>>
Re: How to Recovery Backup Snapshot at SolrCloud?
Posted by Furkan KAMACI <fu...@gmail.com>.
I had index and tlog folder under my data folder. I have a snapshot folder
too when I make backup. However what will I do next if I want to use
backup, will I remove index and tlog folders and put just my snapshot
folder? What folks do?
2013/4/30 Michael Della Bitta <mi...@appinions.com>
> Presumably you'd only be restoring a backup in the face of a catastrophe.
>
> Yes, you'd need to stop the node. And the transaction logs may not be
> useful in this case. You'd have trouble reconciling them with the version
> of the index in your backup I would think.
>
> Anybody who knows more about this want to chime in?
>
>
> Michael Della Bitta
>
> ------------------------------------------------
> Appinions
> 18 East 41st Street, 2nd Floor
> New York, NY 10017-6271
>
> www.appinions.com
>
> Where Influence Isn’t a Game
>
>
> On Tue, Apr 30, 2013 at 11:03 AM, Furkan KAMACI <furkankamaci@gmail.com
> >wrote:
>
> > Should I stop the node first? And what will happen to transaction logs?
> > Should I backup it too?
> >
> > 2013/4/30 Michael Della Bitta <mi...@appinions.com>
> >
> > > That directory is the data directory for the core... you'd just swap it
> > in.
> > >
> > >
> > > Michael Della Bitta
> > >
> > > ------------------------------------------------
> > > Appinions
> > > 18 East 41st Street, 2nd Floor
> > > New York, NY 10017-6271
> > >
> > > www.appinions.com
> > >
> > > Where Influence Isn’t a Game
> > >
> > >
> > > On Tue, Apr 30, 2013 at 8:06 AM, Furkan KAMACI <furkankamaci@gmail.com
> > > >wrote:
> > >
> > > > Hi Folks;
> > > >
> > > > I can backup my indexes at SolrCloud via
> > > > http://_master_host_:_port_/solr/replication?command=backup
> > > > and it creates a file called snapshot. I know that I should pull that
> > > > directory any other safe place (a backup store) However what should I
> > do
> > > to
> > > > make a recovery from that backup file?
> > > >
> > >
> >
>
Re: How to Recovery Backup Snapshot at SolrCloud?
Posted by Michael Della Bitta <mi...@appinions.com>.
Presumably you'd only be restoring a backup in the face of a catastrophe.
Yes, you'd need to stop the node. And the transaction logs may not be
useful in this case. You'd have trouble reconciling them with the version
of the index in your backup I would think.
Anybody who knows more about this want to chime in?
Michael Della Bitta
------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
www.appinions.com
Where Influence Isn’t a Game
On Tue, Apr 30, 2013 at 11:03 AM, Furkan KAMACI <fu...@gmail.com>wrote:
> Should I stop the node first? And what will happen to transaction logs?
> Should I backup it too?
>
> 2013/4/30 Michael Della Bitta <mi...@appinions.com>
>
> > That directory is the data directory for the core... you'd just swap it
> in.
> >
> >
> > Michael Della Bitta
> >
> > ------------------------------------------------
> > Appinions
> > 18 East 41st Street, 2nd Floor
> > New York, NY 10017-6271
> >
> > www.appinions.com
> >
> > Where Influence Isn’t a Game
> >
> >
> > On Tue, Apr 30, 2013 at 8:06 AM, Furkan KAMACI <furkankamaci@gmail.com
> > >wrote:
> >
> > > Hi Folks;
> > >
> > > I can backup my indexes at SolrCloud via
> > > http://_master_host_:_port_/solr/replication?command=backup
> > > and it creates a file called snapshot. I know that I should pull that
> > > directory any other safe place (a backup store) However what should I
> do
> > to
> > > make a recovery from that backup file?
> > >
> >
>
Re: How to Recovery Backup Snapshot at SolrCloud?
Posted by Furkan KAMACI <fu...@gmail.com>.
Should I stop the node first? And what will happen to transaction logs?
Should I backup it too?
2013/4/30 Michael Della Bitta <mi...@appinions.com>
> That directory is the data directory for the core... you'd just swap it in.
>
>
> Michael Della Bitta
>
> ------------------------------------------------
> Appinions
> 18 East 41st Street, 2nd Floor
> New York, NY 10017-6271
>
> www.appinions.com
>
> Where Influence Isn’t a Game
>
>
> On Tue, Apr 30, 2013 at 8:06 AM, Furkan KAMACI <furkankamaci@gmail.com
> >wrote:
>
> > Hi Folks;
> >
> > I can backup my indexes at SolrCloud via
> > http://_master_host_:_port_/solr/replication?command=backup
> > and it creates a file called snapshot. I know that I should pull that
> > directory any other safe place (a backup store) However what should I do
> to
> > make a recovery from that backup file?
> >
>
Re: How to Recovery Backup Snapshot at SolrCloud?
Posted by Michael Della Bitta <mi...@appinions.com>.
That directory is the data directory for the core... you'd just swap it in.
Michael Della Bitta
------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
www.appinions.com
Where Influence Isn’t a Game
On Tue, Apr 30, 2013 at 8:06 AM, Furkan KAMACI <fu...@gmail.com>wrote:
> Hi Folks;
>
> I can backup my indexes at SolrCloud via
> http://_master_host_:_port_/solr/replication?command=backup
> and it creates a file called snapshot. I know that I should pull that
> directory any other safe place (a backup store) However what should I do to
> make a recovery from that backup file?
>