You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Andy D'Arcy Jewell <an...@sysmicro.co.uk> on 2012/12/03 15:07:23 UTC
Backing up SolR 4.0
Hi all.
I'm new to SolR, and I have recently had to set up a SolR server running
4.0.
I've been searching for info on backing it up, but all I've managed to
come up with is "it'll be different" or "you'll be able to do push
replication" or using http and the command=backup parameter, which
doesn't sound like it will be effective for a production setup (unless
I've got that wrong)...
I was wondering if I can just stop or suspend the SolR server, then do
an LVM snapshot of the data store, before bringing it back on line, but
I'm not sure if that will cut it. I gather merely rsyncing the data
files won't do...
Can anyone give me a pointer to that "easy-to-find" document I have so
far failed to find? Or failing that, maybe some sound advice on how to
proceed?
Regards,
-Andy
--
Andy D'Arcy Jewell
SysMicro Limited
Linux Support
E: andy.jewell@sysmicro.co.uk
W: www.sysmicro.co.uk
Re: Pause and resume indexing on SolR 4 for backups
Posted by Upayavira <uv...@odoko.co.uk>.
That's neat, but wouldn't that run on every commit? How would you use it
to, say, back up once a day?
Upayavira
On Thu, Dec 20, 2012, at 01:57 PM, Markus Jelsma wrote:
> You can use the postCommit event in updateHandler to execute a task.
>
> -----Original message-----
> > From:Upayavira <uv...@odoko.co.uk>
> > Sent: Thu 20-Dec-2012 14:45
> > To: solr-user@lucene.apache.org
> > Subject: Re: Pause and resume indexing on SolR 4 for backups
> >
> > The backup directory should just be a clone of the index files. I'm
> > curious to know whether it is a cp -r or a cp -lr that the replication
> > handler produces.
> >
> > You would prevent commits by telling your app not to commit. That is,
> > Solr only commits when it is *told* to.
> >
> > Unless you use autocommit, in which case I guess you could monitor your
> > logs for the last commit, and do your backup a 10 seconds after that.
> >
> > Upayavira
> >
> > On Thu, Dec 20, 2012, at 12:44 PM, Andy D'Arcy Jewell wrote:
> > > On 20/12/12 11:58, Upayavira wrote:
> > > > I've never used it, but the replication handler has an option:
> > > >
> > > > http://master_host:port/solr/replication?command=backup
> > > >
> > > > Which will take you a backup.
> > > I've looked at that this morning as suggested by Markus Jelsma. Looks
> > > good, but I'll have to work out how to use the resultant backup
> > > directory. I've been dealing with another unrelated issue in the
> > > mean-time and I haven't had a chance to look for any docu so far.
> > > > Also something to note, if you don't want to use the above, and you are
> > > > running on Unix, you can create fast 'hard link' clones of lucene
> > > > indexes. Doing:
> > > >
> > > > cp -lr data data.bak
> > > >
> > > > will copy your index instantly. If you can avoid doing this when a
> > > > commit is happening, then you'll have a good index copy, that will take
> > > > no space on your disk and be made instantly. This is because it just
> > > > copies the directory structure, not the files themselves, and given
> > > > files in a lucene index never change (they are only ever deleted or
> > > > replaced), this works as a good copy technique for backing up.
> > > That's the approach that Shawn Heisey proposed, and what I've been
> > > working towards, but it still leaves open the question of how to
> > > *pause* SolR or prevent commits during the backup (otherwise we have a
> > > potential race condition).
> > >
> > > -Andy
> > >
> > >
> > > --
> > > Andy D'Arcy Jewell
> > >
> > > SysMicro Limited
> > > Linux Support
> > > E: andy.jewell@sysmicro.co.uk
> > > W: www.sysmicro.co.uk
> > >
> >
RE: Pause and resume indexing on SolR 4 for backups
Posted by Markus Jelsma <ma...@openindex.io>.
You can use the postCommit event in updateHandler to execute a task.
-----Original message-----
> From:Upayavira <uv...@odoko.co.uk>
> Sent: Thu 20-Dec-2012 14:45
> To: solr-user@lucene.apache.org
> Subject: Re: Pause and resume indexing on SolR 4 for backups
>
> The backup directory should just be a clone of the index files. I'm
> curious to know whether it is a cp -r or a cp -lr that the replication
> handler produces.
>
> You would prevent commits by telling your app not to commit. That is,
> Solr only commits when it is *told* to.
>
> Unless you use autocommit, in which case I guess you could monitor your
> logs for the last commit, and do your backup a 10 seconds after that.
>
> Upayavira
>
> On Thu, Dec 20, 2012, at 12:44 PM, Andy D'Arcy Jewell wrote:
> > On 20/12/12 11:58, Upayavira wrote:
> > > I've never used it, but the replication handler has an option:
> > >
> > > http://master_host:port/solr/replication?command=backup
> > >
> > > Which will take you a backup.
> > I've looked at that this morning as suggested by Markus Jelsma. Looks
> > good, but I'll have to work out how to use the resultant backup
> > directory. I've been dealing with another unrelated issue in the
> > mean-time and I haven't had a chance to look for any docu so far.
> > > Also something to note, if you don't want to use the above, and you are
> > > running on Unix, you can create fast 'hard link' clones of lucene
> > > indexes. Doing:
> > >
> > > cp -lr data data.bak
> > >
> > > will copy your index instantly. If you can avoid doing this when a
> > > commit is happening, then you'll have a good index copy, that will take
> > > no space on your disk and be made instantly. This is because it just
> > > copies the directory structure, not the files themselves, and given
> > > files in a lucene index never change (they are only ever deleted or
> > > replaced), this works as a good copy technique for backing up.
> > That's the approach that Shawn Heisey proposed, and what I've been
> > working towards, but it still leaves open the question of how to
> > *pause* SolR or prevent commits during the backup (otherwise we have a
> > potential race condition).
> >
> > -Andy
> >
> >
> > --
> > Andy D'Arcy Jewell
> >
> > SysMicro Limited
> > Linux Support
> > E: andy.jewell@sysmicro.co.uk
> > W: www.sysmicro.co.uk
> >
>
Re: Pause and resume indexing on SolR 4 for backups
Posted by Upayavira <uv...@odoko.co.uk>.
Are you sure a commit didn't happen between? Also, a background merge
might have happened.
As to using a backup, you are right, just stop solr, put the snapshot
into index/data, and restart.
Upayavira
On Thu, Dec 20, 2012, at 05:16 PM, Andy D'Arcy Jewell wrote:
> On 20/12/12 13:38, Upayavira wrote:
> > The backup directory should just be a clone of the index files. I'm
> > curious to know whether it is a cp -r or a cp -lr that the replication
> > handler produces.
> >
> > You would prevent commits by telling your app not to commit. That is,
> > Solr only commits when it is *told* to.
> >
> > Unless you use autocommit, in which case I guess you could monitor your
> > logs for the last commit, and do your backup a 10 seconds after that.
> >
> >
> Hmm. Strange - the files created by the backup API don't seem to
> correlate exactly with the files stored under the solr data directory:
>
> andydj@me-solr01:~$ find /tmp/snapshot.20121220155853703/
> /tmp/snapshot.20121220155853703/
> /tmp/snapshot.20121220155853703/_2vq.fdx
> /tmp/snapshot.20121220155853703/_2vq_Lucene40_0.tim
> /tmp/snapshot.20121220155853703/segments_2vs
> /tmp/snapshot.20121220155853703/_2vq_nrm.cfs
> /tmp/snapshot.20121220155853703/_2vq.fnm
> /tmp/snapshot.20121220155853703/_2vq_nrm.cfe
> /tmp/snapshot.20121220155853703/_2vq_Lucene40_0.frq
> /tmp/snapshot.20121220155853703/_2vq.fdt
> /tmp/snapshot.20121220155853703/_2vq.si
> /tmp/snapshot.20121220155853703/_2vq_Lucene40_0.tip
> andydj@me-solr01:~$ find /var/lib/solr/data/index/
> /var/lib/solr/data/index/
> /var/lib/solr/data/index/_2w6_Lucene40_0.frq
> /var/lib/solr/data/index/_2w6.si
> /var/lib/solr/data/index/segments_2w8
> /var/lib/solr/data/index/write.lock
> /var/lib/solr/data/index/_2w6_nrm.cfs
> /var/lib/solr/data/index/_2w6.fdx
> /var/lib/solr/data/index/_2w6_Lucene40_0.tip
> /var/lib/solr/data/index/_2w6_nrm.cfe
> /var/lib/solr/data/index/segments.gen
> /var/lib/solr/data/index/_2w6.fnm
> /var/lib/solr/data/index/_2w6.fdt
> /var/lib/solr/data/index/_2w6_Lucene40_0.tim
>
> Am I correct in thinking that to restore from this backup, I would need
> to do the following?
>
> 1. Stop Tomcat (or maybe just solr)
> 2. Remove all files under /var/lib/solr/data/index/
> 3. Move/copy files from /tmp/snapshot.20121220155853703/ to
> /var/lib/solr/data/index/
> 4. Restart Tomcat (or just solr)
>
>
> Thanks everyone who's pitched in on this! Once I've got this working,
> I'll document it.
> -Andy
>
> --
> Andy D'Arcy Jewell
>
> SysMicro Limited
> Linux Support
> E: andy.jewell@sysmicro.co.uk
> W: www.sysmicro.co.uk
>
Re: Pause and resume indexing on SolR 4 for backups
Posted by Andy D'Arcy Jewell <an...@sysmicro.co.uk>.
On 20/12/12 13:38, Upayavira wrote:
> The backup directory should just be a clone of the index files. I'm
> curious to know whether it is a cp -r or a cp -lr that the replication
> handler produces.
>
> You would prevent commits by telling your app not to commit. That is,
> Solr only commits when it is *told* to.
>
> Unless you use autocommit, in which case I guess you could monitor your
> logs for the last commit, and do your backup a 10 seconds after that.
>
>
Hmm. Strange - the files created by the backup API don't seem to
correlate exactly with the files stored under the solr data directory:
andydj@me-solr01:~$ find /tmp/snapshot.20121220155853703/
/tmp/snapshot.20121220155853703/
/tmp/snapshot.20121220155853703/_2vq.fdx
/tmp/snapshot.20121220155853703/_2vq_Lucene40_0.tim
/tmp/snapshot.20121220155853703/segments_2vs
/tmp/snapshot.20121220155853703/_2vq_nrm.cfs
/tmp/snapshot.20121220155853703/_2vq.fnm
/tmp/snapshot.20121220155853703/_2vq_nrm.cfe
/tmp/snapshot.20121220155853703/_2vq_Lucene40_0.frq
/tmp/snapshot.20121220155853703/_2vq.fdt
/tmp/snapshot.20121220155853703/_2vq.si
/tmp/snapshot.20121220155853703/_2vq_Lucene40_0.tip
andydj@me-solr01:~$ find /var/lib/solr/data/index/
/var/lib/solr/data/index/
/var/lib/solr/data/index/_2w6_Lucene40_0.frq
/var/lib/solr/data/index/_2w6.si
/var/lib/solr/data/index/segments_2w8
/var/lib/solr/data/index/write.lock
/var/lib/solr/data/index/_2w6_nrm.cfs
/var/lib/solr/data/index/_2w6.fdx
/var/lib/solr/data/index/_2w6_Lucene40_0.tip
/var/lib/solr/data/index/_2w6_nrm.cfe
/var/lib/solr/data/index/segments.gen
/var/lib/solr/data/index/_2w6.fnm
/var/lib/solr/data/index/_2w6.fdt
/var/lib/solr/data/index/_2w6_Lucene40_0.tim
Am I correct in thinking that to restore from this backup, I would need
to do the following?
1. Stop Tomcat (or maybe just solr)
2. Remove all files under /var/lib/solr/data/index/
3. Move/copy files from /tmp/snapshot.20121220155853703/ to
/var/lib/solr/data/index/
4. Restart Tomcat (or just solr)
Thanks everyone who's pitched in on this! Once I've got this working,
I'll document it.
-Andy
--
Andy D'Arcy Jewell
SysMicro Limited
Linux Support
E: andy.jewell@sysmicro.co.uk
W: www.sysmicro.co.uk
Re: Pause and resume indexing on SolR 4 for backups
Posted by Upayavira <uv...@odoko.co.uk>.
The backup directory should just be a clone of the index files. I'm
curious to know whether it is a cp -r or a cp -lr that the replication
handler produces.
You would prevent commits by telling your app not to commit. That is,
Solr only commits when it is *told* to.
Unless you use autocommit, in which case I guess you could monitor your
logs for the last commit, and do your backup a 10 seconds after that.
Upayavira
On Thu, Dec 20, 2012, at 12:44 PM, Andy D'Arcy Jewell wrote:
> On 20/12/12 11:58, Upayavira wrote:
> > I've never used it, but the replication handler has an option:
> >
> > http://master_host:port/solr/replication?command=backup
> >
> > Which will take you a backup.
> I've looked at that this morning as suggested by Markus Jelsma. Looks
> good, but I'll have to work out how to use the resultant backup
> directory. I've been dealing with another unrelated issue in the
> mean-time and I haven't had a chance to look for any docu so far.
> > Also something to note, if you don't want to use the above, and you are
> > running on Unix, you can create fast 'hard link' clones of lucene
> > indexes. Doing:
> >
> > cp -lr data data.bak
> >
> > will copy your index instantly. If you can avoid doing this when a
> > commit is happening, then you'll have a good index copy, that will take
> > no space on your disk and be made instantly. This is because it just
> > copies the directory structure, not the files themselves, and given
> > files in a lucene index never change (they are only ever deleted or
> > replaced), this works as a good copy technique for backing up.
> That's the approach that Shawn Heisey proposed, and what I've been
> working towards, but it still leaves open the question of how to
> *pause* SolR or prevent commits during the backup (otherwise we have a
> potential race condition).
>
> -Andy
>
>
> --
> Andy D'Arcy Jewell
>
> SysMicro Limited
> Linux Support
> E: andy.jewell@sysmicro.co.uk
> W: www.sysmicro.co.uk
>
Re: Pause and resume indexing on SolR 4 for backups
Posted by Andy D'Arcy Jewell <an...@sysmicro.co.uk>.
On 20/12/12 11:58, Upayavira wrote:
> I've never used it, but the replication handler has an option:
>
> http://master_host:port/solr/replication?command=backup
>
> Which will take you a backup.
I've looked at that this morning as suggested by Markus Jelsma. Looks
good, but I'll have to work out how to use the resultant backup
directory. I've been dealing with another unrelated issue in the
mean-time and I haven't had a chance to look for any docu so far.
> Also something to note, if you don't want to use the above, and you are
> running on Unix, you can create fast 'hard link' clones of lucene
> indexes. Doing:
>
> cp -lr data data.bak
>
> will copy your index instantly. If you can avoid doing this when a
> commit is happening, then you'll have a good index copy, that will take
> no space on your disk and be made instantly. This is because it just
> copies the directory structure, not the files themselves, and given
> files in a lucene index never change (they are only ever deleted or
> replaced), this works as a good copy technique for backing up.
That's the approach that Shawn Heisey proposed, and what I've been
working towards, but it still leaves open the question of how to
*pause* SolR or prevent commits during the backup (otherwise we have a
potential race condition).
-Andy
--
Andy D'Arcy Jewell
SysMicro Limited
Linux Support
E: andy.jewell@sysmicro.co.uk
W: www.sysmicro.co.uk
Re: Pause and resume indexing on SolR 4 for backups
Posted by Upayavira <uv...@odoko.co.uk>.
I've never used it, but the replication handler has an option:
http://master_host:port/solr/replication?command=backup
Which will take you a backup.
Also something to note, if you don't want to use the above, and you are
running on Unix, you can create fast 'hard link' clones of lucene
indexes. Doing:
cp -lr data data.bak
will copy your index instantly. If you can avoid doing this when a
commit is happening, then you'll have a good index copy, that will take
no space on your disk and be made instantly. This is because it just
copies the directory structure, not the files themselves, and given
files in a lucene index never change (they are only ever deleted or
replaced), this works as a good copy technique for backing up.
Upayavira
On Thu, Dec 20, 2012, at 10:34 AM, Markus Jelsma wrote:
> You can use the replication handler to fetch a complete snapshot of the
> index over HTTP.
> http://wiki.apache.org/solr/SolrReplication#HTTP_API
>
>
> -----Original message-----
> > From:Andy D'Arcy Jewell <an...@sysmicro.co.uk>
> > Sent: Thu 20-Dec-2012 11:23
> > To: solr-user@lucene.apache.org
> > Subject: Pause and resume indexing on SolR 4 for backups
> >
> > Hi all.
> >
> > Can anyone advise me of a way to pause and resume SolR 4 so I can
> > perform a backup? I need to be able to revert to a usable (though not
> > necessarily complete) index after a crash or other "disaster" more
> > quickly than a re-index operation would yield.
> >
> > I can't yet afford the "extravagance" of a separate SolR replica just
> > for backups, and I'm not sure if I'll ever have the luxury. I'm
> > currently running with just one node, be we are not yet live.
> >
> > I can think of the following ways to do this, each with various downsides:
> >
> > 1) Just backup the existing index files whilst indexing continues
> > + Easy
> > + Fast
> > - Incomplete
> > - Potential for corruption? (e.g. partial files)
> >
> > 2) Stop/Start Tomcat
> > + Easy
> > - Very slow and I/O, CPU intensive
> > - Client gets errors when trying to connect
> >
> > 3) Block/unblock SolR port with IpTables
> > + Fast
> > - Client gets errors when trying to connect
> > - Have to wait for existing transactions to complete (not sure how,
> > maybe watch socket FD's in /proc)
> >
> > 4) Pause/Restart SolR service
> > + Fast ? (hopefully)
> > - Client gets errors when trying to connect
> >
> > In any event, the web app will have to gracefully handle unavailability
> > of SolR, probably by displaying a "down for maintenance" message, but
> > this should preferably be only a very short amount of time.
> >
> > Can anyone comment on my proposed solutions above, or provide any
> > additional ones?
> >
> > Thanks for any input you can provide!
> >
> > -Andy
> >
> > --
> > Andy D'Arcy Jewell
> >
> > SysMicro Limited
> > Linux Support
> > E: andy.jewell@sysmicro.co.uk
> > W: www.sysmicro.co.uk
> >
> >
RE: Pause and resume indexing on SolR 4 for backups
Posted by Markus Jelsma <ma...@openindex.io>.
You can use the replication handler to fetch a complete snapshot of the index over HTTP.
http://wiki.apache.org/solr/SolrReplication#HTTP_API
-----Original message-----
> From:Andy D'Arcy Jewell <an...@sysmicro.co.uk>
> Sent: Thu 20-Dec-2012 11:23
> To: solr-user@lucene.apache.org
> Subject: Pause and resume indexing on SolR 4 for backups
>
> Hi all.
>
> Can anyone advise me of a way to pause and resume SolR 4 so I can
> perform a backup? I need to be able to revert to a usable (though not
> necessarily complete) index after a crash or other "disaster" more
> quickly than a re-index operation would yield.
>
> I can't yet afford the "extravagance" of a separate SolR replica just
> for backups, and I'm not sure if I'll ever have the luxury. I'm
> currently running with just one node, be we are not yet live.
>
> I can think of the following ways to do this, each with various downsides:
>
> 1) Just backup the existing index files whilst indexing continues
> + Easy
> + Fast
> - Incomplete
> - Potential for corruption? (e.g. partial files)
>
> 2) Stop/Start Tomcat
> + Easy
> - Very slow and I/O, CPU intensive
> - Client gets errors when trying to connect
>
> 3) Block/unblock SolR port with IpTables
> + Fast
> - Client gets errors when trying to connect
> - Have to wait for existing transactions to complete (not sure how,
> maybe watch socket FD's in /proc)
>
> 4) Pause/Restart SolR service
> + Fast ? (hopefully)
> - Client gets errors when trying to connect
>
> In any event, the web app will have to gracefully handle unavailability
> of SolR, probably by displaying a "down for maintenance" message, but
> this should preferably be only a very short amount of time.
>
> Can anyone comment on my proposed solutions above, or provide any
> additional ones?
>
> Thanks for any input you can provide!
>
> -Andy
>
> --
> Andy D'Arcy Jewell
>
> SysMicro Limited
> Linux Support
> E: andy.jewell@sysmicro.co.uk
> W: www.sysmicro.co.uk
>
>
Re: Pause and resume indexing on SolR 4 for backups
Posted by Gora Mohanty <go...@mimirtech.com>.
On 20 December 2012 16:14, Andy D'Arcy Jewell
<an...@sysmicro.co.uk> wrote:
[...]
> It's attached to a web-app, which accepts uploads and will be available
> 24/7, with a global audience, so "pausing" it may be rather difficult (tho I
> may put this to the developer - it may for instance be possible if he has a
> small number of choke points for input into SolR).
[...]
It adds work for the web developer, but one could pause indexing,
put indexing requests into some kind of a queuing system, do the
backup, and flush the queue when the backup is done.
Regards,
Gora
Re: Pause and resume indexing on SolR 4 for backups
Posted by Andy D'Arcy Jewell <an...@sysmicro.co.uk>.
On 20/12/12 10:24, Gora Mohanty wrote:
>
> Unless I am missing something, the index is only being written to
> when you are adding/updating the index. So, the question is how
> is this being done in your case, and could you pause indexing for
> the duration of the backup?
>
> Regards,
> Gora
It's attached to a web-app, which accepts uploads and will be available
24/7, with a global audience, so "pausing" it may be rather difficult
(tho I may put this to the developer - it may for instance be possible
if he has a small number of choke points for input into SolR).
Thanks.
--
Andy D'Arcy Jewell
SysMicro Limited
Linux Support
T: 0844 9918804
M: 07961605631
E: andy.jewell@sysmicro.co.uk
W: www.sysmicro.co.uk
Re: Pause and resume indexing on SolR 4 for backups
Posted by Gora Mohanty <go...@mimirtech.com>.
On 20 December 2012 15:46, Andy D'Arcy Jewell
<an...@sysmicro.co.uk> wrote:
> Hi all.
>
> Can anyone advise me of a way to pause and resume SolR 4 so I can perform a
> backup? I need to be able to revert to a usable (though not necessarily
> complete) index after a crash or other "disaster" more quickly than a
> re-index operation would yield.
[...]
Unless I am missing something, the index is only being written to
when you are adding/updating the index. So, the question is how
is this being done in your case, and could you pause indexing for
the duration of the backup?
Regards,
Gora
Re: Pause and resume indexing on SolR 4 for backups
Posted by Andy D'Arcy Jewell <an...@sysmicro.co.uk>.
On 20/12/12 20:19, alxsss@aim.com wrote:
> Depending on your architecture, why not index the same data into two machines? One will be your prod another your backup?
Because we're trying to keep costs and complexity low whilst in the
development stage ;-)
But more seriously, this will obviously be a must sooner or later.
--
Andy D'Arcy Jewell
SysMicro Limited
Linux Support
T: 0844 9918804
M: 07961605631
E: andy.jewell@sysmicro.co.uk
W: www.sysmicro.co.uk
Re: Pause and resume indexing on SolR 4 for backups
Posted by al...@aim.com.
Depending on your architecture, why not index the same data into two machines? One will be your prod another your backup?
Thanks.
Alex.
-----Original Message-----
From: Upayavira <uv...@odoko.co.uk>
To: solr-user <so...@lucene.apache.org>
Sent: Thu, Dec 20, 2012 11:51 am
Subject: Re: Pause and resume indexing on SolR 4 for backups
You're saying that there's no chance to catch it in the middle of
writing the segments file?
Having said that, the segments file is pretty small, so the chance would
be pretty slim.
Upayavira
On Thu, Dec 20, 2012, at 06:45 PM, Lance Norskog wrote:
> To be clear: 1) is fine. Lucene index updates are carefully sequenced so
> that the index is never in a bogus state. All data files are written and
> flushed to disk, then the segments.* files are written that match the
> data files. You can capture the files with a set of hard links to create
> a backup.
>
> The CheckIndex program will verify the index backup.
> java -cp yourcopy/lucene-core-SOMETHING.jar
> org.apache.lucene.index.CheckIndex collection/data/index
>
> lucene-core-SOMETHING.jar is usually in the solr-webapp directory where
> Solr is unpacked.
>
> On 12/20/2012 02:16 AM, Andy D'Arcy Jewell wrote:
> > Hi all.
> >
> > Can anyone advise me of a way to pause and resume SolR 4 so I can
> > perform a backup? I need to be able to revert to a usable (though not
> > necessarily complete) index after a crash or other "disaster" more
> > quickly than a re-index operation would yield.
> >
> > I can't yet afford the "extravagance" of a separate SolR replica just
> > for backups, and I'm not sure if I'll ever have the luxury. I'm
> > currently running with just one node, be we are not yet live.
> >
> > I can think of the following ways to do this, each with various
> > downsides:
> >
> > 1) Just backup the existing index files whilst indexing continues
> > + Easy
> > + Fast
> > - Incomplete
> > - Potential for corruption? (e.g. partial files)
> >
> > 2) Stop/Start Tomcat
> > + Easy
> > - Very slow and I/O, CPU intensive
> > - Client gets errors when trying to connect
> >
> > 3) Block/unblock SolR port with IpTables
> > + Fast
> > - Client gets errors when trying to connect
> > - Have to wait for existing transactions to complete (not sure
> > how, maybe watch socket FD's in /proc)
> >
> > 4) Pause/Restart SolR service
> > + Fast ? (hopefully)
> > - Client gets errors when trying to connect
> >
> > In any event, the web app will have to gracefully handle
> > unavailability of SolR, probably by displaying a "down for
> > maintenance" message, but this should preferably be only a very short
> > amount of time.
> >
> > Can anyone comment on my proposed solutions above, or provide any
> > additional ones?
> >
> > Thanks for any input you can provide!
> >
> > -Andy
> >
>
Re: Pause and resume indexing on SolR 4 for backups
Posted by Upayavira <uv...@odoko.co.uk>.
You're saying that there's no chance to catch it in the middle of
writing the segments file?
Having said that, the segments file is pretty small, so the chance would
be pretty slim.
Upayavira
On Thu, Dec 20, 2012, at 06:45 PM, Lance Norskog wrote:
> To be clear: 1) is fine. Lucene index updates are carefully sequenced so
> that the index is never in a bogus state. All data files are written and
> flushed to disk, then the segments.* files are written that match the
> data files. You can capture the files with a set of hard links to create
> a backup.
>
> The CheckIndex program will verify the index backup.
> java -cp yourcopy/lucene-core-SOMETHING.jar
> org.apache.lucene.index.CheckIndex collection/data/index
>
> lucene-core-SOMETHING.jar is usually in the solr-webapp directory where
> Solr is unpacked.
>
> On 12/20/2012 02:16 AM, Andy D'Arcy Jewell wrote:
> > Hi all.
> >
> > Can anyone advise me of a way to pause and resume SolR 4 so I can
> > perform a backup? I need to be able to revert to a usable (though not
> > necessarily complete) index after a crash or other "disaster" more
> > quickly than a re-index operation would yield.
> >
> > I can't yet afford the "extravagance" of a separate SolR replica just
> > for backups, and I'm not sure if I'll ever have the luxury. I'm
> > currently running with just one node, be we are not yet live.
> >
> > I can think of the following ways to do this, each with various
> > downsides:
> >
> > 1) Just backup the existing index files whilst indexing continues
> > + Easy
> > + Fast
> > - Incomplete
> > - Potential for corruption? (e.g. partial files)
> >
> > 2) Stop/Start Tomcat
> > + Easy
> > - Very slow and I/O, CPU intensive
> > - Client gets errors when trying to connect
> >
> > 3) Block/unblock SolR port with IpTables
> > + Fast
> > - Client gets errors when trying to connect
> > - Have to wait for existing transactions to complete (not sure
> > how, maybe watch socket FD's in /proc)
> >
> > 4) Pause/Restart SolR service
> > + Fast ? (hopefully)
> > - Client gets errors when trying to connect
> >
> > In any event, the web app will have to gracefully handle
> > unavailability of SolR, probably by displaying a "down for
> > maintenance" message, but this should preferably be only a very short
> > amount of time.
> >
> > Can anyone comment on my proposed solutions above, or provide any
> > additional ones?
> >
> > Thanks for any input you can provide!
> >
> > -Andy
> >
>
Re: Pause and resume indexing on SolR 4 for backups
Posted by Lance Norskog <go...@gmail.com>.
To be clear: 1) is fine. Lucene index updates are carefully sequenced so
that the index is never in a bogus state. All data files are written and
flushed to disk, then the segments.* files are written that match the
data files. You can capture the files with a set of hard links to create
a backup.
The CheckIndex program will verify the index backup.
java -cp yourcopy/lucene-core-SOMETHING.jar
org.apache.lucene.index.CheckIndex collection/data/index
lucene-core-SOMETHING.jar is usually in the solr-webapp directory where
Solr is unpacked.
On 12/20/2012 02:16 AM, Andy D'Arcy Jewell wrote:
> Hi all.
>
> Can anyone advise me of a way to pause and resume SolR 4 so I can
> perform a backup? I need to be able to revert to a usable (though not
> necessarily complete) index after a crash or other "disaster" more
> quickly than a re-index operation would yield.
>
> I can't yet afford the "extravagance" of a separate SolR replica just
> for backups, and I'm not sure if I'll ever have the luxury. I'm
> currently running with just one node, be we are not yet live.
>
> I can think of the following ways to do this, each with various
> downsides:
>
> 1) Just backup the existing index files whilst indexing continues
> + Easy
> + Fast
> - Incomplete
> - Potential for corruption? (e.g. partial files)
>
> 2) Stop/Start Tomcat
> + Easy
> - Very slow and I/O, CPU intensive
> - Client gets errors when trying to connect
>
> 3) Block/unblock SolR port with IpTables
> + Fast
> - Client gets errors when trying to connect
> - Have to wait for existing transactions to complete (not sure
> how, maybe watch socket FD's in /proc)
>
> 4) Pause/Restart SolR service
> + Fast ? (hopefully)
> - Client gets errors when trying to connect
>
> In any event, the web app will have to gracefully handle
> unavailability of SolR, probably by displaying a "down for
> maintenance" message, but this should preferably be only a very short
> amount of time.
>
> Can anyone comment on my proposed solutions above, or provide any
> additional ones?
>
> Thanks for any input you can provide!
>
> -Andy
>
Pause and resume indexing on SolR 4 for backups
Posted by Andy D'Arcy Jewell <an...@sysmicro.co.uk>.
Hi all.
Can anyone advise me of a way to pause and resume SolR 4 so I can
perform a backup? I need to be able to revert to a usable (though not
necessarily complete) index after a crash or other "disaster" more
quickly than a re-index operation would yield.
I can't yet afford the "extravagance" of a separate SolR replica just
for backups, and I'm not sure if I'll ever have the luxury. I'm
currently running with just one node, be we are not yet live.
I can think of the following ways to do this, each with various downsides:
1) Just backup the existing index files whilst indexing continues
+ Easy
+ Fast
- Incomplete
- Potential for corruption? (e.g. partial files)
2) Stop/Start Tomcat
+ Easy
- Very slow and I/O, CPU intensive
- Client gets errors when trying to connect
3) Block/unblock SolR port with IpTables
+ Fast
- Client gets errors when trying to connect
- Have to wait for existing transactions to complete (not sure how,
maybe watch socket FD's in /proc)
4) Pause/Restart SolR service
+ Fast ? (hopefully)
- Client gets errors when trying to connect
In any event, the web app will have to gracefully handle unavailability
of SolR, probably by displaying a "down for maintenance" message, but
this should preferably be only a very short amount of time.
Can anyone comment on my proposed solutions above, or provide any
additional ones?
Thanks for any input you can provide!
-Andy
--
Andy D'Arcy Jewell
SysMicro Limited
Linux Support
E: andy.jewell@sysmicro.co.uk
W: www.sysmicro.co.uk
Re: Backing up SolR 4.0
Posted by Shawn Heisey <so...@elyograg.org>.
On 12/4/2012 1:55 AM, Andy D'Arcy Jewell wrote:
> Is there an easy way to tell (say from a shell script) when "all
> commits and merges [are] complete"?
One important bit of information I just thought of: A default Solr 4
config uses a new directory implementation called NRTCachingDirectory,
which in some circumstances may keep part of the newest segment(s) in
RAM. I *hope* that issuing an explicit hard commit will flush that to
disk, but I am not sure. You *might* need to switch to the old
directory implementation to be sure a hardlink backup is complete. Can
one of the committers please comment on this? Assuming that we work out
this detail, the rest of what I've said will be valid.
Detecting when commits are done has to be coordinated with your indexing
program, so depending on how your system works, kicking off the "make
hardlinks" process might need to be part of your indexing program. As
for merges, that's a bit tougher, because Solr 4 and later will do
merges in the background after informing your indexing program that the
commit is complete.
If you grab a hardlink copy while a merge is happening, I do not believe
it will be corrupt in any way, but it may be larger than expected
because it will contain the new segments from the merge. Those segments
would not be referenced by the segments.nnn file, so I *think* that if
you then load that index into Solr, it would ignore the other segments.
I am not sure about that, though. You might be able to use a command
like the following to delete the newer segments from the copy, but I
would not do it without experimentation to be sure it's actually
required, and that it never wipes anything out that you actually need:
find -type f -newer segments.gen | xargs rm -f
> If I keep a replica solely for backup purposes, I assume I can "do
> what I like with it" - presumably replication will resume/catch-up
> when I resume it (I admit, I have a bit of reading to do wrt
> replication - I just skimmed that because it wasn't in my initial brief).
As long as the replica server isn't being actively updated or used for
queries and you temporarily turn off replication, you should be able to
do whatever you want with its index.
> I'm assuming that because you're using hardlinks, that means that SolR
> writes a "new" file when it updates (sortof copy-on-write style)? So
> we are relying on the principle that as long as you have at least one
> remaining reference to the data, it's not deleted...
Yes. Lucene (which Solr uses under the hood) never touches segment files
once they have been written. It only deletes segment files in two
circumstances: 1) Every document in that segment has been deleted from
the index. 2) The data in that segment has been written to a new
segment. The combination of Lucene's update method and hardlink
functionality will ensure that the hardlink copy is always good.
Thanks,
Shawn
Re: Backing up SolR 4.0
Posted by Andy D'Arcy Jewell <an...@sysmicro.co.uk>.
On 03/12/12 18:04, Shawn Heisey wrote:
>
> Serious production Solr installs require at least two copies of your
> index. Failures *will* happen, and sometimes they'll be the kind of
> failures that will take down an entire machine. You can plan for some
> failures -- redundant power supply and RAID are important for this.
> Some failures will cause downtime, though -- multiple disk failures,
> motherboard, CPU, memory, software problems wiping out your index,
> user error, etc.If you have at least one other copy of your index,
> you'll be able to keep the system operational while you fix the down
> machine.
>
> Replication is a very good way to accomplish getting two or more
> copies of your index. I would expect that most production Solr
> installations use either plain replication or SolrCloud. I do my
> redundancy a different way that gives me a lot more flexibility, but
> replication is a VERY solid way to go.
>
> If you are running on a UNIX/Linux platform (just about anything
> *other* than Windows), and backups via replication are not enough for
> you, you can use the hardlink capability in the OS to avoid taking
> Solr down while you make backups. Here's the basic sequence:
>
> 1) Pause indexing, wait for all commits and merges to complete.
> 2) Create a target directory on the same filesystem as your Solr index.
> 3) Make hardlinks of all files in your Solr index in the target
> directory.
> 4) Resume indexing.
> 5) Copy the target directory to your backup location at your leisure.
> 6) Delete the hardlink copies from the target directory.
>
> Making hardlinks is a near-instantaneous operation. The way that
> Solr/Lucene works will guarantee that your hardlink copy will continue
> to be a valid index snapshot no matter what happens to the live
> index. If you can make the backup and get the hardlinks deleted
> before your index undergoes a merge, the hardlinks will use very
> little extra disk space.
>
> If you leave the hardlink copies around, eventually your live index
> will diverge to the point where the copy has different files and
> therefore takes up disk space. If you have a *LOT* of extra disk
> space on the Solr server, you can keep multiple hardlink copies around
> as snapshots.
>
> Recent versions of Windows do have features similar to UNIX links, so
> there may in fact be a way to do this on Windows. I will leave that
> for someone else to pursue.
>
> Thanks,
> Shawn
>
Thanks Shawn, that's very informative. I get twitchy with anything where
you "can't" back it up (memcached excepted). As an administrator, it's
my job to recover from failures, and backups are kind of my comfort blanket.
I'm running on Linux (on Debian Squeeze) in a fully virtual
environment. Initially, I think I'll have to just schedule the backup
for the early hours (local time) but as we grow, I can see I'll have to
use replication to do it seamlessly. The system is necessarily small
right now, as we haven't yet gone live, butwe are anticipating rapid
growth, so replication has always been on the cards.
Is there an easy way to tell (say from a shell script) when "all commits
and merges [are] complete"?
If I keep a replica solely for backup purposes, I assume I can "do what
I like with it" - presumably replication will resume/catch-up when I
resume it (I admit, I have a bit of reading to do wrt replication - I
just skimmed that because it wasn't in my initial brief).
I'm assuming that because you're using hardlinks, that means that SolR
writes a "new" file when it updates (sortof copy-on-write style)? So we
are relying on the principle that as long as you have at least one
remaining reference to the data, it's not deleted...
Thanks once again!
-Andy
--
Andy D'Arcy Jewell
SysMicro Limited
Linux Support
E: andy.jewell@sysmicro.co.uk
W: www.sysmicro.co.uk
Re: Backing up SolR 4.0
Posted by Shawn Heisey <so...@elyograg.org>.
On 12/3/2012 9:47 AM, Andy D'Arcy Jewell wrote:
> However, wouldn't re-creating the index on a large dataset take an
> inordinate amount of time? The system I will be backing up is likely
> to undergo rapid development and thus schema changes, so I need some
> kind of insurance against corruption if we need to roll-back after a
> change.
>
> How should I go about creating multiplebackup versions I can put aside
> (e.g. on tape) to hedge against the down-time which would be required
> to regenerate the indexes from scratch?
Serious production Solr installs require at least two copies of your
index. Failures *will* happen, and sometimes they'll be the kind of
failures that will take down an entire machine. You can plan for some
failures -- redundant power supply and RAID are important for this.
Some failures will cause downtime, though -- multiple disk failures,
motherboard, CPU, memory, software problems wiping out your index, user
error, etc.If you have at least one other copy of your index, you'll be
able to keep the system operational while you fix the down machine.
Replication is a very good way to accomplish getting two or more copies
of your index. I would expect that most production Solr installations
use either plain replication or SolrCloud. I do my redundancy a
different way that gives me a lot more flexibility, but replication is a
VERY solid way to go.
If you are running on a UNIX/Linux platform (just about anything *other*
than Windows), and backups via replication are not enough for you, you
can use the hardlink capability in the OS to avoid taking Solr down
while you make backups. Here's the basic sequence:
1) Pause indexing, wait for all commits and merges to complete.
2) Create a target directory on the same filesystem as your Solr index.
3) Make hardlinks of all files in your Solr index in the target directory.
4) Resume indexing.
5) Copy the target directory to your backup location at your leisure.
6) Delete the hardlink copies from the target directory.
Making hardlinks is a near-instantaneous operation. The way that
Solr/Lucene works will guarantee that your hardlink copy will continue
to be a valid index snapshot no matter what happens to the live index.
If you can make the backup and get the hardlinks deleted before your
index undergoes a merge, the hardlinks will use very little extra disk
space.
If you leave the hardlink copies around, eventually your live index will
diverge to the point where the copy has different files and therefore
takes up disk space. If you have a *LOT* of extra disk space on the
Solr server, you can keep multiple hardlink copies around as snapshots.
Recent versions of Windows do have features similar to UNIX links, so
there may in fact be a way to do this on Windows. I will leave that for
someone else to pursue.
Thanks,
Shawn
Re: Backing up SolR 4.0
Posted by Andy D'Arcy Jewell <an...@sysmicro.co.uk>.
On 03/12/12 16:39, Erick Erickson wrote:
> There's no real need to do what you ask.
>
> First thing is that you should always be prepared, in the worst-case
> scenario, to regenerate your entire index.
>
> That said, perhaps the easiest way to back up Solr is just to use
> master/slave replication. Consider having a machine that's a slave to the
> master (but not necessarily searched against) and periodically poll your
> master (say daily or whatever your interval is). You can configure Solr to
> keep N copies of the index as extra insurance. These will be fairly static
> so if you_really_ wanted to you could just copy the <solrhome>/data
> directory somewhere, but I don't know if that's necessary.
>
> See:http://wiki.apache.org/solr/SolrReplication
>
> Best
> Erick
Hi Erick,
Thanks for that, I'll take a look.
However, wouldn't re-creating the index on a large dataset take an
inordinate amount of time? The system I will be backing up is likely to
undergo rapid development and thus schema changes, so I need some kind
of insurance against corruption if we need to roll-back after a change.
How should I go about creating multiplebackup versions I can put aside
(e.g. on tape) to hedge against the down-time which would be required to
regenerate the indexes from scratch?
Regards,
-Andy
--
Andy D'Arcy Jewell
SysMicro Limited
Linux Support
E: andy.jewell@sysmicro.co.uk
W: www.sysmicro.co.uk
Re: Backing up SolR 4.0
Posted by Erick Erickson <er...@gmail.com>.
There's no real need to do what you ask.
First thing is that you should always be prepared, in the worst-case
scenario, to regenerate your entire index.
That said, perhaps the easiest way to back up Solr is just to use
master/slave replication. Consider having a machine that's a slave to the
master (but not necessarily searched against) and periodically poll your
master (say daily or whatever your interval is). You can configure Solr to
keep N copies of the index as extra insurance. These will be fairly static
so if you _really_ wanted to you could just copy the <solrhome>/data
directory somewhere, but I don't know if that's necessary.
See: http://wiki.apache.org/solr/SolrReplication
Best
Erick
On Mon, Dec 3, 2012 at 6:07 AM, Andy D'Arcy Jewell <
andy.jewell@sysmicro.co.uk> wrote:
> Hi all.
>
> I'm new to SolR, and I have recently had to set up a SolR server running
> 4.0.
>
> I've been searching for info on backing it up, but all I've managed to
> come up with is "it'll be different" or "you'll be able to do push
> replication" or using http and the command=backup parameter, which doesn't
> sound like it will be effective for a production setup (unless I've got
> that wrong)...
>
>
> I was wondering if I can just stop or suspend the SolR server, then do an
> LVM snapshot of the data store, before bringing it back on line, but I'm
> not sure if that will cut it. I gather merely rsyncing the data files won't
> do...
>
> Can anyone give me a pointer to that "easy-to-find" document I have so far
> failed to find? Or failing that, maybe some sound advice on how to proceed?
>
> Regards,
> -Andy
>
>
>
>
> --
> Andy D'Arcy Jewell
>
> SysMicro Limited
> Linux Support
> E: andy.jewell@sysmicro.co.uk
> W: www.sysmicro.co.uk
>
>