Auto recovery of solr

[Poorna Murali <po...@gmail.com>]

Hi All,

If we have an alerting mechanism which says if a solr node is  down, is it
possible to configure the alert with some action which will restart that
solr node and recover automatically?
Please advise.

Thanks,
Poorna

Re: Auto recovery of solr

[Shawn Heisey <ap...@elyograg.org>]

On 6/21/2022 4:54 AM, Poorna Murali wrote:
> If we have an alerting mechanism which says if a solr node is down, is it
> possible to configure the alert with some action which will restart that
> solr node and recover automatically?
> Please advise.

I agree with Dave.

When the environment is valid (OS and Java are working correctly and 
there is plenty of disk space, memory, and other resources like 
available file handles), Solr does not crash.  If it HAS crashed, then 
something is wrong that needs fixing, and simply restarting Solr is 
almost certainly going to lead to it crashing again.  Because of this, 
Solr does not come with anything that will automatically restart it.  If 
you want that, you'll need to implement it as part of your monitoring 
... and it's not something that we recommend.

Thanks,
Shawn

Re: Auto recovery of solr

[David Hastings <ha...@gmail.com>]

OOM errors do become annoying, so implemented
https://www.tecmint.com/clear-ram-memory-cache-buffer-and-swap-space-on-linux/

to the restart sh as well, but as Shawn has said, the ones deployed have
never gone down, granted each are given 31gb of heap space on a 200gb+ Ram
server.  sometimes its just better to throw money/hardware at it

On Tue, Jun 21, 2022 at 9:34 AM Markus Jelsma <ma...@openindex.io>
wrote:

> We have never seen one of our Solr instances go down due to running out of
> disk space. Instead, ours sometimes go down due to OutOfMemory when there
> is a sudden extreme spike in activity. If that happens, Solr gets stopped.
>
> Our daemons are all controlled by systemd, they are then restarted
> automatically and usually join the cluster again without issues. For us, we
> have less work because of the automatic restarts, but only in case of these
> OutOfMemory errors. As said, this does not work for disk space issues.
>
> Op di 21 jun. 2022 om 14:40 schreef Dave <ha...@gmail.com>:
>
> > In my experience if solr goes down it’s because it ran out of disk space,
> > so if you automatically just bring it back up again it will just go down
> > again. There are simple bash scripts you can make to run for standalone
> > solr that will do what you want, you just need to be sure they destroy
> and
> > child solr processes that may still be lingering around
> >
> > > On Jun 21, 2022, at 8:16 AM, Poorna Murali <po...@gmail.com>
> > wrote:
> > >
> > > Hi All,
> > >
> > > If we have an alerting mechanism which says if a solr node is  down, is
> > it
> > > possible to configure the alert with some action which will restart
> that
> > > solr node and recover automatically?
> > > Please advise.
> > >
> > > Thanks,
> > > Poorna
> >
>

Re: Auto recovery of solr

[Markus Jelsma <ma...@openindex.io>]

We have never seen one of our Solr instances go down due to running out of
disk space. Instead, ours sometimes go down due to OutOfMemory when there
is a sudden extreme spike in activity. If that happens, Solr gets stopped.

Our daemons are all controlled by systemd, they are then restarted
automatically and usually join the cluster again without issues. For us, we
have less work because of the automatic restarts, but only in case of these
OutOfMemory errors. As said, this does not work for disk space issues.

Op di 21 jun. 2022 om 14:40 schreef Dave <ha...@gmail.com>:

> In my experience if solr goes down it’s because it ran out of disk space,
> so if you automatically just bring it back up again it will just go down
> again. There are simple bash scripts you can make to run for standalone
> solr that will do what you want, you just need to be sure they destroy and
> child solr processes that may still be lingering around
>
> > On Jun 21, 2022, at 8:16 AM, Poorna Murali <po...@gmail.com>
> wrote:
> >
> > Hi All,
> >
> > If we have an alerting mechanism which says if a solr node is  down, is
> it
> > possible to configure the alert with some action which will restart that
> > solr node and recover automatically?
> > Please advise.
> >
> > Thanks,
> > Poorna
>

Re: Auto recovery of solr

[Dave <ha...@gmail.com>]

In my experience if solr goes down it’s because it ran out of disk space, so if you automatically just bring it back up again it will just go down again. There are simple bash scripts you can make to run for standalone solr that will do what you want, you just need to be sure they destroy and child solr processes that may still be lingering around

> On Jun 21, 2022, at 8:16 AM, Poorna Murali <po...@gmail.com> wrote:
> 
> Hi All,
> 
> If we have an alerting mechanism which says if a solr node is  down, is it
> possible to configure the alert with some action which will restart that
> solr node and recover automatically?
> Please advise.
> 
> Thanks,
> Poorna

Re: Auto recovery of solr

["Mark H. Wood" <mw...@iupui.edu>]

On Tue, Jun 21, 2022 at 04:24:15PM +0530, Poorna Murali wrote:
> If we have an alerting mechanism which says if a solr node is  down, is it
> possible to configure the alert with some action which will restart that
> solr node and recover automatically?
> Please advise.

Until I figure out why Solr crashes from time to time, I have it
started by Monit, which watches the process and will restart it if it
disappears.  We were already using Monit for other stuff so it was
simple to just add a configuration for Solr.

https://mmonit.com/monit/

Whether your alerting mechanism can do similar things, depends on that
mechanism.

But I agree that the best solution will be to find and fix the problem
so that the process monitor has nothing to do.

-- 
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu