Re: Troubleshooting solr errors

[Shawn Heisey <ap...@elyograg.org>]

On 4/25/2017 12:05 PM, Daniel Miller wrote:
> The problem isn't a particular email message - I get a cascade of
> those errors (every time a new message is received) once the server
> "breaks".  The fix is to restart the server.  I did find a Java heap
> error in the log - so I've increased the memory allocation (now to
> -Xms512m -Xmx2048m).  I had thought that a heap failure would result
> in "simple" termination - and that systemd would restart it
> appropriately - but obviously I'm missing something.

Erick covered some of this already:

The init script that the service installer script installs on a
non-windows system can start Solr, but it will not automatically restart
it if it dies.  That would require you to write something special,
probably a very custom systemd service specification, rather than use
the init script.  Automatically restarting on death is not a good idea
-- it is VERY likely that whatever caused the death is going to happen
again.

Another detail, at least on non-windows systems, is that recent Solr
versions include a script that kills the process on OutOfMemoryError
(OOME).  This is done because program operation is completely
unpredictable after that error occurs -- we have no way of knowing what
Solr will do.  There's an issue in Jira to add OOME killing to the
Windows script.

FYI, the stacktrace from an OutOfMemoryError regarding the heap is
highly unlikely to give you anything useful about why the process ran
out of memory, since *any* memory allocation in any software running in
the JVM can trigger the error.

Other errors besides OOME should never terminate Solr unless there's an
enormous bug somewhere.  That bug might be in Java itself, or even the OS.

Thanks,
Shawn