Stop Solr Node (in distress)?

[Brian Narsi <bn...@gmail.com>]

We have had a situation where Solr node was in distress due to hard drive
being full and the queries became very slow. Since our Solr cluster has two
nodes with indexes being fully available on both the nodes, we think that
one good solution would be to just stop the Solr instance on a distressed
node. That way the other node will continue to serve queries until the
distressed node is fixed.

Can folks please share their experiences in such a situation?

What are some good ways to handle this? How/when should we detect that the
host (linux) is in distress and then stop the solr node?

Thanks,

Re: Stop Solr Node (in distress)?

[Susheel Kumar <su...@gmail.com>]

You can detect your node in distress by using different tools like Monit (
https://mmonit.com/monit/ open source) which alerts by email when disk
space utilization goes above a threshold or memory usage goes above
threshold etc.  You can also have Monit to restart Solr process in some of
those situations etc.

Wiley is another commercial solution.

Thanks,
Susheel

On Mon, Dec 19, 2016 at 9:07 PM, Erick Erickson <er...@gmail.com>
wrote:

> The first question is _why_ is your disk full? Older versions of Solr
> could, for instance, accumulate solr console log files forever. If
> that's the case, just stop the Solr instance on that node, remove the
> Solr log files, fix the log4j.properties file to not append to console
> forever and you're done.
>
> If it's more complicated, I'd
> 1> Spin up a new Solr node with adequate disk space.
> 2> use the Collections API ADDREPLICA command to add new replicas to
> this node. The indexes will be synchronized etc.
> 3> once <2> is complete and the newly added replicas are green, issue
> a DELETEREPLICA on the replicas on the Solr node in distress.
>
> You could issue the DELETEREPLICA  before creating a new Solr instance
> on a new node, but doing it in the order I indicated has the fewest
> chances for doing something awful like deleting the replica on the
> good node by mistake without having a healthy replica in your
> cluster....
>
> Best,
> Erick
>
> On Mon, Dec 19, 2016 at 5:56 PM, Brian Narsi <bn...@gmail.com> wrote:
> > We have had a situation where Solr node was in distress due to hard drive
> > being full and the queries became very slow. Since our Solr cluster has
> two
> > nodes with indexes being fully available on both the nodes, we think that
> > one good solution would be to just stop the Solr instance on a distressed
> > node. That way the other node will continue to serve queries until the
> > distressed node is fixed.
> >
> > Can folks please share their experiences in such a situation?
> >
> > What are some good ways to handle this? How/when should we detect that
> the
> > host (linux) is in distress and then stop the solr node?
> >
> > Thanks,
>

Re: Stop Solr Node (in distress)?

[Erick Erickson <er...@gmail.com>]

The first question is _why_ is your disk full? Older versions of Solr
could, for instance, accumulate solr console log files forever. If
that's the case, just stop the Solr instance on that node, remove the
Solr log files, fix the log4j.properties file to not append to console
forever and you're done.

If it's more complicated, I'd
1> Spin up a new Solr node with adequate disk space.
2> use the Collections API ADDREPLICA command to add new replicas to
this node. The indexes will be synchronized etc.
3> once <2> is complete and the newly added replicas are green, issue
a DELETEREPLICA on the replicas on the Solr node in distress.

You could issue the DELETEREPLICA  before creating a new Solr instance
on a new node, but doing it in the order I indicated has the fewest
chances for doing something awful like deleting the replica on the
good node by mistake without having a healthy replica in your
cluster....

Best,
Erick

On Mon, Dec 19, 2016 at 5:56 PM, Brian Narsi <bn...@gmail.com> wrote:
> We have had a situation where Solr node was in distress due to hard drive
> being full and the queries became very slow. Since our Solr cluster has two
> nodes with indexes being fully available on both the nodes, we think that
> one good solution would be to just stop the Solr instance on a distressed
> node. That way the other node will continue to serve queries until the
> distressed node is fixed.
>
> Can folks please share their experiences in such a situation?
>
> What are some good ways to handle this? How/when should we detect that the
> host (linux) is in distress and then stop the solr node?
>
> Thanks,

Re: Stop Solr Node (in distress)?

[Charlie Hull <ch...@flax.co.uk>]

On 20/12/2016 01:56, Brian Narsi wrote:
> We have had a situation where Solr node was in distress due to hard drive
> being full and the queries became very slow. Since our Solr cluster has two
> nodes with indexes being fully available on both the nodes, we think that
> one good solution would be to just stop the Solr instance on a distressed
> node. That way the other node will continue to serve queries until the
> distressed node is fixed.
>
> Can folks please share their experiences in such a situation?
>
> What are some good ways to handle this? How/when should we detect that the
> host (linux) is in distress and then stop the solr node?
>
> Thanks,
>
We ran into some disk space issues with a recent project, and built 
something that blocks when space is low: you might find it useful. 
Details at 
http://www.flax.co.uk/blog/2016/04/21/running-disk-space-elasticsearch-solr/

Cheers

Charlie


-- 
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk