You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Luis Carlos Guerrero Covo <lc...@gmail.com> on 2013/07/15 18:21:56 UTC
Clearing old nodes from zookeper without restarting solrcloud cluster
Hi,
Is there an easy way to clear zookeeper of all offline solr nodes without
restarting the cluster? We are having some stability issues and we think it
maybe due to the leader querying old offline nodes.
thank you,
Luis Guerrero
Re: Clearing old nodes from zookeper without restarting solrcloud cluster
Posted by Luis Carlos Guerrero Covo <lc...@gmail.com>.
Hey andre, that isn't a possibility for us right now since we are
terminating nodes using aws autoscaling policies. We'll have to either
change our policies so that we can have some kind of graceful shutdown
where we get the possibility to unload cores or update zookeeper's cluster
state every once in a while to clear old offline nodes. Thanks for the help!
On Wed, Jul 17, 2013 at 2:23 AM, Andre Bois-Crettez
<an...@kelkoo.com>wrote:
> Indeed we are using UNLOAD of cores before shutting down extra replica
> nodes, works well but already said, it needs such nodes to be up.
> Once UNLOADed it is possible to stop them, works well for our use case.
>
> But if nodes are already down, maybe it is possible to manually create
> and upload a cleaned /clusterstate.json to Zookeeper ?
>
>
> André
>
>
> On 07/16/2013 11:18 PM, Marcin Rzewucki wrote:
>
>> Unloading a core is the known way to unregister a solr node in zookeeper
>> (and not use for further querying). It works for me. If you didn't do that
>> like this, unused nodes may remain in the cluster state and Solr may try
>> to
>> use them without a success. I'd suggest to start some machine with the old
>> name, run solr, join the cluster for a while, unload a core to unregister
>> it from the cluster and shutdown host at the end. This way you could have
>> clear cluster state.
>>
>>
>>
>> On 16 July 2013 14:41, Luis Carlos Guerrero Covo
>> <lc...@gmail.com>**wrote:
>>
>> Thanks, I was actually asking about deleting nodes from the cluster state
>>> not cores, unless you can unload cores specific to an already offline
>>> node
>>> from zookeeper.
>>>
>>> --
>>> André Bois-Crettez
>>>
>>> Search technology, Kelkoo
>>> http://www.kelkoo.com/
>>>
>>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 8, rue du Sentier 75002 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>
--
Luis Carlos Guerrero Covo
M.S. Computer Engineering
(57) 3183542047
Re: Clearing old nodes from zookeper without restarting solrcloud
cluster
Posted by Andre Bois-Crettez <an...@kelkoo.com>.
Indeed we are using UNLOAD of cores before shutting down extra replica
nodes, works well but already said, it needs such nodes to be up.
Once UNLOADed it is possible to stop them, works well for our use case.
But if nodes are already down, maybe it is possible to manually create
and upload a cleaned /clusterstate.json to Zookeeper ?
André
On 07/16/2013 11:18 PM, Marcin Rzewucki wrote:
> Unloading a core is the known way to unregister a solr node in zookeeper
> (and not use for further querying). It works for me. If you didn't do that
> like this, unused nodes may remain in the cluster state and Solr may try to
> use them without a success. I'd suggest to start some machine with the old
> name, run solr, join the cluster for a while, unload a core to unregister
> it from the cluster and shutdown host at the end. This way you could have
> clear cluster state.
>
>
>
> On 16 July 2013 14:41, Luis Carlos Guerrero Covo
> <lc...@gmail.com>wrote:
>
>> Thanks, I was actually asking about deleting nodes from the cluster state
>> not cores, unless you can unload cores specific to an already offline node
>> from zookeeper.
>>
>> --
>> André Bois-Crettez
>>
>> Search technology, Kelkoo
>> http://www.kelkoo.com/
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris
Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.
Re: Clearing old nodes from zookeper without restarting solrcloud cluster
Posted by Marcin Rzewucki <mr...@gmail.com>.
Unloading a core is the known way to unregister a solr node in zookeeper
(and not use for further querying). It works for me. If you didn't do that
like this, unused nodes may remain in the cluster state and Solr may try to
use them without a success. I'd suggest to start some machine with the old
name, run solr, join the cluster for a while, unload a core to unregister
it from the cluster and shutdown host at the end. This way you could have
clear cluster state.
On 16 July 2013 14:41, Luis Carlos Guerrero Covo
<lc...@gmail.com>wrote:
> Thanks, I was actually asking about deleting nodes from the cluster state
> not cores, unless you can unload cores specific to an already offline node
> from zookeeper.
>
>
> On Tue, Jul 16, 2013 at 1:55 AM, Marcin Rzewucki <mrzewucki@gmail.com
> >wrote:
>
> > Hi,
> >
> > You should use CoreAdmin API (or Solr Admin page) and UNLOAD unneeded
> > cores. This will unregister them from the zookeeper (cluster state will
> be
> > updated), so they won't be used for querying any longer. Solrcloud
> restart
> > is not needed in this case.
> >
> > Regards.
> >
> >
> > On 16 July 2013 06:18, Ali, Saqib <do...@gmail.com> wrote:
> >
> > > Hello Luis,
> > >
> > > I don't think that is possible. If you delete clusterstate.json from
> > > zookeeper, you will need to restart the nodes...... I could be very
> wrong
> > > about this....
> > >
> > > Saqib
> > >
> > >
> > > On Mon, Jul 15, 2013 at 8:50 PM, Luis Carlos Guerrero Covo <
> > > lcguerrerocovo@gmail.com> wrote:
> > >
> > > > I know that you can clear zookeeper's data directoy using the CLI
> with
> > > the
> > > > clear command, I just want to know if its possible to update the
> > > cluster's
> > > > state without wiping everything out. Anyone have any
> ideas/suggestions?
> > > >
> > > >
> > > > On Mon, Jul 15, 2013 at 11:21 AM, Luis Carlos Guerrero Covo <
> > > > lcguerrerocovo@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Is there an easy way to clear zookeeper of all offline solr nodes
> > > without
> > > > > restarting the cluster? We are having some stability issues and we
> > > think
> > > > it
> > > > > maybe due to the leader querying old offline nodes.
> > > > >
> > > > > thank you,
> > > > >
> > > > > Luis Guerrero
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Luis Carlos Guerrero Covo
> > > > M.S. Computer Engineering
> > > > (57) 3183542047
> > > >
> > >
> >
>
>
>
> --
> Luis Carlos Guerrero Covo
> M.S. Computer Engineering
> (57) 3183542047
>
Re: Clearing old nodes from zookeper without restarting solrcloud cluster
Posted by Luis Carlos Guerrero Covo <lc...@gmail.com>.
Thanks, I was actually asking about deleting nodes from the cluster state
not cores, unless you can unload cores specific to an already offline node
from zookeeper.
On Tue, Jul 16, 2013 at 1:55 AM, Marcin Rzewucki <mr...@gmail.com>wrote:
> Hi,
>
> You should use CoreAdmin API (or Solr Admin page) and UNLOAD unneeded
> cores. This will unregister them from the zookeeper (cluster state will be
> updated), so they won't be used for querying any longer. Solrcloud restart
> is not needed in this case.
>
> Regards.
>
>
> On 16 July 2013 06:18, Ali, Saqib <do...@gmail.com> wrote:
>
> > Hello Luis,
> >
> > I don't think that is possible. If you delete clusterstate.json from
> > zookeeper, you will need to restart the nodes...... I could be very wrong
> > about this....
> >
> > Saqib
> >
> >
> > On Mon, Jul 15, 2013 at 8:50 PM, Luis Carlos Guerrero Covo <
> > lcguerrerocovo@gmail.com> wrote:
> >
> > > I know that you can clear zookeeper's data directoy using the CLI with
> > the
> > > clear command, I just want to know if its possible to update the
> > cluster's
> > > state without wiping everything out. Anyone have any ideas/suggestions?
> > >
> > >
> > > On Mon, Jul 15, 2013 at 11:21 AM, Luis Carlos Guerrero Covo <
> > > lcguerrerocovo@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Is there an easy way to clear zookeeper of all offline solr nodes
> > without
> > > > restarting the cluster? We are having some stability issues and we
> > think
> > > it
> > > > maybe due to the leader querying old offline nodes.
> > > >
> > > > thank you,
> > > >
> > > > Luis Guerrero
> > > >
> > >
> > >
> > >
> > > --
> > > Luis Carlos Guerrero Covo
> > > M.S. Computer Engineering
> > > (57) 3183542047
> > >
> >
>
--
Luis Carlos Guerrero Covo
M.S. Computer Engineering
(57) 3183542047
Re: Clearing old nodes from zookeper without restarting solrcloud cluster
Posted by Marcin Rzewucki <mr...@gmail.com>.
Hi,
You should use CoreAdmin API (or Solr Admin page) and UNLOAD unneeded
cores. This will unregister them from the zookeeper (cluster state will be
updated), so they won't be used for querying any longer. Solrcloud restart
is not needed in this case.
Regards.
On 16 July 2013 06:18, Ali, Saqib <do...@gmail.com> wrote:
> Hello Luis,
>
> I don't think that is possible. If you delete clusterstate.json from
> zookeeper, you will need to restart the nodes...... I could be very wrong
> about this....
>
> Saqib
>
>
> On Mon, Jul 15, 2013 at 8:50 PM, Luis Carlos Guerrero Covo <
> lcguerrerocovo@gmail.com> wrote:
>
> > I know that you can clear zookeeper's data directoy using the CLI with
> the
> > clear command, I just want to know if its possible to update the
> cluster's
> > state without wiping everything out. Anyone have any ideas/suggestions?
> >
> >
> > On Mon, Jul 15, 2013 at 11:21 AM, Luis Carlos Guerrero Covo <
> > lcguerrerocovo@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Is there an easy way to clear zookeeper of all offline solr nodes
> without
> > > restarting the cluster? We are having some stability issues and we
> think
> > it
> > > maybe due to the leader querying old offline nodes.
> > >
> > > thank you,
> > >
> > > Luis Guerrero
> > >
> >
> >
> >
> > --
> > Luis Carlos Guerrero Covo
> > M.S. Computer Engineering
> > (57) 3183542047
> >
>
Re: Clearing old nodes from zookeper without restarting solrcloud cluster
Posted by "Ali, Saqib" <do...@gmail.com>.
Hello Luis,
I don't think that is possible. If you delete clusterstate.json from
zookeeper, you will need to restart the nodes...... I could be very wrong
about this....
Saqib
On Mon, Jul 15, 2013 at 8:50 PM, Luis Carlos Guerrero Covo <
lcguerrerocovo@gmail.com> wrote:
> I know that you can clear zookeeper's data directoy using the CLI with the
> clear command, I just want to know if its possible to update the cluster's
> state without wiping everything out. Anyone have any ideas/suggestions?
>
>
> On Mon, Jul 15, 2013 at 11:21 AM, Luis Carlos Guerrero Covo <
> lcguerrerocovo@gmail.com> wrote:
>
> > Hi,
> >
> > Is there an easy way to clear zookeeper of all offline solr nodes without
> > restarting the cluster? We are having some stability issues and we think
> it
> > maybe due to the leader querying old offline nodes.
> >
> > thank you,
> >
> > Luis Guerrero
> >
>
>
>
> --
> Luis Carlos Guerrero Covo
> M.S. Computer Engineering
> (57) 3183542047
>
Re: Clearing old nodes from zookeper without restarting solrcloud cluster
Posted by Luis Carlos Guerrero Covo <lc...@gmail.com>.
I know that you can clear zookeeper's data directoy using the CLI with the
clear command, I just want to know if its possible to update the cluster's
state without wiping everything out. Anyone have any ideas/suggestions?
On Mon, Jul 15, 2013 at 11:21 AM, Luis Carlos Guerrero Covo <
lcguerrerocovo@gmail.com> wrote:
> Hi,
>
> Is there an easy way to clear zookeeper of all offline solr nodes without
> restarting the cluster? We are having some stability issues and we think it
> maybe due to the leader querying old offline nodes.
>
> thank you,
>
> Luis Guerrero
>
--
Luis Carlos Guerrero Covo
M.S. Computer Engineering
(57) 3183542047