You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bill Au <bi...@gmail.com> on 2013/01/02 23:51:45 UTC
Solr 4.0 SolrCloud with AWS Auto Scaling
Is anyone running Solr 4.0 SolrCloud with AWS auto scaling?
My concern is that as AWS auto scaling add and remove instances to
SolrCloud, the number of nodes in SolrCloud Zookeeper config will grow
indefinitely as removed instances will never be used again. AWS auto
scaling will keep on adding new instances, and there is no way to remove
them from Zookeeper, right? What's the effect of have all these phantom
nodes?
Bill
Re: Solr 4.0 SolrCloud with AWS Auto Scaling
Posted by Bill Au <bi...@gmail.com>.
With AWS auto scaling, one can specify a minimum number of instances for an
auto scaling group. So there should never be an insufficient number of
replicas. Once can also specify a termination policy so that the newly
added nodes are removed first.
But with SolrCloud as long as there are enough replicas there is no wrong
node to remove, right?
AWS Beanstalk seems to be a wrapper for AWS auto scaling and other AWS
elastic services. I am not sure if it offers the detail-grained control
that you have when using auto scaling directly.
On Wed, Jan 2, 2013 at 11:14 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> We've considered using AWS Beanstalk (hmm, what's the difference between
> AWS auto scaling and elastic beanstalk? not sure.) for search-lucene.com ,
> but the idea of something adding and removing nodes seems scary. The
> scariest part to me is automatic removal of wrong nodes that ends up in
> data loss or insufficient number of replicas.
>
> But if somebody has done thing and has written up a how-to, I'd love to see
> it!
>
> Otis
> --
> Solr & ElasticSearch Support
> http://sematext.com/
>
>
>
>
>
> On Wed, Jan 2, 2013 at 5:51 PM, Bill Au <bi...@gmail.com> wrote:
>
> > Is anyone running Solr 4.0 SolrCloud with AWS auto scaling?
> >
> > My concern is that as AWS auto scaling add and remove instances to
> > SolrCloud, the number of nodes in SolrCloud Zookeeper config will grow
> > indefinitely as removed instances will never be used again. AWS auto
> > scaling will keep on adding new instances, and there is no way to remove
> > them from Zookeeper, right? What's the effect of have all these phantom
> > nodes?
> >
> > Bill
> >
>
Re: Solr 4.0 SolrCloud with AWS Auto Scaling
Posted by Otis Gospodnetic <ot...@gmail.com>.
We've considered using AWS Beanstalk (hmm, what's the difference between
AWS auto scaling and elastic beanstalk? not sure.) for search-lucene.com ,
but the idea of something adding and removing nodes seems scary. The
scariest part to me is automatic removal of wrong nodes that ends up in
data loss or insufficient number of replicas.
But if somebody has done thing and has written up a how-to, I'd love to see
it!
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Wed, Jan 2, 2013 at 5:51 PM, Bill Au <bi...@gmail.com> wrote:
> Is anyone running Solr 4.0 SolrCloud with AWS auto scaling?
>
> My concern is that as AWS auto scaling add and remove instances to
> SolrCloud, the number of nodes in SolrCloud Zookeeper config will grow
> indefinitely as removed instances will never be used again. AWS auto
> scaling will keep on adding new instances, and there is no way to remove
> them from Zookeeper, right? What's the effect of have all these phantom
> nodes?
>
> Bill
>
Re: Solr 4.0 SolrCloud with AWS Auto Scaling
Posted by Bill Au <bi...@gmail.com>.
thanks for pointing me to Solr's Zookeeper servlet. I will look at the
source to see how I can use to fulfill my needs.
Bill
On Thu, Jan 3, 2013 at 6:43 PM, Mark Miller <ma...@gmail.com> wrote:
> Technically, you want to make sure zookeeper reports the node as live and
> active.
>
> You could use the same api that the UI uses for that - the
> localhost:port/solr/zookeeper (I think?) servlet.
>
> If you can't reach it for a node, it's obviously down - if you can reach
> it, parse the json and see if it notes the node as active?
>
> Not quite as clean as you'd like prob. Might be worth a JIRA issue to look
> at further options.
>
> - Mark
>
> On Jan 3, 2013, at 5:54 PM, Bill Au <bi...@gmail.com> wrote:
>
> > Thanks, Mark.
> >
> > That does remove the node. And it seems to do so permanently. Even
> when I
> > restart Solr after unloading, it does not join the SolrCloud cluster.
> And
> > I can get it to re-join the cluster by creating the core.
> >
> > Anyone know if there is an API to determine the state of a node. When
> AWS
> > auto scaling add a new node, I need to make sure it has before active
> > before I enable it in the load balancer.
> >
> > Bill
> >
> >
> >
> >
> > On Thu, Jan 3, 2013 at 9:10 AM, Mark Miller <ma...@gmail.com>
> wrote:
> >
> >>
> >> http://wiki.apache.org/solr/CoreAdmin#UNLOAD
> >>
> >> - Mark
> >>
> >> On Jan 3, 2013, at 9:06 AM, Bill Au <bi...@gmail.com> wrote:
> >>
> >>> Mark,
> >>> What do you mean by "unload them"?
> >>>
> >>> I am using an AWS load balancer with my auto scaling group in stead of
> >>> using Solr's built-in load balancer. I am no sharding my index. I am
> >>> using SolrCloud for replication only. I am doing local search on each
> >>> instance and sending all updates to the shard leader directly because I
> >>> want to minimize traffic between nodes during search and update
> >>>
> >>> Bill
> >>>
> >>>
> >>> On Wed, Jan 2, 2013 at 6:47 PM, Mark Miller <ma...@gmail.com>
> >> wrote:
> >>>
> >>>>
> >>>> On Jan 2, 2013, at 5:51 PM, Bill Au <bi...@gmail.com> wrote:
> >>>>
> >>>>> Is anyone running Solr 4.0 SolrCloud with AWS auto scaling?
> >>>>>
> >>>>> My concern is that as AWS auto scaling add and remove instances to
> >>>>> SolrCloud, the number of nodes in SolrCloud Zookeeper config will
> grow
> >>>>> indefinitely as removed instances will never be used again. AWS auto
> >>>>> scaling will keep on adding new instances, and there is no way to
> >> remove
> >>>>> them from Zookeeper, right?
> >>>>
> >>>> You can unload them and that removes them.
> >>>>
> >>>>> What's the effect of have all these phantom
> >>>>> nodes?
> >>>>
> >>>> Unless they are only replicas, they would need to be removed.
> >>>>
> >>>> Also, unless you are using elastic ips,
> >>>> https://issues.apache.org/jira/browse/SOLR-4078 may be of interest.
> >>>>
> >>>> - Mark
> >>
> >>
>
>
Re: Solr 4.0 SolrCloud with AWS Auto Scaling
Posted by Mark Miller <ma...@gmail.com>.
Technically, you want to make sure zookeeper reports the node as live and active.
You could use the same api that the UI uses for that - the localhost:port/solr/zookeeper (I think?) servlet.
If you can't reach it for a node, it's obviously down - if you can reach it, parse the json and see if it notes the node as active?
Not quite as clean as you'd like prob. Might be worth a JIRA issue to look at further options.
- Mark
On Jan 3, 2013, at 5:54 PM, Bill Au <bi...@gmail.com> wrote:
> Thanks, Mark.
>
> That does remove the node. And it seems to do so permanently. Even when I
> restart Solr after unloading, it does not join the SolrCloud cluster. And
> I can get it to re-join the cluster by creating the core.
>
> Anyone know if there is an API to determine the state of a node. When AWS
> auto scaling add a new node, I need to make sure it has before active
> before I enable it in the load balancer.
>
> Bill
>
>
>
>
> On Thu, Jan 3, 2013 at 9:10 AM, Mark Miller <ma...@gmail.com> wrote:
>
>>
>> http://wiki.apache.org/solr/CoreAdmin#UNLOAD
>>
>> - Mark
>>
>> On Jan 3, 2013, at 9:06 AM, Bill Au <bi...@gmail.com> wrote:
>>
>>> Mark,
>>> What do you mean by "unload them"?
>>>
>>> I am using an AWS load balancer with my auto scaling group in stead of
>>> using Solr's built-in load balancer. I am no sharding my index. I am
>>> using SolrCloud for replication only. I am doing local search on each
>>> instance and sending all updates to the shard leader directly because I
>>> want to minimize traffic between nodes during search and update
>>>
>>> Bill
>>>
>>>
>>> On Wed, Jan 2, 2013 at 6:47 PM, Mark Miller <ma...@gmail.com>
>> wrote:
>>>
>>>>
>>>> On Jan 2, 2013, at 5:51 PM, Bill Au <bi...@gmail.com> wrote:
>>>>
>>>>> Is anyone running Solr 4.0 SolrCloud with AWS auto scaling?
>>>>>
>>>>> My concern is that as AWS auto scaling add and remove instances to
>>>>> SolrCloud, the number of nodes in SolrCloud Zookeeper config will grow
>>>>> indefinitely as removed instances will never be used again. AWS auto
>>>>> scaling will keep on adding new instances, and there is no way to
>> remove
>>>>> them from Zookeeper, right?
>>>>
>>>> You can unload them and that removes them.
>>>>
>>>>> What's the effect of have all these phantom
>>>>> nodes?
>>>>
>>>> Unless they are only replicas, they would need to be removed.
>>>>
>>>> Also, unless you are using elastic ips,
>>>> https://issues.apache.org/jira/browse/SOLR-4078 may be of interest.
>>>>
>>>> - Mark
>>
>>
Re: Solr 4.0 SolrCloud with AWS Auto Scaling
Posted by Bill Au <bi...@gmail.com>.
Thanks, Mark.
That does remove the node. And it seems to do so permanently. Even when I
restart Solr after unloading, it does not join the SolrCloud cluster. And
I can get it to re-join the cluster by creating the core.
Anyone know if there is an API to determine the state of a node. When AWS
auto scaling add a new node, I need to make sure it has before active
before I enable it in the load balancer.
Bill
On Thu, Jan 3, 2013 at 9:10 AM, Mark Miller <ma...@gmail.com> wrote:
>
> http://wiki.apache.org/solr/CoreAdmin#UNLOAD
>
> - Mark
>
> On Jan 3, 2013, at 9:06 AM, Bill Au <bi...@gmail.com> wrote:
>
> > Mark,
> > What do you mean by "unload them"?
> >
> > I am using an AWS load balancer with my auto scaling group in stead of
> > using Solr's built-in load balancer. I am no sharding my index. I am
> > using SolrCloud for replication only. I am doing local search on each
> > instance and sending all updates to the shard leader directly because I
> > want to minimize traffic between nodes during search and update
> >
> > Bill
> >
> >
> > On Wed, Jan 2, 2013 at 6:47 PM, Mark Miller <ma...@gmail.com>
> wrote:
> >
> >>
> >> On Jan 2, 2013, at 5:51 PM, Bill Au <bi...@gmail.com> wrote:
> >>
> >>> Is anyone running Solr 4.0 SolrCloud with AWS auto scaling?
> >>>
> >>> My concern is that as AWS auto scaling add and remove instances to
> >>> SolrCloud, the number of nodes in SolrCloud Zookeeper config will grow
> >>> indefinitely as removed instances will never be used again. AWS auto
> >>> scaling will keep on adding new instances, and there is no way to
> remove
> >>> them from Zookeeper, right?
> >>
> >> You can unload them and that removes them.
> >>
> >>> What's the effect of have all these phantom
> >>> nodes?
> >>
> >> Unless they are only replicas, they would need to be removed.
> >>
> >> Also, unless you are using elastic ips,
> >> https://issues.apache.org/jira/browse/SOLR-4078 may be of interest.
> >>
> >> - Mark
>
>
Re: Solr 4.0 SolrCloud with AWS Auto Scaling
Posted by Mark Miller <ma...@gmail.com>.
http://wiki.apache.org/solr/CoreAdmin#UNLOAD
- Mark
On Jan 3, 2013, at 9:06 AM, Bill Au <bi...@gmail.com> wrote:
> Mark,
> What do you mean by "unload them"?
>
> I am using an AWS load balancer with my auto scaling group in stead of
> using Solr's built-in load balancer. I am no sharding my index. I am
> using SolrCloud for replication only. I am doing local search on each
> instance and sending all updates to the shard leader directly because I
> want to minimize traffic between nodes during search and update
>
> Bill
>
>
> On Wed, Jan 2, 2013 at 6:47 PM, Mark Miller <ma...@gmail.com> wrote:
>
>>
>> On Jan 2, 2013, at 5:51 PM, Bill Au <bi...@gmail.com> wrote:
>>
>>> Is anyone running Solr 4.0 SolrCloud with AWS auto scaling?
>>>
>>> My concern is that as AWS auto scaling add and remove instances to
>>> SolrCloud, the number of nodes in SolrCloud Zookeeper config will grow
>>> indefinitely as removed instances will never be used again. AWS auto
>>> scaling will keep on adding new instances, and there is no way to remove
>>> them from Zookeeper, right?
>>
>> You can unload them and that removes them.
>>
>>> What's the effect of have all these phantom
>>> nodes?
>>
>> Unless they are only replicas, they would need to be removed.
>>
>> Also, unless you are using elastic ips,
>> https://issues.apache.org/jira/browse/SOLR-4078 may be of interest.
>>
>> - Mark
Re: Solr 4.0 SolrCloud with AWS Auto Scaling
Posted by Bill Au <bi...@gmail.com>.
Mark,
What do you mean by "unload them"?
I am using an AWS load balancer with my auto scaling group in stead of
using Solr's built-in load balancer. I am no sharding my index. I am
using SolrCloud for replication only. I am doing local search on each
instance and sending all updates to the shard leader directly because I
want to minimize traffic between nodes during search and update
Bill
On Wed, Jan 2, 2013 at 6:47 PM, Mark Miller <ma...@gmail.com> wrote:
>
> On Jan 2, 2013, at 5:51 PM, Bill Au <bi...@gmail.com> wrote:
>
> > Is anyone running Solr 4.0 SolrCloud with AWS auto scaling?
> >
> > My concern is that as AWS auto scaling add and remove instances to
> > SolrCloud, the number of nodes in SolrCloud Zookeeper config will grow
> > indefinitely as removed instances will never be used again. AWS auto
> > scaling will keep on adding new instances, and there is no way to remove
> > them from Zookeeper, right?
>
> You can unload them and that removes them.
>
> > What's the effect of have all these phantom
> > nodes?
>
> Unless they are only replicas, they would need to be removed.
>
> Also, unless you are using elastic ips,
> https://issues.apache.org/jira/browse/SOLR-4078 may be of interest.
>
> - Mark
Re: Solr 4.0 SolrCloud with AWS Auto Scaling
Posted by Mark Miller <ma...@gmail.com>.
On Jan 2, 2013, at 5:51 PM, Bill Au <bi...@gmail.com> wrote:
> Is anyone running Solr 4.0 SolrCloud with AWS auto scaling?
>
> My concern is that as AWS auto scaling add and remove instances to
> SolrCloud, the number of nodes in SolrCloud Zookeeper config will grow
> indefinitely as removed instances will never be used again. AWS auto
> scaling will keep on adding new instances, and there is no way to remove
> them from Zookeeper, right?
You can unload them and that removes them.
> What's the effect of have all these phantom
> nodes?
Unless they are only replicas, they would need to be removed.
Also, unless you are using elastic ips, https://issues.apache.org/jira/browse/SOLR-4078 may be of interest.
- Mark