You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sujatha Arun <su...@gmail.com> on 2012/04/28 09:54:20 UTC
Scaling Solr - Suggestions !!
Hello,
*Background* :For each of our customers, we create 3 solr webapps with
different search schema's,serving different search requirements and we
have about 70 customers.So we have about 210 webapps curently .
*Hardware*: Single Server , one JVM , Heap memory 19GB ,Total Ram :32GB ,
Permgen initally 1GB ,now increased to 2GB.
*Solr Indexes* : Most are the order of a few MB ,about 2 big index of
about 3GB each
*Scaling Step 1 *: We saw the permgen value go upto to nearly 850 mb ,when
we created so many webapps ,hence now we are moving to solr cores and we
are going to have about 50 cores per webapp ,bringing the number of webapps
to about 5 . We want to distribute the cores with multiple webapps to avoid
a single point of failure.
*Requirement* :
- We need to only scale the cores horizontally ,whose index sizes are
big.
- We also require permission based search for each webapp ,would solr
NRT fit our needs ,where we can index the permission into the document
,which would mean there would be frequent addition and deletion of
permissions to the documents across cores.
- We also require automatic fail over
What technology would be ideal fit given Solr Cloud ,Katta , Solandra
,Lily,Elastic Search etc [Preferably Open source] [ We would be required to
maintain many webapps with multicores ] and what about the commercial
offering given out use case
Thanks.
Regards,
Sujatha
Re: Scaling Solr - Suggestions !!
Posted by Sujatha Arun <su...@gmail.com>.
I was copying the indexes from webapp to cores ,when this happened .It
could have been an error from my end ,but just worried that an issue with
one core would reflect on webapp .
Regards
Sujatha
On Mon, Apr 30, 2012 at 7:20 PM, Erick Erickson <er...@gmail.com>wrote:
> I'd get to the root of why indexes are corrupt! This should
> be very unusual. If you're seeing this at all frequently,
> it indicates something is very wrong and starting bunches
> of JVMs up is a band-aid over a much more serious
> problem.
>
> Are you, by chance, doing a kill -9? or other hard-abort?
>
> Best
> Erick
>
> On Mon, Apr 30, 2012 at 12:22 AM, Sujatha Arun <su...@gmail.com>
> wrote:
> > Now the reason ,I have used different webapps instead of a single one for
> > the cores is ,while prototyping ,I discovered that ,when one of the cores
> > index is corrupt ,the entire webapp does not start up and the same must
> be
> > true of "too many open files" etc ,that is to say if there is an issue
> > withe any one core [Schema /index] ,the entire webapp does not start up.
> >
> > Thanks for your suggestion.
> >
> >
> > Regards
> > Sujatha
> >
> >
> >
> >
> >
> >
> >
> > On Sat, Apr 28, 2012 at 6:49 PM, Michael Della Bitta <
> > michael.della.bitta@appinions.com> wrote:
> >
> >> Just my opinion, but I'm not sure I see the value in deploying the cores
> >> to different webapps in a single container on a single machine to avoid
> >> a single point of failure... You still have a single point of failure at
> >> the process level down to the hardware, which when you think about it,
> >> is mostly everything. But perhaps you're at least using more than one
> >> container.
> >>
> >> It sounds to me that the easiest route to scalability for you would be
> >> to add more machines. Unless your cores are particularly complex or your
> >> traffic is heavy, a 3GB core should be no match for a single machine.
> >> And the traffic problem can be solved by replication and load balancing.
> >>
> >> Michael
> >>
> >> On Sat, 2012-04-28 at 13:24 +0530, Sujatha Arun wrote:
> >> > Hello,
> >> >
> >> > *Background* :For each of our customers, we create 3 solr webapps
> with
> >> > different search schema's,serving different search requirements and
> we
> >> > have about 70 customers.So we have about 210 webapps curently .
> >> >
> >> > *Hardware*: Single Server , one JVM , Heap memory 19GB ,Total Ram
> :32GB ,
> >> > Permgen initally 1GB ,now increased to 2GB.
> >> >
> >> > *Solr Indexes* : Most are the order of a few MB ,about 2 big index of
> >> > about 3GB each
> >> >
> >> > *Scaling Step 1 *: We saw the permgen value go upto to nearly 850 mb
> >> ,when
> >> > we created so many webapps ,hence now we are moving to solr cores
> and we
> >> > are going to have about 50 cores per webapp ,bringing the number of
> >> webapps
> >> > to about 5 . We want to distribute the cores with multiple webapps to
> >> avoid
> >> > a single point of failure.
> >> >
> >> >
> >> > *Requirement* :
> >> >
> >> >
> >> > - We need to only scale the cores horizontally ,whose index sizes
> >> are
> >> > big.
> >> > - We also require permission based search for each webapp ,would
> >> solr
> >> > NRT fit our needs ,where we can index the permission into the
> document
> >> > ,which would mean there would be frequent addition and deletion
> of
> >> > permissions to the documents across cores.
> >> > - We also require automatic fail over
> >> >
> >> > What technology would be ideal fit given Solr Cloud ,Katta , Solandra
> >> > ,Lily,Elastic Search etc [Preferably Open source] [ We would be
> required
> >> to
> >> > maintain many webapps with multicores ] and what about the commercial
> >> > offering given out use case
> >> >
> >> > Thanks.
> >> >
> >> > Regards,
> >> > Sujatha
> >>
> >>
> >>
>
Re: Scaling Solr - Suggestions !!
Posted by Erick Erickson <er...@gmail.com>.
I'd get to the root of why indexes are corrupt! This should
be very unusual. If you're seeing this at all frequently,
it indicates something is very wrong and starting bunches
of JVMs up is a band-aid over a much more serious
problem.
Are you, by chance, doing a kill -9? or other hard-abort?
Best
Erick
On Mon, Apr 30, 2012 at 12:22 AM, Sujatha Arun <su...@gmail.com> wrote:
> Now the reason ,I have used different webapps instead of a single one for
> the cores is ,while prototyping ,I discovered that ,when one of the cores
> index is corrupt ,the entire webapp does not start up and the same must be
> true of "too many open files" etc ,that is to say if there is an issue
> withe any one core [Schema /index] ,the entire webapp does not start up.
>
> Thanks for your suggestion.
>
>
> Regards
> Sujatha
>
>
>
>
>
>
>
> On Sat, Apr 28, 2012 at 6:49 PM, Michael Della Bitta <
> michael.della.bitta@appinions.com> wrote:
>
>> Just my opinion, but I'm not sure I see the value in deploying the cores
>> to different webapps in a single container on a single machine to avoid
>> a single point of failure... You still have a single point of failure at
>> the process level down to the hardware, which when you think about it,
>> is mostly everything. But perhaps you're at least using more than one
>> container.
>>
>> It sounds to me that the easiest route to scalability for you would be
>> to add more machines. Unless your cores are particularly complex or your
>> traffic is heavy, a 3GB core should be no match for a single machine.
>> And the traffic problem can be solved by replication and load balancing.
>>
>> Michael
>>
>> On Sat, 2012-04-28 at 13:24 +0530, Sujatha Arun wrote:
>> > Hello,
>> >
>> > *Background* :For each of our customers, we create 3 solr webapps with
>> > different search schema's,serving different search requirements and we
>> > have about 70 customers.So we have about 210 webapps curently .
>> >
>> > *Hardware*: Single Server , one JVM , Heap memory 19GB ,Total Ram :32GB ,
>> > Permgen initally 1GB ,now increased to 2GB.
>> >
>> > *Solr Indexes* : Most are the order of a few MB ,about 2 big index of
>> > about 3GB each
>> >
>> > *Scaling Step 1 *: We saw the permgen value go upto to nearly 850 mb
>> ,when
>> > we created so many webapps ,hence now we are moving to solr cores and we
>> > are going to have about 50 cores per webapp ,bringing the number of
>> webapps
>> > to about 5 . We want to distribute the cores with multiple webapps to
>> avoid
>> > a single point of failure.
>> >
>> >
>> > *Requirement* :
>> >
>> >
>> > - We need to only scale the cores horizontally ,whose index sizes
>> are
>> > big.
>> > - We also require permission based search for each webapp ,would
>> solr
>> > NRT fit our needs ,where we can index the permission into the document
>> > ,which would mean there would be frequent addition and deletion of
>> > permissions to the documents across cores.
>> > - We also require automatic fail over
>> >
>> > What technology would be ideal fit given Solr Cloud ,Katta , Solandra
>> > ,Lily,Elastic Search etc [Preferably Open source] [ We would be required
>> to
>> > maintain many webapps with multicores ] and what about the commercial
>> > offering given out use case
>> >
>> > Thanks.
>> >
>> > Regards,
>> > Sujatha
>>
>>
>>
Re: Scaling Solr - Suggestions !!
Posted by Sujatha Arun <su...@gmail.com>.
Now the reason ,I have used different webapps instead of a single one for
the cores is ,while prototyping ,I discovered that ,when one of the cores
index is corrupt ,the entire webapp does not start up and the same must be
true of "too many open files" etc ,that is to say if there is an issue
withe any one core [Schema /index] ,the entire webapp does not start up.
Thanks for your suggestion.
Regards
Sujatha
On Sat, Apr 28, 2012 at 6:49 PM, Michael Della Bitta <
michael.della.bitta@appinions.com> wrote:
> Just my opinion, but I'm not sure I see the value in deploying the cores
> to different webapps in a single container on a single machine to avoid
> a single point of failure... You still have a single point of failure at
> the process level down to the hardware, which when you think about it,
> is mostly everything. But perhaps you're at least using more than one
> container.
>
> It sounds to me that the easiest route to scalability for you would be
> to add more machines. Unless your cores are particularly complex or your
> traffic is heavy, a 3GB core should be no match for a single machine.
> And the traffic problem can be solved by replication and load balancing.
>
> Michael
>
> On Sat, 2012-04-28 at 13:24 +0530, Sujatha Arun wrote:
> > Hello,
> >
> > *Background* :For each of our customers, we create 3 solr webapps with
> > different search schema's,serving different search requirements and we
> > have about 70 customers.So we have about 210 webapps curently .
> >
> > *Hardware*: Single Server , one JVM , Heap memory 19GB ,Total Ram :32GB ,
> > Permgen initally 1GB ,now increased to 2GB.
> >
> > *Solr Indexes* : Most are the order of a few MB ,about 2 big index of
> > about 3GB each
> >
> > *Scaling Step 1 *: We saw the permgen value go upto to nearly 850 mb
> ,when
> > we created so many webapps ,hence now we are moving to solr cores and we
> > are going to have about 50 cores per webapp ,bringing the number of
> webapps
> > to about 5 . We want to distribute the cores with multiple webapps to
> avoid
> > a single point of failure.
> >
> >
> > *Requirement* :
> >
> >
> > - We need to only scale the cores horizontally ,whose index sizes
> are
> > big.
> > - We also require permission based search for each webapp ,would
> solr
> > NRT fit our needs ,where we can index the permission into the document
> > ,which would mean there would be frequent addition and deletion of
> > permissions to the documents across cores.
> > - We also require automatic fail over
> >
> > What technology would be ideal fit given Solr Cloud ,Katta , Solandra
> > ,Lily,Elastic Search etc [Preferably Open source] [ We would be required
> to
> > maintain many webapps with multicores ] and what about the commercial
> > offering given out use case
> >
> > Thanks.
> >
> > Regards,
> > Sujatha
>
>
>
Re: Scaling Solr - Suggestions !!
Posted by Michael Della Bitta <mi...@appinions.com>.
Just my opinion, but I'm not sure I see the value in deploying the cores
to different webapps in a single container on a single machine to avoid
a single point of failure... You still have a single point of failure at
the process level down to the hardware, which when you think about it,
is mostly everything. But perhaps you're at least using more than one
container.
It sounds to me that the easiest route to scalability for you would be
to add more machines. Unless your cores are particularly complex or your
traffic is heavy, a 3GB core should be no match for a single machine.
And the traffic problem can be solved by replication and load balancing.
Michael
On Sat, 2012-04-28 at 13:24 +0530, Sujatha Arun wrote:
> Hello,
>
> *Background* :For each of our customers, we create 3 solr webapps with
> different search schema's,serving different search requirements and we
> have about 70 customers.So we have about 210 webapps curently .
>
> *Hardware*: Single Server , one JVM , Heap memory 19GB ,Total Ram :32GB ,
> Permgen initally 1GB ,now increased to 2GB.
>
> *Solr Indexes* : Most are the order of a few MB ,about 2 big index of
> about 3GB each
>
> *Scaling Step 1 *: We saw the permgen value go upto to nearly 850 mb ,when
> we created so many webapps ,hence now we are moving to solr cores and we
> are going to have about 50 cores per webapp ,bringing the number of webapps
> to about 5 . We want to distribute the cores with multiple webapps to avoid
> a single point of failure.
>
>
> *Requirement* :
>
>
> - We need to only scale the cores horizontally ,whose index sizes are
> big.
> - We also require permission based search for each webapp ,would solr
> NRT fit our needs ,where we can index the permission into the document
> ,which would mean there would be frequent addition and deletion of
> permissions to the documents across cores.
> - We also require automatic fail over
>
> What technology would be ideal fit given Solr Cloud ,Katta , Solandra
> ,Lily,Elastic Search etc [Preferably Open source] [ We would be required to
> maintain many webapps with multicores ] and what about the commercial
> offering given out use case
>
> Thanks.
>
> Regards,
> Sujatha