You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by revas <re...@gmail.com> on 2009/02/25 09:08:40 UTC
Number of webapps
Hi
I am sure this question has been repeated many times over and there has been
several generic answers ,but i am looking for specific answers.
I have a single server whose configuration i give below,this being the only
server we have at present ,the requirement is everytime we create a new
website ,we create two instances for the same one for content search and one
for product search ,both have faceting requirements.
there are about 25 fields for product schema and abt 20 for content schema
,we do not store the content in the server ,the content is only indexed.
Assuming that we currently have 10 websites ,which means we have 20 webapps
running on this server each having about 1000 documents and size of the
index is approximately 50mb currently .The index size of each is expected to
grow continlously as more products are added.
We recenlty got the followng error on creation of a new webapp ?
SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
terminating thread
Feb 24, 2009 6:22:16 AM
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
terminating thread
Sent at 12:32 PM on Wednesday
What would this mean?Given the above,How many such webapps can we have
on this server?
*Server config*
OS: Red Hat Enterprise Linux ES 4 - 64 Bit
# Processor: Dual AMD Opteron Dual Core 270 2.0 GHz
# 4GB DDR RAM
# Hard Drive: 73GB SCSI
# Hard Drive: 73GB SCSI
thanks
Re: Number of webapps
Posted by revas <re...@gmail.com>.
HI,
How do i get the info on the current setting of MaxPermSize?
Regards
Sujahta
On 2/27/09, Alexander Ramos Jardim <al...@gmail.com> wrote:
>
> Another simple solution for your requirement is to use multicore. This way
> you will have only one Solr webapp loaded with as many indexes as you need.
>
> See more at http://wiki.apache.org/solr/MultiCore
>
> 2009/2/25 Michael Della Bitta <md...@gmail.com>
>
> > Unfortunately, I think the way this works is the container creates a
> > Classloader for each context and loads the contents of the .war into
> > that, regardless of whether each context references the same .war
> > file. All those classes are stored in permanent generation space, and
> > I'm fairly sure if you restart a context individually with the manager
> > application, a new ClassLoader for the context is created and the
> > permanent generation space the old one was consuming is simply leaked.
> >
> > Something that is crazy enough to work might be to unpack the Solr
> > .war and move all the .jar files and class files that don't contain
> > servlet API classes to .jars in $TOMCAT_HOME/lib, and then repack the
> > .war without these files. These would then be loaded by the common
> > classloader once per container, instead of once per context. You can
> > read more about this classloader business here:
> > http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html (might
> > need a different URL depending on the version of Tomcat you're
> > running).
> >
> > Michael
> >
> > On Wed, Feb 25, 2009 at 11:42 AM, revas <re...@gmail.com> wrote:
> > > thanks will try that .I also have the war file for each solr instance
> in
> > the
> > > home directory of the instance ,would that be the problem ?
> > >
> > > if i were to have common war file for n instances ,would there be any
> > issue?
> > >
> > > regards
> > > revas
> > >
> > > On 2/25/09, Michael Della Bitta <md...@gmail.com> wrote:
> > >>
> > >> It's possible I don't know enough about Solr's internals and there's a
> > >> better solution than this, and it's surprising me that you're running
> > >> out of PermGen space before you're running out of heap, but maybe
> > >> you've already increased the general heap size without tweaking
> > >> PermGen, and loading all the classes involved in loading 20 contexts
> > >> is putting you over. In any case, you might try adding the following
> > >> option to CATALINA_OPTS: -XX:MaxPermSize=256m. If you don't know where
> > >> to put something like that, you might try adding the following line to
> > >> $TOMCAT_HOME/bin/startup.sh:
> > >>
> > >> export CATALINA_OPTS="-XX:MaxPermSize=256m ${CATALINA_OPTS}"
> > >>
> > >> If that value (256) doesn't alleviate the problem, you might try
> > increasing
> > >> it.
> > >>
> > >> Hope that helps,
> > >>
> > >> Michael Della Bitta
> > >>
> > >>
> > >> On Wed, Feb 25, 2009 at 3:08 AM, revas <re...@gmail.com> wrote:
> > >> > Hi
> > >> >
> > >> > I am sure this question has been repeated many times over and there
> > has
> > >> been
> > >> > several generic answers ,but i am looking for specific answers.
> > >> >
> > >> > I have a single server whose configuration i give below,this being
> the
> > >> only
> > >> > server we have at present ,the requirement is everytime we create a
> > new
> > >> > website ,we create two instances for the same one for content search
> > and
> > >> one
> > >> > for product search ,both have faceting requirements.
> > >> >
> > >> > there are about 25 fields for product schema and abt 20 for content
> > >> schema
> > >> > ,we do not store the content in the server ,the content is only
> > indexed.
> > >> >
> > >> > Assuming that we currently have 10 websites ,which means we have 20
> > >> webapps
> > >> > running on this server each having about 1000 documents and size of
> > the
> > >> > index is approximately 50mb currently .The index size of each is
> > expected
> > >> to
> > >> > grow continlously as more products are added.
> > >> >
> > >> >
> > >> > We recenlty got the followng error on creation of a new webapp ?
> > >> > SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen
> > space)
> > >> > executing
> > org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> > >> > terminating thread
> > >> > Feb 24, 2009 6:22:16 AM
> > >> > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
> > >> > SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
> > >> > executing
> > org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> > >> > terminating thread
> > >> > Sent at 12:32 PM on Wednesday
> > >> >
> > >> > What would this mean?Given the above,How many such webapps can we
> > have
> > >> > on this server?
> > >> >
> > >> > *Server config*
> > >> >
> > >> > OS: Red Hat Enterprise Linux ES 4 - 64 Bit
> > >> > # Processor: Dual AMD Opteron Dual Core 270 2.0 GHz
> > >> > # 4GB DDR RAM
> > >> > # Hard Drive: 73GB SCSI
> > >> > # Hard Drive: 73GB SCSI
> > >> >
> > >> > thanks
> > >> >
> > >>
> > >
> >
>
>
>
> --
> Alexander Ramos Jardim
>
Re: Number of webapps
Posted by Alexander Ramos Jardim <al...@gmail.com>.
Another simple solution for your requirement is to use multicore. This way
you will have only one Solr webapp loaded with as many indexes as you need.
See more at http://wiki.apache.org/solr/MultiCore
2009/2/25 Michael Della Bitta <md...@gmail.com>
> Unfortunately, I think the way this works is the container creates a
> Classloader for each context and loads the contents of the .war into
> that, regardless of whether each context references the same .war
> file. All those classes are stored in permanent generation space, and
> I'm fairly sure if you restart a context individually with the manager
> application, a new ClassLoader for the context is created and the
> permanent generation space the old one was consuming is simply leaked.
>
> Something that is crazy enough to work might be to unpack the Solr
> .war and move all the .jar files and class files that don't contain
> servlet API classes to .jars in $TOMCAT_HOME/lib, and then repack the
> .war without these files. These would then be loaded by the common
> classloader once per container, instead of once per context. You can
> read more about this classloader business here:
> http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html (might
> need a different URL depending on the version of Tomcat you're
> running).
>
> Michael
>
> On Wed, Feb 25, 2009 at 11:42 AM, revas <re...@gmail.com> wrote:
> > thanks will try that .I also have the war file for each solr instance in
> the
> > home directory of the instance ,would that be the problem ?
> >
> > if i were to have common war file for n instances ,would there be any
> issue?
> >
> > regards
> > revas
> >
> > On 2/25/09, Michael Della Bitta <md...@gmail.com> wrote:
> >>
> >> It's possible I don't know enough about Solr's internals and there's a
> >> better solution than this, and it's surprising me that you're running
> >> out of PermGen space before you're running out of heap, but maybe
> >> you've already increased the general heap size without tweaking
> >> PermGen, and loading all the classes involved in loading 20 contexts
> >> is putting you over. In any case, you might try adding the following
> >> option to CATALINA_OPTS: -XX:MaxPermSize=256m. If you don't know where
> >> to put something like that, you might try adding the following line to
> >> $TOMCAT_HOME/bin/startup.sh:
> >>
> >> export CATALINA_OPTS="-XX:MaxPermSize=256m ${CATALINA_OPTS}"
> >>
> >> If that value (256) doesn't alleviate the problem, you might try
> increasing
> >> it.
> >>
> >> Hope that helps,
> >>
> >> Michael Della Bitta
> >>
> >>
> >> On Wed, Feb 25, 2009 at 3:08 AM, revas <re...@gmail.com> wrote:
> >> > Hi
> >> >
> >> > I am sure this question has been repeated many times over and there
> has
> >> been
> >> > several generic answers ,but i am looking for specific answers.
> >> >
> >> > I have a single server whose configuration i give below,this being the
> >> only
> >> > server we have at present ,the requirement is everytime we create a
> new
> >> > website ,we create two instances for the same one for content search
> and
> >> one
> >> > for product search ,both have faceting requirements.
> >> >
> >> > there are about 25 fields for product schema and abt 20 for content
> >> schema
> >> > ,we do not store the content in the server ,the content is only
> indexed.
> >> >
> >> > Assuming that we currently have 10 websites ,which means we have 20
> >> webapps
> >> > running on this server each having about 1000 documents and size of
> the
> >> > index is approximately 50mb currently .The index size of each is
> expected
> >> to
> >> > grow continlously as more products are added.
> >> >
> >> >
> >> > We recenlty got the followng error on creation of a new webapp ?
> >> > SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen
> space)
> >> > executing
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> >> > terminating thread
> >> > Feb 24, 2009 6:22:16 AM
> >> > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
> >> > SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
> >> > executing
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> >> > terminating thread
> >> > Sent at 12:32 PM on Wednesday
> >> >
> >> > What would this mean?Given the above,How many such webapps can we
> have
> >> > on this server?
> >> >
> >> > *Server config*
> >> >
> >> > OS: Red Hat Enterprise Linux ES 4 - 64 Bit
> >> > # Processor: Dual AMD Opteron Dual Core 270 2.0 GHz
> >> > # 4GB DDR RAM
> >> > # Hard Drive: 73GB SCSI
> >> > # Hard Drive: 73GB SCSI
> >> >
> >> > thanks
> >> >
> >>
> >
>
--
Alexander Ramos Jardim
Re: Number of webapps
Posted by Michael Della Bitta <md...@gmail.com>.
Unfortunately, I think the way this works is the container creates a
Classloader for each context and loads the contents of the .war into
that, regardless of whether each context references the same .war
file. All those classes are stored in permanent generation space, and
I'm fairly sure if you restart a context individually with the manager
application, a new ClassLoader for the context is created and the
permanent generation space the old one was consuming is simply leaked.
Something that is crazy enough to work might be to unpack the Solr
.war and move all the .jar files and class files that don't contain
servlet API classes to .jars in $TOMCAT_HOME/lib, and then repack the
.war without these files. These would then be loaded by the common
classloader once per container, instead of once per context. You can
read more about this classloader business here:
http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html (might
need a different URL depending on the version of Tomcat you're
running).
Michael
On Wed, Feb 25, 2009 at 11:42 AM, revas <re...@gmail.com> wrote:
> thanks will try that .I also have the war file for each solr instance in the
> home directory of the instance ,would that be the problem ?
>
> if i were to have common war file for n instances ,would there be any issue?
>
> regards
> revas
>
> On 2/25/09, Michael Della Bitta <md...@gmail.com> wrote:
>>
>> It's possible I don't know enough about Solr's internals and there's a
>> better solution than this, and it's surprising me that you're running
>> out of PermGen space before you're running out of heap, but maybe
>> you've already increased the general heap size without tweaking
>> PermGen, and loading all the classes involved in loading 20 contexts
>> is putting you over. In any case, you might try adding the following
>> option to CATALINA_OPTS: -XX:MaxPermSize=256m. If you don't know where
>> to put something like that, you might try adding the following line to
>> $TOMCAT_HOME/bin/startup.sh:
>>
>> export CATALINA_OPTS="-XX:MaxPermSize=256m ${CATALINA_OPTS}"
>>
>> If that value (256) doesn't alleviate the problem, you might try increasing
>> it.
>>
>> Hope that helps,
>>
>> Michael Della Bitta
>>
>>
>> On Wed, Feb 25, 2009 at 3:08 AM, revas <re...@gmail.com> wrote:
>> > Hi
>> >
>> > I am sure this question has been repeated many times over and there has
>> been
>> > several generic answers ,but i am looking for specific answers.
>> >
>> > I have a single server whose configuration i give below,this being the
>> only
>> > server we have at present ,the requirement is everytime we create a new
>> > website ,we create two instances for the same one for content search and
>> one
>> > for product search ,both have faceting requirements.
>> >
>> > there are about 25 fields for product schema and abt 20 for content
>> schema
>> > ,we do not store the content in the server ,the content is only indexed.
>> >
>> > Assuming that we currently have 10 websites ,which means we have 20
>> webapps
>> > running on this server each having about 1000 documents and size of the
>> > index is approximately 50mb currently .The index size of each is expected
>> to
>> > grow continlously as more products are added.
>> >
>> >
>> > We recenlty got the followng error on creation of a new webapp ?
>> > SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
>> > executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
>> > terminating thread
>> > Feb 24, 2009 6:22:16 AM
>> > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>> > SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
>> > executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
>> > terminating thread
>> > Sent at 12:32 PM on Wednesday
>> >
>> > What would this mean?Given the above,How many such webapps can we have
>> > on this server?
>> >
>> > *Server config*
>> >
>> > OS: Red Hat Enterprise Linux ES 4 - 64 Bit
>> > # Processor: Dual AMD Opteron Dual Core 270 2.0 GHz
>> > # 4GB DDR RAM
>> > # Hard Drive: 73GB SCSI
>> > # Hard Drive: 73GB SCSI
>> >
>> > thanks
>> >
>>
>
Re: Number of webapps
Posted by revas <re...@gmail.com>.
thanks will try that .I also have the war file for each solr instance in the
home directory of the instance ,would that be the problem ?
if i were to have common war file for n instances ,would there be any issue?
regards
revas
On 2/25/09, Michael Della Bitta <md...@gmail.com> wrote:
>
> It's possible I don't know enough about Solr's internals and there's a
> better solution than this, and it's surprising me that you're running
> out of PermGen space before you're running out of heap, but maybe
> you've already increased the general heap size without tweaking
> PermGen, and loading all the classes involved in loading 20 contexts
> is putting you over. In any case, you might try adding the following
> option to CATALINA_OPTS: -XX:MaxPermSize=256m. If you don't know where
> to put something like that, you might try adding the following line to
> $TOMCAT_HOME/bin/startup.sh:
>
> export CATALINA_OPTS="-XX:MaxPermSize=256m ${CATALINA_OPTS}"
>
> If that value (256) doesn't alleviate the problem, you might try increasing
> it.
>
> Hope that helps,
>
> Michael Della Bitta
>
>
> On Wed, Feb 25, 2009 at 3:08 AM, revas <re...@gmail.com> wrote:
> > Hi
> >
> > I am sure this question has been repeated many times over and there has
> been
> > several generic answers ,but i am looking for specific answers.
> >
> > I have a single server whose configuration i give below,this being the
> only
> > server we have at present ,the requirement is everytime we create a new
> > website ,we create two instances for the same one for content search and
> one
> > for product search ,both have faceting requirements.
> >
> > there are about 25 fields for product schema and abt 20 for content
> schema
> > ,we do not store the content in the server ,the content is only indexed.
> >
> > Assuming that we currently have 10 websites ,which means we have 20
> webapps
> > running on this server each having about 1000 documents and size of the
> > index is approximately 50mb currently .The index size of each is expected
> to
> > grow continlously as more products are added.
> >
> >
> > We recenlty got the followng error on creation of a new webapp ?
> > SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
> > executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> > terminating thread
> > Feb 24, 2009 6:22:16 AM
> > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
> > SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
> > executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> > terminating thread
> > Sent at 12:32 PM on Wednesday
> >
> > What would this mean?Given the above,How many such webapps can we have
> > on this server?
> >
> > *Server config*
> >
> > OS: Red Hat Enterprise Linux ES 4 - 64 Bit
> > # Processor: Dual AMD Opteron Dual Core 270 2.0 GHz
> > # 4GB DDR RAM
> > # Hard Drive: 73GB SCSI
> > # Hard Drive: 73GB SCSI
> >
> > thanks
> >
>
Re: Number of webapps
Posted by Michael Della Bitta <md...@gmail.com>.
It's possible I don't know enough about Solr's internals and there's a
better solution than this, and it's surprising me that you're running
out of PermGen space before you're running out of heap, but maybe
you've already increased the general heap size without tweaking
PermGen, and loading all the classes involved in loading 20 contexts
is putting you over. In any case, you might try adding the following
option to CATALINA_OPTS: -XX:MaxPermSize=256m. If you don't know where
to put something like that, you might try adding the following line to
$TOMCAT_HOME/bin/startup.sh:
export CATALINA_OPTS="-XX:MaxPermSize=256m ${CATALINA_OPTS}"
If that value (256) doesn't alleviate the problem, you might try increasing it.
Hope that helps,
Michael Della Bitta
On Wed, Feb 25, 2009 at 3:08 AM, revas <re...@gmail.com> wrote:
> Hi
>
> I am sure this question has been repeated many times over and there has been
> several generic answers ,but i am looking for specific answers.
>
> I have a single server whose configuration i give below,this being the only
> server we have at present ,the requirement is everytime we create a new
> website ,we create two instances for the same one for content search and one
> for product search ,both have faceting requirements.
>
> there are about 25 fields for product schema and abt 20 for content schema
> ,we do not store the content in the server ,the content is only indexed.
>
> Assuming that we currently have 10 websites ,which means we have 20 webapps
> running on this server each having about 1000 documents and size of the
> index is approximately 50mb currently .The index size of each is expected to
> grow continlously as more products are added.
>
>
> We recenlty got the followng error on creation of a new webapp ?
> SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
> executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> terminating thread
> Feb 24, 2009 6:22:16 AM
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
> SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
> executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> terminating thread
> Sent at 12:32 PM on Wednesday
>
> What would this mean?Given the above,How many such webapps can we have
> on this server?
>
> *Server config*
>
> OS: Red Hat Enterprise Linux ES 4 - 64 Bit
> # Processor: Dual AMD Opteron Dual Core 270 2.0 GHz
> # 4GB DDR RAM
> # Hard Drive: 73GB SCSI
> # Hard Drive: 73GB SCSI
>
> thanks
>
Re: Number of webapps
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
running multiple webapps look like a bad idea. This is the very reason
solr has the multicore feature.
permgen size is a jvm option I guess it would be something like
-XX:MaxPermSize
On Wed, Feb 25, 2009 at 1:38 PM, revas <re...@gmail.com> wrote:
> Hi
>
> I am sure this question has been repeated many times over and there has been
> several generic answers ,but i am looking for specific answers.
>
> I have a single server whose configuration i give below,this being the only
> server we have at present ,the requirement is everytime we create a new
> website ,we create two instances for the same one for content search and one
> for product search ,both have faceting requirements.
>
> there are about 25 fields for product schema and abt 20 for content schema
> ,we do not store the content in the server ,the content is only indexed.
>
> Assuming that we currently have 10 websites ,which means we have 20 webapps
> running on this server each having about 1000 documents and size of the
> index is approximately 50mb currently .The index size of each is expected to
> grow continlously as more products are added.
>
>
> We recenlty got the followng error on creation of a new webapp ?
> SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
> executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> terminating thread
> Feb 24, 2009 6:22:16 AM
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
> SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space)
> executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@1c2534f,
> terminating thread
> Sent at 12:32 PM on Wednesday
>
> What would this mean?Given the above,How many such webapps can we have
> on this server?
>
> *Server config*
>
> OS: Red Hat Enterprise Linux ES 4 - 64 Bit
> # Processor: Dual AMD Opteron Dual Core 270 2.0 GHz
> # 4GB DDR RAM
> # Hard Drive: 73GB SCSI
> # Hard Drive: 73GB SCSI
>
> thanks
>
--
--Noble Paul