You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Artem Russakovskii (JIRA)" <ji...@apache.org> on 2009/10/02 03:50:23 UTC
[jira] Commented: (SOLR-1482) Solr master and slave freeze after
query
[ https://issues.apache.org/jira/browse/SOLR-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761449#action_12761449 ]
Artem Russakovskii commented on SOLR-1482:
------------------------------------------
I'm getting an error even just trying to access a single shard's admin interface, even after adjusting -XX:MaxPermSize=512m
{quote}
==> catalina.out <==
Oct 1, 2009 6:47:06 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError: PermGen space
at java.lang.Throwable.getStackTraceElement(Native Method)
at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
at java.lang.Throwable.printStackTrace(Throwable.java:510)
at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:72)
at org.apache.juli.FileHandler.publish(FileHandler.java:129)
at java.util.logging.Logger.log(Logger.java:458)
at java.util.logging.Logger.doLog(Logger.java:480)
at java.util.logging.Logger.logp(Logger.java:680)
at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167)
at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:324)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)
{quote}
:-/
> Solr master and slave freeze after query
> ----------------------------------------
>
> Key: SOLR-1482
> URL: https://issues.apache.org/jira/browse/SOLR-1482
> Project: Solr
> Issue Type: Bug
> Affects Versions: 1.4
> Environment: Nightly 9/28/09.
> 14 individual instances per server, using JNDI.
> replicateAfter commit, 5 min interval polling.
> All caches are currently commented out, on both slave and master.
> Lots of ongoing commits - large chunks of data, each accompanied by a commit. This is to guarantee that anything we think is now in Solr remains there in case the server crashes.
> Reporter: Artem Russakovskii
> Priority: Critical
>
> We're having issues with the deployment of 2 master-slave setups.
> One of the master-slave setups is OK (so far) but on the other both the master and the slave keep freezing, but only after I send a query to them. And by freezing I mean indefinite hanging, with almost no output to log, no errors, nothing. It's as if there's some sort of a deadlock. The hanging servers need to be killed with -9, otherwise they keep hanging.
> The query I send queries all instances at the same time using the ?shards= syntax.
> On the slave, the logs just stop - nothing shows up anymore after the query is issued. On the master, they're a bit more descriptive. This information seeps through very-very slowly, as you can see from the timestamps:
> {quote}
> SEVERE: java.lang.OutOfMemoryError: PermGen space
> Oct 1, 2009 2:16:00 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: PermGen space
> Oct 1, 2009 2:19:37 PM org.apache.catalina.connector.CoyoteAdapter service
> SEVERE: An exception or error occurred in the container during the request processing
> java.lang.OutOfMemoryError: PermGen space
> Oct 1, 2009 2:19:37 PM org.apache.coyote.http11.Http11Processor process
> SEVERE: Error processing request
> java.lang.OutOfMemoryError: PermGen space
> Oct 1, 2009 2:19:39 PM org.apache.catalina.connector.CoyoteAdapter service
> SEVERE: An exception or error occurred in the container during the request processing
> java.lang.OutOfMemoryError: PermGen space
> Exception in thread "ContainerBackException in thread "pool-29-threadOct 1, 2009 2:21:47 PM org.apache.catalina.connector.CoyoteAdapter service
> SEVERE: An exception or error occurred in the container during the request processing
> java.lang.OutOfMemoryError: PermGen space
> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
> SEVERE: Error reading request, ignored
> java.lang.OutOfMemoryError: PermGen space
> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
> SEVERE: Error reading request, ignored
> java.lang.OutOfMemoryError: PermGen space
> -22" java.lang.OutOfMemoryError: PermGen space
> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
> SEVERE: Error reading request, ignored
> java.lang.OutOfMemoryError: PermGen space
> Exception in thread "http-8080-42" Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
> SEVERE: Error reading request, ignored
> java.lang.OutOfMemoryError: PermGen space
> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
> SEVERE: Error reading request, ignored
> java.lang.OutOfMemoryError: PermGen space
> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
> SEVERE: Error reading request, ignored
> java.lang.OutOfMemoryError: PermGen space
> Exception in thread "http-8080-26" Exception in thread "http-8080-32" Exception in thread "http-8080-25" Exception in thread "http-8080-22" Exception in thread "http-8080-15" Exception in thread "http-8080-45" Exception in thread "http-8080-13" Exception in thread "http-8080-48" Exception in thread "http-8080-7" Exception in thread "http-8080-38" Exception in thread "http-8080-39" Exception in thread "http-8080-28" Exception in thread "http-8080-1" Exception in thread "http-8080-2" Exception in thread "http-8080-12" Exception in thread "http-8080-44" Exception in thread "http-8080-47" Exception in thread "http-8080-29" Exception in thread "http-8080-33" Exception in thread "http-8080-27" Exception in thread "http-8080-36" Exception in thread "http-8080-113" Exception in thread "http-8080-112" Exception in thread "http-8080-37" Exception in thread "http-8080-18" java.lang.OutOfMemoryError: PermGen space
> java.lang.OutOfMemoryError: PermGen space
> java.lang.OutOfMemoryError: PermGen space
> java.lang.OutOfMemoryError: PermGen space
> java.lang.OutOfMemoryError: PermGen space
> Exception in thread "http-8080-34" java.lang.OutOfMemoryError: PermGen space
> java.lang.OutOfMemoryError: PermGen space
> Exception in thread "http-8080-103"
> {quote}
> So the problem seems to be related to PermGen space. I found http://www.nabble.com/Number-of-webapps-td22198080.html and tried -XX:MaxPermSize=256m, but it didn't fix the problem. The current CATALINA_OPTS looks like this:
> {quote}
> export CATALINA_OPTS="-XX:MaxPermSize=256m -Xmx6500m -XX:+UseConcMarkSweepGC"
> {quote}
> Is the only solution at this point going multicore, as Noble suggested (is Noble your first name? I always assumed it was Paul and Noble was part of the nickname)? Will multicore get rid of the problem, before we spend time looking at it? For multicore, will the existing data dirs be compatible or would a complete reindex be needed?
> I'm willing to provide any information to you guys, just not sure what at the moment. I'm also open to communicating outside of JIRA, at artem [_aT_] plaxo {dot} com.
> Thanks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (SOLR-1482) Solr master and slave freeze after
query
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
What is your index size? if you have enough RAM , try out a bigger
perm gen size. If you shard, you will not be able to use the same
data dir. The data has to be split among shards. Which mean
reindexing.
--Noble
On Fri, Oct 2, 2009 at 7:20 AM, Artem Russakovskii (JIRA)
<ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/SOLR-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12761449#action_12761449 ]
>
> Artem Russakovskii commented on SOLR-1482:
> ------------------------------------------
>
> I'm getting an error even just trying to access a single shard's admin interface, even after adjusting -XX:MaxPermSize=512m
>
> {quote}
> ==> catalina.out <==
> Oct 1, 2009 6:47:06 PM org.apache.coyote.http11.Http11Processor process
> SEVERE: Error processing request
> java.lang.OutOfMemoryError: PermGen space
> at java.lang.Throwable.getStackTraceElement(Native Method)
> at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
> at java.lang.Throwable.printStackTrace(Throwable.java:510)
> at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:72)
> at org.apache.juli.FileHandler.publish(FileHandler.java:129)
> at java.util.logging.Logger.log(Logger.java:458)
> at java.util.logging.Logger.doLog(Logger.java:480)
> at java.util.logging.Logger.logp(Logger.java:680)
> at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:167)
> at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:135)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:324)
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
> at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
> at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
> at java.lang.Thread.run(Thread.java:619)
> {quote}
>
> :-/
>
>> Solr master and slave freeze after query
>> ----------------------------------------
>>
>> Key: SOLR-1482
>> URL: https://issues.apache.org/jira/browse/SOLR-1482
>> Project: Solr
>> Issue Type: Bug
>> Affects Versions: 1.4
>> Environment: Nightly 9/28/09.
>> 14 individual instances per server, using JNDI.
>> replicateAfter commit, 5 min interval polling.
>> All caches are currently commented out, on both slave and master.
>> Lots of ongoing commits - large chunks of data, each accompanied by a commit. This is to guarantee that anything we think is now in Solr remains there in case the server crashes.
>> Reporter: Artem Russakovskii
>> Priority: Critical
>>
>> We're having issues with the deployment of 2 master-slave setups.
>> One of the master-slave setups is OK (so far) but on the other both the master and the slave keep freezing, but only after I send a query to them. And by freezing I mean indefinite hanging, with almost no output to log, no errors, nothing. It's as if there's some sort of a deadlock. The hanging servers need to be killed with -9, otherwise they keep hanging.
>> The query I send queries all instances at the same time using the ?shards= syntax.
>> On the slave, the logs just stop - nothing shows up anymore after the query is issued. On the master, they're a bit more descriptive. This information seeps through very-very slowly, as you can see from the timestamps:
>> {quote}
>> SEVERE: java.lang.OutOfMemoryError: PermGen space
>> Oct 1, 2009 2:16:00 PM org.apache.solr.common.SolrException log
>> SEVERE: java.lang.OutOfMemoryError: PermGen space
>> Oct 1, 2009 2:19:37 PM org.apache.catalina.connector.CoyoteAdapter service
>> SEVERE: An exception or error occurred in the container during the request processing
>> java.lang.OutOfMemoryError: PermGen space
>> Oct 1, 2009 2:19:37 PM org.apache.coyote.http11.Http11Processor process
>> SEVERE: Error processing request
>> java.lang.OutOfMemoryError: PermGen space
>> Oct 1, 2009 2:19:39 PM org.apache.catalina.connector.CoyoteAdapter service
>> SEVERE: An exception or error occurred in the container during the request processing
>> java.lang.OutOfMemoryError: PermGen space
>> Exception in thread "ContainerBackException in thread "pool-29-threadOct 1, 2009 2:21:47 PM org.apache.catalina.connector.CoyoteAdapter service
>> SEVERE: An exception or error occurred in the container during the request processing
>> java.lang.OutOfMemoryError: PermGen space
>> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
>> SEVERE: Error reading request, ignored
>> java.lang.OutOfMemoryError: PermGen space
>> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
>> SEVERE: Error reading request, ignored
>> java.lang.OutOfMemoryError: PermGen space
>> -22" java.lang.OutOfMemoryError: PermGen space
>> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
>> SEVERE: Error reading request, ignored
>> java.lang.OutOfMemoryError: PermGen space
>> Exception in thread "http-8080-42" Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
>> SEVERE: Error reading request, ignored
>> java.lang.OutOfMemoryError: PermGen space
>> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
>> SEVERE: Error reading request, ignored
>> java.lang.OutOfMemoryError: PermGen space
>> Oct 1, 2009 2:21:47 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler process
>> SEVERE: Error reading request, ignored
>> java.lang.OutOfMemoryError: PermGen space
>> Exception in thread "http-8080-26" Exception in thread "http-8080-32" Exception in thread "http-8080-25" Exception in thread "http-8080-22" Exception in thread "http-8080-15" Exception in thread "http-8080-45" Exception in thread "http-8080-13" Exception in thread "http-8080-48" Exception in thread "http-8080-7" Exception in thread "http-8080-38" Exception in thread "http-8080-39" Exception in thread "http-8080-28" Exception in thread "http-8080-1" Exception in thread "http-8080-2" Exception in thread "http-8080-12" Exception in thread "http-8080-44" Exception in thread "http-8080-47" Exception in thread "http-8080-29" Exception in thread "http-8080-33" Exception in thread "http-8080-27" Exception in thread "http-8080-36" Exception in thread "http-8080-113" Exception in thread "http-8080-112" Exception in thread "http-8080-37" Exception in thread "http-8080-18" java.lang.OutOfMemoryError: PermGen space
>> java.lang.OutOfMemoryError: PermGen space
>> java.lang.OutOfMemoryError: PermGen space
>> java.lang.OutOfMemoryError: PermGen space
>> java.lang.OutOfMemoryError: PermGen space
>> Exception in thread "http-8080-34" java.lang.OutOfMemoryError: PermGen space
>> java.lang.OutOfMemoryError: PermGen space
>> Exception in thread "http-8080-103"
>> {quote}
>> So the problem seems to be related to PermGen space. I found http://www.nabble.com/Number-of-webapps-td22198080.html and tried -XX:MaxPermSize=256m, but it didn't fix the problem. The current CATALINA_OPTS looks like this:
>> {quote}
>> export CATALINA_OPTS="-XX:MaxPermSize=256m -Xmx6500m -XX:+UseConcMarkSweepGC"
>> {quote}
>> Is the only solution at this point going multicore, as Noble suggested (is Noble your first name? I always assumed it was Paul and Noble was part of the nickname)? Will multicore get rid of the problem, before we spend time looking at it? For multicore, will the existing data dirs be compatible or would a complete reindex be needed?
>> I'm willing to provide any information to you guys, just not sure what at the moment. I'm also open to communicating outside of JIRA, at artem [_aT_] plaxo {dot} com.
>> Thanks.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com