You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Atanas Atanasov <at...@gmail.com> on 2014/04/10 08:40:46 UTC
Tomcat creates a thread for each SOLR core
Hi, guys,
I need some help. After updating to SOLR 4.4 the tomcat process is
consuming about 2GBs of memory, the CPU usage is about 40% after the start
for about 10 minutes. However, the bigger problem is, I have about 1000
cores and seems that for each core a thread is created. The process has
more than 1000 threads and everything is extremely slow. Creating or
unloading a core even without documents takes about 20 minutes. Searching
is more or less good, but storing also takes a lot.
Is there some configuration I missed or that I did wrong? There aren't many
calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine has
24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2.
Index is uppdated every 30 seconds/10 000 documents.
I haven't checked the number of threads before the update, because I didn't
have to, it was working just fine. Any suggestion will be highly
appreciated, thank you in advance.
Regards,
Atanas
Re: Tomcat creates a thread for each SOLR core
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/22/2014 1:28 AM, Atanas Atanasov wrote:
> Just a clarification if someone decides to reproduce it:
> 1. 64 bit apache tomcat installation (7 or 8) -> solr 4.4 deployed, any JVM.
> 2. Create 1000 cores.
> 3. Restart Apache Tomcat.
> 4. Create 1 core.
>
> I hope I explained it well, if not just ask.
Here's a screenshot of the thread count graph on jconsole for one of my
active Solr processes. This is Solr 4.6.1, running on an upgraded
version (8.1.14) of the stripped down Jetty included in the Solr example.
https://dl.dropboxusercontent.com/u/97770508/threads-idxb2.png
I've got nineteen cores on each of my systems. About four or five of
those cores are active at any one time on each machine. My query rate
is extremely low. The circled numbers are the 15 minute rate in queries
per second for the machine where I got the jconsole graphs. Each line
is one core.
https://dl.dropboxusercontent.com/u/97770508/threads-status.png
So with 16 total cores (four in active use) and less than 8 queries per
second, I've got about 150 threads. The default maxThreads value on
Tomcat is 200. The maxThreads on the Jetty included in the Solr example
has been set to 10000, but I believe it also defaults to 200. For a
larger install, I do not think that 200 threads would be enough.
Looking at a thread dump in the admin UI, it looks like almost all of my
threads are either sitting in "wait" or "park". Eight threads had a
green checkmark, which I believe means they were running. The rest had
an hourglass with a yellow triangle.
If you repeat your test with the Jetty included in Solr, what happens?
Thanks,
Shawn
Re: Tomcat creates a thread for each SOLR core
Posted by Atanas Atanasov <at...@gmail.com>.
Hi to all,
I have some other information that might be useful. Installing previous
java did not work. The creation of a core does not take 20 minutes,
it seems that it happens right away but there is no response and looks
stuck. After I restart the apache tomcat
the core is there and working without any problems. In the SOLR admin panel
this is what thread dump shows:
http-nio-8080-exec-2 (27)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53dbcec
- sun.misc.Unsafe.park(Native Method)
- java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
-
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
- org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
- org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
-
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
-
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
- java.lang.Thread.run(Thread.java:744)
21091.3352ms
20420.5309ms
and
- sun.management.ThreadImpl.getThreadInfo1(Native Method)
- sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:172)
-
org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody(ThreadDumpHandler.java:69)
-
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
- org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
-
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
-
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
-
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
-
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
-
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
-
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
-
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
-
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:136)
-
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:74)
-
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
-
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
-
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:526)
-
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1017)
-
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:652)
-
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222)
-
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1575)
-
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1533)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
-
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
- java.lang.Thread.run(Thread.java:744)
they both seem to be stuck.
Just a clarification if someone decides to reproduce it:
1. 64 bit apache tomcat installation (7 or 8) -> solr 4.4 deployed, any JVM.
2. Create 1000 cores.
3. Restart Apache Tomcat.
4. Create 1 core.
I hope I explained it well, if not just ask.
Thanks in advance,
Atanas
On Tue, Apr 15, 2014 at 9:58 AM, Atanas Atanasov <at...@gmail.com>wrote:
> Hello again,
>
> Current situation is, after setting the two options in order not to load
> the cores on start up
> and ramBufferSizeMB=32 Tomcat is stable, responsive, threads reach 60 as a
> maximum.
> Browsing and storing are fast. I should note that I have many cores with
> small amount of documents.
> Unfortunately the problem with the creation of a new core taking 20
> minutes still exists.
> Next step will be downgrading to Java 7u25. Any other suggestions will be
> highly appreciated. Thanks in advance.
>
> P.S previous SOLR version from which I updated was 3.6.
>
> Regards,
> Atanas Atanasov
>
Re: Tomcat creates a thread for each SOLR core
Posted by Atanas Atanasov <at...@gmail.com>.
Hello again,
Current situation is, after setting the two options in order not to load
the cores on start up
and ramBufferSizeMB=32 Tomcat is stable, responsive, threads reach 60 as a
maximum.
Browsing and storing are fast. I should note that I have many cores with
small amount of documents.
Unfortunately the problem with the creation of a new core taking 20 minutes
still exists.
Next step will be downgrading to Java 7u25. Any other suggestions will be
highly appreciated. Thanks in advance.
P.S previous SOLR version from which I updated was 3.6.
Regards,
Atanas Atanasov
On Thu, Apr 10, 2014 at 6:06 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/10/2014 12:40 AM, Atanas Atanasov wrote:
> > I need some help. After updating to SOLR 4.4 the tomcat process is
> > consuming about 2GBs of memory, the CPU usage is about 40% after the
> start
> > for about 10 minutes. However, the bigger problem is, I have about 1000
> > cores and seems that for each core a thread is created. The process has
> > more than 1000 threads and everything is extremely slow. Creating or
> > unloading a core even without documents takes about 20 minutes. Searching
> > is more or less good, but storing also takes a lot.
> > Is there some configuration I missed or that I did wrong? There aren't
> many
> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine
> has
> > 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2.
> > Index is uppdated every 30 seconds/10 000 documents.
> > I haven't checked the number of threads before the update, because I
> didn't
> > have to, it was working just fine. Any suggestion will be highly
> > appreciated, thank you in advance.
>
> If creating a core takes 20 minutes, that sounds to me like the JVM is
> doing constant full garbage collections to free up enough memory for
> basic system operation. It could also be explained by temporary work
> threads having to wait to execute because the servlet container will not
> allow them to run.
>
> When indexing is happening, each core will set aside some memory for
> buffering index updates. By default, the value of ramBufferSizeMB is
> 100. If all your cores are indexing at once, multiply the indexing
> buffer by 1000, and you'll require 100GB of heap memory. You'll need to
> greatly reduce that buffer size. This buffer was 32MB by default in 4.0
> and earlier. If you are not setting this value, this change sounds like
> it might fully explain what you are seeing.
>
> https://issues.apache.org/jira/browse/SOLR-4074
>
> What version did you upgrade from? Solr 4.x is a very different beast
> than earlier major versions. I believe there may have been some changes
> made to reduce memory usage in versions after 4.4.0.
>
> The jetty that comes with Solr is configured to allow 10,000 threads.
> Most people don't have that many, even on a temporary basis, but bad
> things happen when the servlet container will not allow Solr to start as
> many as it requires. I believe that the typical default maxThreads
> value you'll find in a servlet container config is 200.
>
> Erick's right about a 6GB heap being very small for what you are trying
> to do. Putting 1000 cores on one machine is something I would never
> try. If it became a requirement I had to deal with, I wouldn't try it
> unless the machine had a lot more CPU cores, hundreds of gigabytes of
> RAM, and a lot of extremely fast disk space.
>
> If this worked before a Solr upgrade, I'm amazed. Congratulations to
> you for fine work!
>
> NB: Oracle Java 7u25 is what you should be using. 7u40 through 7u51
> have known bugs that affect Solr/Lucene. These should be fixed by 7u60.
> A pre-release of that is available now, and it should be generally
> available in May 2014.
>
> Thanks,
> Shawn
>
>
Re: Tomcat creates a thread for each SOLR core
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/10/2014 12:40 AM, Atanas Atanasov wrote:
> I need some help. After updating to SOLR 4.4 the tomcat process is
> consuming about 2GBs of memory, the CPU usage is about 40% after the start
> for about 10 minutes. However, the bigger problem is, I have about 1000
> cores and seems that for each core a thread is created. The process has
> more than 1000 threads and everything is extremely slow. Creating or
> unloading a core even without documents takes about 20 minutes. Searching
> is more or less good, but storing also takes a lot.
> Is there some configuration I missed or that I did wrong? There aren't many
> calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine has
> 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2.
> Index is uppdated every 30 seconds/10 000 documents.
> I haven't checked the number of threads before the update, because I didn't
> have to, it was working just fine. Any suggestion will be highly
> appreciated, thank you in advance.
If creating a core takes 20 minutes, that sounds to me like the JVM is
doing constant full garbage collections to free up enough memory for
basic system operation. It could also be explained by temporary work
threads having to wait to execute because the servlet container will not
allow them to run.
When indexing is happening, each core will set aside some memory for
buffering index updates. By default, the value of ramBufferSizeMB is
100. If all your cores are indexing at once, multiply the indexing
buffer by 1000, and you'll require 100GB of heap memory. You'll need to
greatly reduce that buffer size. This buffer was 32MB by default in 4.0
and earlier. If you are not setting this value, this change sounds like
it might fully explain what you are seeing.
https://issues.apache.org/jira/browse/SOLR-4074
What version did you upgrade from? Solr 4.x is a very different beast
than earlier major versions. I believe there may have been some changes
made to reduce memory usage in versions after 4.4.0.
The jetty that comes with Solr is configured to allow 10,000 threads.
Most people don't have that many, even on a temporary basis, but bad
things happen when the servlet container will not allow Solr to start as
many as it requires. I believe that the typical default maxThreads
value you'll find in a servlet container config is 200.
Erick's right about a 6GB heap being very small for what you are trying
to do. Putting 1000 cores on one machine is something I would never
try. If it became a requirement I had to deal with, I wouldn't try it
unless the machine had a lot more CPU cores, hundreds of gigabytes of
RAM, and a lot of extremely fast disk space.
If this worked before a Solr upgrade, I'm amazed. Congratulations to
you for fine work!
NB: Oracle Java 7u25 is what you should be using. 7u40 through 7u51
have known bugs that affect Solr/Lucene. These should be fixed by 7u60.
A pre-release of that is available now, and it should be generally
available in May 2014.
Thanks,
Shawn
Re: Tomcat creates a thread for each SOLR core
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
On Thu, Apr 10, 2014 at 2:14 PM, Atanas Atanasov <at...@gmail.com> wrote:
> SEVERE: null:ClientAbortException: java.net.SocketException: Software
> caused connection abort: socket write error
Separate issue, but most likely the client closed the browser and the
server had nowhere to send the respond too. So, it complaint. Happens
if your serving process is too slow.
The other one might be the same or might be different. The server
sends headers and expects the body to follow. Then, during processing
of the body, an error occurs. The server changes its mind and wants to
send an error (e.g. HTTP 500 instead of HTTP 200), but it's too late.
The headers were already sent out. So, it complains to the log file
instead. The real question is not this exception, but the internal
error that caused the server to change its mind.
I would concentrate on speed first and see if these problems go away.
Regards,
Alex.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency
Re: Tomcat creates a thread for each SOLR core
Posted by Aman Tandon <am...@gmail.com>.
OKay okay i am a centos user in office, windows at home :D
Thanks
Aman Tandon
On Thu, Apr 10, 2014 at 8:01 PM, Atanas Atanasov <at...@gmail.com>wrote:
> Hi,
>
> I see the threads of the tomcat7.exe process in the Windows Task manager.
>
> Regards,
> Atanas Atanasov
>
>
> On Thu, Apr 10, 2014 at 5:28 PM, Aman Tandon <amantandon.10@gmail.com
> >wrote:
>
> > Hi Atanas,
> >
> > I have a question that, how do know that how much thread does the tomcat
> > has?
> >
> > Thanks
> > Aman Tandon
> >
> >
> > On Thu, Apr 10, 2014 at 7:53 PM, Erick Erickson <erickerickson@gmail.com
> > >wrote:
> >
> > > I don't expect having equal values to make a noticeable difference,
> > > except possibly in some corner cases. Setting them equal is mostly for
> > > avoiding surprises...
> > >
> > > Erick
> > >
> > > On Thu, Apr 10, 2014 at 7:17 AM, Atanas Atanasov <atanasovit@gmail.com
> >
> > > wrote:
> > > > Thanks for the tip,
> > > >
> > > > I already set the core properties. Now tomcat has only 27 threads
> after
> > > > start up, which is awesome.
> > > > Works fine, first search is not noticeably slower than before.
> > > > I'll put equal values for Xmx and Xms and see if there will be any
> > > > difference.
> > > >
> > > > Regards,
> > > > Atanas
> > > >
> > > >
> > > > On Thu, Apr 10, 2014 at 5:11 PM, Erick Erickson <
> > erickerickson@gmail.com
> > > >wrote:
> > > >
> > > >> Trying to fit 1,000 cores in 6G of memory is... interesting. That's
> a
> > > >> lot of stuff in a small amount of memory. I hope these cores'
> indexes
> > > >> are tiny.
> > > >>
> > > >> The lazy-loading bit for cores has a price. The first user in will
> pay
> > > >> the warmup penalty for that core while it loads. This may or may not
> > > >> be noticeable but be aware of it. You may or may not want
> autowarming
> > > >> in place.
> > > >>
> > > >> You can also specify how many cores are kept in memory at one time,
> > > >> they go into an LRU cache and are aged out after they serve their
> last
> > > >> outstanding request.
> > > >>
> > > >> BTW, current Java practice seems to be setting Xmx and Xms to the
> same
> > > >> value, 6G in your case.
> > > >>
> > > >> Good Luck!
> > > >> Erick
> > > >>
> > > >> On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov <
> > atanasovit@gmail.com
> > > >
> > > >> wrote:
> > > >> > Thanks for the quick responses,
> > > >> > I have allocated 1GB min and 6 GB max memory to Java. The cache
> > > settings
> > > >> > are the default ones (maybe this is a good point to start).
> > > >> > All cores share the same schema and config.
> > > >> > I'll try setting the
> > > >> > loadOnStartup=*false* transient=*true *options for each core and
> see
> > > what
> > > >> > will happen.
> > > >> >
> > > >> > Those are the exceptions from the log files:
> > > >> > SEVERE: Servlet.service() for servlet [default] in context with
> path
> > > >> > [/solrt] threw exception
> > > >> > java.lang.IllegalStateException: Cannot call sendError() after the
> > > >> response
> > > >> > has been committed
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> > > >> > at
> > > >>
> > >
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
> > > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> > > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source)
> > > >> > at java.lang.Thread.run(Unknown Source)
> > > >> >
> > > >> > AND
> > > >> >
> > > >> > SEVERE: null:ClientAbortException: java.net.SocketException:
> > Software
> > > >> > caused connection abort: socket write error
> > > >> > at
> > > >>
> > >
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371)
> > > >> > at
> > > >>
> > org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
> > > >> > at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
> > > >> > at sun.nio.cs.StreamEncoder.flush(Unknown Source)
> > > >> > at java.io.OutputStreamWriter.flush(Unknown Source)
> > > >> > at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> > > >> > at
> > > >>
> > >
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
> > > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> > > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > Source)
> > > >> > at java.lang.Thread.run(Unknown Source)
> > > >> > Caused by: java.net.SocketException: Software caused connection
> > abort:
> > > >> > socket write error
> > > >> > at java.net.SocketOutputStream.socketWrite0(Native Method)
> > > >> > at java.net.SocketOutputStream.socketWrite(Unknown Source)
> > > >> > at java.net.SocketOutputStream.write(Unknown Source)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
> > > >> > at
> > > org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799)
> > > >> > at org.apache.coyote.Response.action(Response.java:174)
> > > >> > at
> > > >>
> > >
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366)
> > > >> > ... 24 more
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Thu, Apr 10, 2014 at 9:51 AM, Alexandre Rafalovitch
> > > >> > <ar...@gmail.com>wrote:
> > > >> >
> > > >> >> Are you using all those cores at once? If not, there is a recent
> > > >> >> settings to allow solr to load cores on demand.
> > > >> >>
> > > >> >> If you are using them all, perhaps you need to look into
> splitting
> > > >> >> them to different machines (horizontal scaling).
> > > >> >>
> > > >> >> What about your caches? How many additional structures you have
> > > >> >> configured for each core? How much memory you allocated to the
> Java
> > > >> >> process. You are probably running out of memory and thrashing
> with
> > a
> > > >> >> swap. I am not even sure Java process can access that much memory
> > in
> > > >> >> one process. You might be better off running multiple Tomcat/Solr
> > > >> >> instances on the same machine with different subsets of cores.
> > > >> >>
> > > >> >> Regards,
> > > >> >> Alex.
> > > >> >> P.s. This is general advice, I don't know the specific issues
> > around
> > > >> >> that version of Solr/Tomcat.
> > > >> >> Personal website: http://www.outerthoughts.com/
> > > >> >> Current project: http://www.solr-start.com/ - Accelerating your
> > Solr
> > > >> >> proficiency
> > > >> >>
> > > >> >>
> > > >> >> On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov <
> > > atanasovit@gmail.com>
> > > >> >> wrote:
> > > >> >> > Hi, guys,
> > > >> >> >
> > > >> >> > I need some help. After updating to SOLR 4.4 the tomcat process
> > is
> > > >> >> > consuming about 2GBs of memory, the CPU usage is about 40%
> after
> > > the
> > > >> >> start
> > > >> >> > for about 10 minutes. However, the bigger problem is, I have
> > about
> > > >> 1000
> > > >> >> > cores and seems that for each core a thread is created. The
> > process
> > > >> has
> > > >> >> > more than 1000 threads and everything is extremely slow.
> Creating
> > > or
> > > >> >> > unloading a core even without documents takes about 20 minutes.
> > > >> Searching
> > > >> >> > is more or less good, but storing also takes a lot.
> > > >> >> > Is there some configuration I missed or that I did wrong? There
> > > aren't
> > > >> >> many
> > > >> >> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The
> > > >> machine
> > > >> >> has
> > > >> >> > 24 GBs of RAM, a CPU with 16 cores and is running Windows
> Server
> > > 2008
> > > >> R2.
> > > >> >> > Index is uppdated every 30 seconds/10 000 documents.
> > > >> >> > I haven't checked the number of threads before the update,
> > because
> > > I
> > > >> >> didn't
> > > >> >> > have to, it was working just fine. Any suggestion will be
> highly
> > > >> >> > appreciated, thank you in advance.
> > > >> >> >
> > > >> >> > Regards,
> > > >> >> > Atanas
> > > >> >>
> > > >>
> > >
> >
>
Re: Tomcat creates a thread for each SOLR core
Posted by Atanas Atanasov <at...@gmail.com>.
Hi,
I see the threads of the tomcat7.exe process in the Windows Task manager.
Regards,
Atanas Atanasov
On Thu, Apr 10, 2014 at 5:28 PM, Aman Tandon <am...@gmail.com>wrote:
> Hi Atanas,
>
> I have a question that, how do know that how much thread does the tomcat
> has?
>
> Thanks
> Aman Tandon
>
>
> On Thu, Apr 10, 2014 at 7:53 PM, Erick Erickson <erickerickson@gmail.com
> >wrote:
>
> > I don't expect having equal values to make a noticeable difference,
> > except possibly in some corner cases. Setting them equal is mostly for
> > avoiding surprises...
> >
> > Erick
> >
> > On Thu, Apr 10, 2014 at 7:17 AM, Atanas Atanasov <at...@gmail.com>
> > wrote:
> > > Thanks for the tip,
> > >
> > > I already set the core properties. Now tomcat has only 27 threads after
> > > start up, which is awesome.
> > > Works fine, first search is not noticeably slower than before.
> > > I'll put equal values for Xmx and Xms and see if there will be any
> > > difference.
> > >
> > > Regards,
> > > Atanas
> > >
> > >
> > > On Thu, Apr 10, 2014 at 5:11 PM, Erick Erickson <
> erickerickson@gmail.com
> > >wrote:
> > >
> > >> Trying to fit 1,000 cores in 6G of memory is... interesting. That's a
> > >> lot of stuff in a small amount of memory. I hope these cores' indexes
> > >> are tiny.
> > >>
> > >> The lazy-loading bit for cores has a price. The first user in will pay
> > >> the warmup penalty for that core while it loads. This may or may not
> > >> be noticeable but be aware of it. You may or may not want autowarming
> > >> in place.
> > >>
> > >> You can also specify how many cores are kept in memory at one time,
> > >> they go into an LRU cache and are aged out after they serve their last
> > >> outstanding request.
> > >>
> > >> BTW, current Java practice seems to be setting Xmx and Xms to the same
> > >> value, 6G in your case.
> > >>
> > >> Good Luck!
> > >> Erick
> > >>
> > >> On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov <
> atanasovit@gmail.com
> > >
> > >> wrote:
> > >> > Thanks for the quick responses,
> > >> > I have allocated 1GB min and 6 GB max memory to Java. The cache
> > settings
> > >> > are the default ones (maybe this is a good point to start).
> > >> > All cores share the same schema and config.
> > >> > I'll try setting the
> > >> > loadOnStartup=*false* transient=*true *options for each core and see
> > what
> > >> > will happen.
> > >> >
> > >> > Those are the exceptions from the log files:
> > >> > SEVERE: Servlet.service() for servlet [default] in context with path
> > >> > [/solrt] threw exception
> > >> > java.lang.IllegalStateException: Cannot call sendError() after the
> > >> response
> > >> > has been committed
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450)
> > >> > at
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695)
> > >> > at
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383)
> > >> > at
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> > >> > at
> > >>
> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> > >> > at
> > >> >
> > >>
> >
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> > >> > at
> > >> >
> > >>
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> > >> > at
> > >> >
> > >>
> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
> > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> > >> > at java.lang.Thread.run(Unknown Source)
> > >> >
> > >> > AND
> > >> >
> > >> > SEVERE: null:ClientAbortException: java.net.SocketException:
> Software
> > >> > caused connection abort: socket write error
> > >> > at
> > >>
> > org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371)
> > >> > at
> > >>
> org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
> > >> > at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
> > >> > at sun.nio.cs.StreamEncoder.flush(Unknown Source)
> > >> > at java.io.OutputStreamWriter.flush(Unknown Source)
> > >> > at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
> > >> > at
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648)
> > >> > at
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)
> > >> > at
> > >> >
> > >>
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> > >> > at
> > >>
> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > >> > at
> > >> >
> > >>
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> > >> > at
> > >> >
> > >>
> >
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> > >> > at
> > >> >
> > >>
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> > >> > at
> > >> >
> > >>
> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
> > >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> > >> > at java.lang.Thread.run(Unknown Source)
> > >> > Caused by: java.net.SocketException: Software caused connection
> abort:
> > >> > socket write error
> > >> > at java.net.SocketOutputStream.socketWrite0(Native Method)
> > >> > at java.net.SocketOutputStream.socketWrite(Unknown Source)
> > >> > at java.net.SocketOutputStream.write(Unknown Source)
> > >> > at
> > >> >
> > >>
> >
> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
> > >> > at
> > org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
> > >> > at
> > >> >
> > >>
> >
> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
> > >> > at
> > >> >
> > >>
> >
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799)
> > >> > at org.apache.coyote.Response.action(Response.java:174)
> > >> > at
> > >>
> > org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366)
> > >> > ... 24 more
> > >> >
> > >> >
> > >> >
> > >> > On Thu, Apr 10, 2014 at 9:51 AM, Alexandre Rafalovitch
> > >> > <ar...@gmail.com>wrote:
> > >> >
> > >> >> Are you using all those cores at once? If not, there is a recent
> > >> >> settings to allow solr to load cores on demand.
> > >> >>
> > >> >> If you are using them all, perhaps you need to look into splitting
> > >> >> them to different machines (horizontal scaling).
> > >> >>
> > >> >> What about your caches? How many additional structures you have
> > >> >> configured for each core? How much memory you allocated to the Java
> > >> >> process. You are probably running out of memory and thrashing with
> a
> > >> >> swap. I am not even sure Java process can access that much memory
> in
> > >> >> one process. You might be better off running multiple Tomcat/Solr
> > >> >> instances on the same machine with different subsets of cores.
> > >> >>
> > >> >> Regards,
> > >> >> Alex.
> > >> >> P.s. This is general advice, I don't know the specific issues
> around
> > >> >> that version of Solr/Tomcat.
> > >> >> Personal website: http://www.outerthoughts.com/
> > >> >> Current project: http://www.solr-start.com/ - Accelerating your
> Solr
> > >> >> proficiency
> > >> >>
> > >> >>
> > >> >> On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov <
> > atanasovit@gmail.com>
> > >> >> wrote:
> > >> >> > Hi, guys,
> > >> >> >
> > >> >> > I need some help. After updating to SOLR 4.4 the tomcat process
> is
> > >> >> > consuming about 2GBs of memory, the CPU usage is about 40% after
> > the
> > >> >> start
> > >> >> > for about 10 minutes. However, the bigger problem is, I have
> about
> > >> 1000
> > >> >> > cores and seems that for each core a thread is created. The
> process
> > >> has
> > >> >> > more than 1000 threads and everything is extremely slow. Creating
> > or
> > >> >> > unloading a core even without documents takes about 20 minutes.
> > >> Searching
> > >> >> > is more or less good, but storing also takes a lot.
> > >> >> > Is there some configuration I missed or that I did wrong? There
> > aren't
> > >> >> many
> > >> >> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The
> > >> machine
> > >> >> has
> > >> >> > 24 GBs of RAM, a CPU with 16 cores and is running Windows Server
> > 2008
> > >> R2.
> > >> >> > Index is uppdated every 30 seconds/10 000 documents.
> > >> >> > I haven't checked the number of threads before the update,
> because
> > I
> > >> >> didn't
> > >> >> > have to, it was working just fine. Any suggestion will be highly
> > >> >> > appreciated, thank you in advance.
> > >> >> >
> > >> >> > Regards,
> > >> >> > Atanas
> > >> >>
> > >>
> >
>
Re: Tomcat creates a thread for each SOLR core
Posted by Aman Tandon <am...@gmail.com>.
Hi Atanas,
I have a question that, how do know that how much thread does the tomcat
has?
Thanks
Aman Tandon
On Thu, Apr 10, 2014 at 7:53 PM, Erick Erickson <er...@gmail.com>wrote:
> I don't expect having equal values to make a noticeable difference,
> except possibly in some corner cases. Setting them equal is mostly for
> avoiding surprises...
>
> Erick
>
> On Thu, Apr 10, 2014 at 7:17 AM, Atanas Atanasov <at...@gmail.com>
> wrote:
> > Thanks for the tip,
> >
> > I already set the core properties. Now tomcat has only 27 threads after
> > start up, which is awesome.
> > Works fine, first search is not noticeably slower than before.
> > I'll put equal values for Xmx and Xms and see if there will be any
> > difference.
> >
> > Regards,
> > Atanas
> >
> >
> > On Thu, Apr 10, 2014 at 5:11 PM, Erick Erickson <erickerickson@gmail.com
> >wrote:
> >
> >> Trying to fit 1,000 cores in 6G of memory is... interesting. That's a
> >> lot of stuff in a small amount of memory. I hope these cores' indexes
> >> are tiny.
> >>
> >> The lazy-loading bit for cores has a price. The first user in will pay
> >> the warmup penalty for that core while it loads. This may or may not
> >> be noticeable but be aware of it. You may or may not want autowarming
> >> in place.
> >>
> >> You can also specify how many cores are kept in memory at one time,
> >> they go into an LRU cache and are aged out after they serve their last
> >> outstanding request.
> >>
> >> BTW, current Java practice seems to be setting Xmx and Xms to the same
> >> value, 6G in your case.
> >>
> >> Good Luck!
> >> Erick
> >>
> >> On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov <atanasovit@gmail.com
> >
> >> wrote:
> >> > Thanks for the quick responses,
> >> > I have allocated 1GB min and 6 GB max memory to Java. The cache
> settings
> >> > are the default ones (maybe this is a good point to start).
> >> > All cores share the same schema and config.
> >> > I'll try setting the
> >> > loadOnStartup=*false* transient=*true *options for each core and see
> what
> >> > will happen.
> >> >
> >> > Those are the exceptions from the log files:
> >> > SEVERE: Servlet.service() for servlet [default] in context with path
> >> > [/solrt] threw exception
> >> > java.lang.IllegalStateException: Cannot call sendError() after the
> >> response
> >> > has been committed
> >> > at
> >> >
> >>
> org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> >> > at
> >> >
> >>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> >> > at
> >> >
> >>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> >> > at
> >> >
> >>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> >> > at
> >>
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> >> > at
> >> >
> >>
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> >> > at
> >> >
> >>
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> >> > at
> >> >
> >>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> >> > at
> >> >
> >>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> >> > at java.lang.Thread.run(Unknown Source)
> >> >
> >> > AND
> >> >
> >> > SEVERE: null:ClientAbortException: java.net.SocketException: Software
> >> > caused connection abort: socket write error
> >> > at
> >>
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371)
> >> > at
> >> org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333)
> >> > at
> >> >
> >>
> org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
> >> > at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
> >> > at sun.nio.cs.StreamEncoder.flush(Unknown Source)
> >> > at java.io.OutputStreamWriter.flush(Unknown Source)
> >> > at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> >> > at
> >> >
> >>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> >> > at
> >> >
> >>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> >> > at
> >> >
> >>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> >> > at
> >>
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> >> > at
> >> >
> >>
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> >> > at
> >> >
> >>
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> >> > at
> >> >
> >>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> >> > at
> >> >
> >>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
> >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> >> > at java.lang.Thread.run(Unknown Source)
> >> > Caused by: java.net.SocketException: Software caused connection abort:
> >> > socket write error
> >> > at java.net.SocketOutputStream.socketWrite0(Native Method)
> >> > at java.net.SocketOutputStream.socketWrite(Unknown Source)
> >> > at java.net.SocketOutputStream.write(Unknown Source)
> >> > at
> >> >
> >>
> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
> >> > at
> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
> >> > at
> >> >
> >>
> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
> >> > at
> >> >
> >>
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799)
> >> > at org.apache.coyote.Response.action(Response.java:174)
> >> > at
> >>
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366)
> >> > ... 24 more
> >> >
> >> >
> >> >
> >> > On Thu, Apr 10, 2014 at 9:51 AM, Alexandre Rafalovitch
> >> > <ar...@gmail.com>wrote:
> >> >
> >> >> Are you using all those cores at once? If not, there is a recent
> >> >> settings to allow solr to load cores on demand.
> >> >>
> >> >> If you are using them all, perhaps you need to look into splitting
> >> >> them to different machines (horizontal scaling).
> >> >>
> >> >> What about your caches? How many additional structures you have
> >> >> configured for each core? How much memory you allocated to the Java
> >> >> process. You are probably running out of memory and thrashing with a
> >> >> swap. I am not even sure Java process can access that much memory in
> >> >> one process. You might be better off running multiple Tomcat/Solr
> >> >> instances on the same machine with different subsets of cores.
> >> >>
> >> >> Regards,
> >> >> Alex.
> >> >> P.s. This is general advice, I don't know the specific issues around
> >> >> that version of Solr/Tomcat.
> >> >> Personal website: http://www.outerthoughts.com/
> >> >> Current project: http://www.solr-start.com/ - Accelerating your Solr
> >> >> proficiency
> >> >>
> >> >>
> >> >> On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov <
> atanasovit@gmail.com>
> >> >> wrote:
> >> >> > Hi, guys,
> >> >> >
> >> >> > I need some help. After updating to SOLR 4.4 the tomcat process is
> >> >> > consuming about 2GBs of memory, the CPU usage is about 40% after
> the
> >> >> start
> >> >> > for about 10 minutes. However, the bigger problem is, I have about
> >> 1000
> >> >> > cores and seems that for each core a thread is created. The process
> >> has
> >> >> > more than 1000 threads and everything is extremely slow. Creating
> or
> >> >> > unloading a core even without documents takes about 20 minutes.
> >> Searching
> >> >> > is more or less good, but storing also takes a lot.
> >> >> > Is there some configuration I missed or that I did wrong? There
> aren't
> >> >> many
> >> >> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The
> >> machine
> >> >> has
> >> >> > 24 GBs of RAM, a CPU with 16 cores and is running Windows Server
> 2008
> >> R2.
> >> >> > Index is uppdated every 30 seconds/10 000 documents.
> >> >> > I haven't checked the number of threads before the update, because
> I
> >> >> didn't
> >> >> > have to, it was working just fine. Any suggestion will be highly
> >> >> > appreciated, thank you in advance.
> >> >> >
> >> >> > Regards,
> >> >> > Atanas
> >> >>
> >>
>
Re: Tomcat creates a thread for each SOLR core
Posted by Erick Erickson <er...@gmail.com>.
I don't expect having equal values to make a noticeable difference,
except possibly in some corner cases. Setting them equal is mostly for
avoiding surprises...
Erick
On Thu, Apr 10, 2014 at 7:17 AM, Atanas Atanasov <at...@gmail.com> wrote:
> Thanks for the tip,
>
> I already set the core properties. Now tomcat has only 27 threads after
> start up, which is awesome.
> Works fine, first search is not noticeably slower than before.
> I'll put equal values for Xmx and Xms and see if there will be any
> difference.
>
> Regards,
> Atanas
>
>
> On Thu, Apr 10, 2014 at 5:11 PM, Erick Erickson <er...@gmail.com>wrote:
>
>> Trying to fit 1,000 cores in 6G of memory is... interesting. That's a
>> lot of stuff in a small amount of memory. I hope these cores' indexes
>> are tiny.
>>
>> The lazy-loading bit for cores has a price. The first user in will pay
>> the warmup penalty for that core while it loads. This may or may not
>> be noticeable but be aware of it. You may or may not want autowarming
>> in place.
>>
>> You can also specify how many cores are kept in memory at one time,
>> they go into an LRU cache and are aged out after they serve their last
>> outstanding request.
>>
>> BTW, current Java practice seems to be setting Xmx and Xms to the same
>> value, 6G in your case.
>>
>> Good Luck!
>> Erick
>>
>> On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov <at...@gmail.com>
>> wrote:
>> > Thanks for the quick responses,
>> > I have allocated 1GB min and 6 GB max memory to Java. The cache settings
>> > are the default ones (maybe this is a good point to start).
>> > All cores share the same schema and config.
>> > I'll try setting the
>> > loadOnStartup=*false* transient=*true *options for each core and see what
>> > will happen.
>> >
>> > Those are the exceptions from the log files:
>> > SEVERE: Servlet.service() for servlet [default] in context with path
>> > [/solrt] threw exception
>> > java.lang.IllegalStateException: Cannot call sendError() after the
>> response
>> > has been committed
>> > at
>> >
>> org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
>> > at
>> >
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>> > at
>> >
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>> > at
>> >
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
>> > at
>> >
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
>> > at
>> >
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
>> > at
>> >
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
>> > at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
>> > at
>> >
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
>> > at
>> >
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>> > at
>> >
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
>> > at
>> >
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
>> > at
>> >
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> > at java.lang.Thread.run(Unknown Source)
>> >
>> > AND
>> >
>> > SEVERE: null:ClientAbortException: java.net.SocketException: Software
>> > caused connection abort: socket write error
>> > at
>> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371)
>> > at
>> org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333)
>> > at
>> >
>> org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
>> > at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
>> > at sun.nio.cs.StreamEncoder.flush(Unknown Source)
>> > at java.io.OutputStreamWriter.flush(Unknown Source)
>> > at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)
>> > at
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
>> > at
>> >
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
>> > at
>> >
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
>> > at
>> >
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
>> > at
>> >
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
>> > at
>> >
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
>> > at
>> >
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
>> > at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
>> > at
>> >
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
>> > at
>> >
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>> > at
>> >
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
>> > at
>> >
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
>> > at
>> >
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
>> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> > at java.lang.Thread.run(Unknown Source)
>> > Caused by: java.net.SocketException: Software caused connection abort:
>> > socket write error
>> > at java.net.SocketOutputStream.socketWrite0(Native Method)
>> > at java.net.SocketOutputStream.socketWrite(Unknown Source)
>> > at java.net.SocketOutputStream.write(Unknown Source)
>> > at
>> >
>> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
>> > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
>> > at
>> >
>> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
>> > at
>> >
>> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799)
>> > at org.apache.coyote.Response.action(Response.java:174)
>> > at
>> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366)
>> > ... 24 more
>> >
>> >
>> >
>> > On Thu, Apr 10, 2014 at 9:51 AM, Alexandre Rafalovitch
>> > <ar...@gmail.com>wrote:
>> >
>> >> Are you using all those cores at once? If not, there is a recent
>> >> settings to allow solr to load cores on demand.
>> >>
>> >> If you are using them all, perhaps you need to look into splitting
>> >> them to different machines (horizontal scaling).
>> >>
>> >> What about your caches? How many additional structures you have
>> >> configured for each core? How much memory you allocated to the Java
>> >> process. You are probably running out of memory and thrashing with a
>> >> swap. I am not even sure Java process can access that much memory in
>> >> one process. You might be better off running multiple Tomcat/Solr
>> >> instances on the same machine with different subsets of cores.
>> >>
>> >> Regards,
>> >> Alex.
>> >> P.s. This is general advice, I don't know the specific issues around
>> >> that version of Solr/Tomcat.
>> >> Personal website: http://www.outerthoughts.com/
>> >> Current project: http://www.solr-start.com/ - Accelerating your Solr
>> >> proficiency
>> >>
>> >>
>> >> On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov <at...@gmail.com>
>> >> wrote:
>> >> > Hi, guys,
>> >> >
>> >> > I need some help. After updating to SOLR 4.4 the tomcat process is
>> >> > consuming about 2GBs of memory, the CPU usage is about 40% after the
>> >> start
>> >> > for about 10 minutes. However, the bigger problem is, I have about
>> 1000
>> >> > cores and seems that for each core a thread is created. The process
>> has
>> >> > more than 1000 threads and everything is extremely slow. Creating or
>> >> > unloading a core even without documents takes about 20 minutes.
>> Searching
>> >> > is more or less good, but storing also takes a lot.
>> >> > Is there some configuration I missed or that I did wrong? There aren't
>> >> many
>> >> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The
>> machine
>> >> has
>> >> > 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008
>> R2.
>> >> > Index is uppdated every 30 seconds/10 000 documents.
>> >> > I haven't checked the number of threads before the update, because I
>> >> didn't
>> >> > have to, it was working just fine. Any suggestion will be highly
>> >> > appreciated, thank you in advance.
>> >> >
>> >> > Regards,
>> >> > Atanas
>> >>
>>
Re: Tomcat creates a thread for each SOLR core
Posted by Atanas Atanasov <at...@gmail.com>.
Thanks for the tip,
I already set the core properties. Now tomcat has only 27 threads after
start up, which is awesome.
Works fine, first search is not noticeably slower than before.
I'll put equal values for Xmx and Xms and see if there will be any
difference.
Regards,
Atanas
On Thu, Apr 10, 2014 at 5:11 PM, Erick Erickson <er...@gmail.com>wrote:
> Trying to fit 1,000 cores in 6G of memory is... interesting. That's a
> lot of stuff in a small amount of memory. I hope these cores' indexes
> are tiny.
>
> The lazy-loading bit for cores has a price. The first user in will pay
> the warmup penalty for that core while it loads. This may or may not
> be noticeable but be aware of it. You may or may not want autowarming
> in place.
>
> You can also specify how many cores are kept in memory at one time,
> they go into an LRU cache and are aged out after they serve their last
> outstanding request.
>
> BTW, current Java practice seems to be setting Xmx and Xms to the same
> value, 6G in your case.
>
> Good Luck!
> Erick
>
> On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov <at...@gmail.com>
> wrote:
> > Thanks for the quick responses,
> > I have allocated 1GB min and 6 GB max memory to Java. The cache settings
> > are the default ones (maybe this is a good point to start).
> > All cores share the same schema and config.
> > I'll try setting the
> > loadOnStartup=*false* transient=*true *options for each core and see what
> > will happen.
> >
> > Those are the exceptions from the log files:
> > SEVERE: Servlet.service() for servlet [default] in context with path
> > [/solrt] threw exception
> > java.lang.IllegalStateException: Cannot call sendError() after the
> response
> > has been committed
> > at
> >
> org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> > at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> > at
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> > at
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> > at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> > at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > at
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> > at
> >
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> > at
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> > at
> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > at java.lang.Thread.run(Unknown Source)
> >
> > AND
> >
> > SEVERE: null:ClientAbortException: java.net.SocketException: Software
> > caused connection abort: socket write error
> > at
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371)
> > at
> org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333)
> > at
> >
> org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
> > at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
> > at sun.nio.cs.StreamEncoder.flush(Unknown Source)
> > at java.io.OutputStreamWriter.flush(Unknown Source)
> > at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> > at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> > at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> > at
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> > at
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> > at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> > at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > at
> >
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> > at
> >
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> > at
> >
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> > at
> >
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > at java.lang.Thread.run(Unknown Source)
> > Caused by: java.net.SocketException: Software caused connection abort:
> > socket write error
> > at java.net.SocketOutputStream.socketWrite0(Native Method)
> > at java.net.SocketOutputStream.socketWrite(Unknown Source)
> > at java.net.SocketOutputStream.write(Unknown Source)
> > at
> >
> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
> > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
> > at
> >
> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
> > at
> >
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799)
> > at org.apache.coyote.Response.action(Response.java:174)
> > at
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366)
> > ... 24 more
> >
> >
> >
> > On Thu, Apr 10, 2014 at 9:51 AM, Alexandre Rafalovitch
> > <ar...@gmail.com>wrote:
> >
> >> Are you using all those cores at once? If not, there is a recent
> >> settings to allow solr to load cores on demand.
> >>
> >> If you are using them all, perhaps you need to look into splitting
> >> them to different machines (horizontal scaling).
> >>
> >> What about your caches? How many additional structures you have
> >> configured for each core? How much memory you allocated to the Java
> >> process. You are probably running out of memory and thrashing with a
> >> swap. I am not even sure Java process can access that much memory in
> >> one process. You might be better off running multiple Tomcat/Solr
> >> instances on the same machine with different subsets of cores.
> >>
> >> Regards,
> >> Alex.
> >> P.s. This is general advice, I don't know the specific issues around
> >> that version of Solr/Tomcat.
> >> Personal website: http://www.outerthoughts.com/
> >> Current project: http://www.solr-start.com/ - Accelerating your Solr
> >> proficiency
> >>
> >>
> >> On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov <at...@gmail.com>
> >> wrote:
> >> > Hi, guys,
> >> >
> >> > I need some help. After updating to SOLR 4.4 the tomcat process is
> >> > consuming about 2GBs of memory, the CPU usage is about 40% after the
> >> start
> >> > for about 10 minutes. However, the bigger problem is, I have about
> 1000
> >> > cores and seems that for each core a thread is created. The process
> has
> >> > more than 1000 threads and everything is extremely slow. Creating or
> >> > unloading a core even without documents takes about 20 minutes.
> Searching
> >> > is more or less good, but storing also takes a lot.
> >> > Is there some configuration I missed or that I did wrong? There aren't
> >> many
> >> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The
> machine
> >> has
> >> > 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008
> R2.
> >> > Index is uppdated every 30 seconds/10 000 documents.
> >> > I haven't checked the number of threads before the update, because I
> >> didn't
> >> > have to, it was working just fine. Any suggestion will be highly
> >> > appreciated, thank you in advance.
> >> >
> >> > Regards,
> >> > Atanas
> >>
>
Re: Tomcat creates a thread for each SOLR core
Posted by Erick Erickson <er...@gmail.com>.
Trying to fit 1,000 cores in 6G of memory is... interesting. That's a
lot of stuff in a small amount of memory. I hope these cores' indexes
are tiny.
The lazy-loading bit for cores has a price. The first user in will pay
the warmup penalty for that core while it loads. This may or may not
be noticeable but be aware of it. You may or may not want autowarming
in place.
You can also specify how many cores are kept in memory at one time,
they go into an LRU cache and are aged out after they serve their last
outstanding request.
BTW, current Java practice seems to be setting Xmx and Xms to the same
value, 6G in your case.
Good Luck!
Erick
On Thu, Apr 10, 2014 at 12:14 AM, Atanas Atanasov <at...@gmail.com> wrote:
> Thanks for the quick responses,
> I have allocated 1GB min and 6 GB max memory to Java. The cache settings
> are the default ones (maybe this is a good point to start).
> All cores share the same schema and config.
> I'll try setting the
> loadOnStartup=*false* transient=*true *options for each core and see what
> will happen.
>
> Those are the exceptions from the log files:
> SEVERE: Servlet.service() for servlet [default] in context with path
> [/solrt] threw exception
> java.lang.IllegalStateException: Cannot call sendError() after the response
> has been committed
> at
> org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450)
> at
> org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> AND
>
> SEVERE: null:ClientAbortException: java.net.SocketException: Software
> caused connection abort: socket write error
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371)
> at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333)
> at
> org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
> at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
> at sun.nio.cs.StreamEncoder.flush(Unknown Source)
> at java.io.OutputStreamWriter.flush(Unknown Source)
> at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
> at
> org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.net.SocketException: Software caused connection abort:
> socket write error
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(Unknown Source)
> at java.net.SocketOutputStream.write(Unknown Source)
> at
> org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
> at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
> at
> org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799)
> at org.apache.coyote.Response.action(Response.java:174)
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366)
> ... 24 more
>
>
>
> On Thu, Apr 10, 2014 at 9:51 AM, Alexandre Rafalovitch
> <ar...@gmail.com>wrote:
>
>> Are you using all those cores at once? If not, there is a recent
>> settings to allow solr to load cores on demand.
>>
>> If you are using them all, perhaps you need to look into splitting
>> them to different machines (horizontal scaling).
>>
>> What about your caches? How many additional structures you have
>> configured for each core? How much memory you allocated to the Java
>> process. You are probably running out of memory and thrashing with a
>> swap. I am not even sure Java process can access that much memory in
>> one process. You might be better off running multiple Tomcat/Solr
>> instances on the same machine with different subsets of cores.
>>
>> Regards,
>> Alex.
>> P.s. This is general advice, I don't know the specific issues around
>> that version of Solr/Tomcat.
>> Personal website: http://www.outerthoughts.com/
>> Current project: http://www.solr-start.com/ - Accelerating your Solr
>> proficiency
>>
>>
>> On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov <at...@gmail.com>
>> wrote:
>> > Hi, guys,
>> >
>> > I need some help. After updating to SOLR 4.4 the tomcat process is
>> > consuming about 2GBs of memory, the CPU usage is about 40% after the
>> start
>> > for about 10 minutes. However, the bigger problem is, I have about 1000
>> > cores and seems that for each core a thread is created. The process has
>> > more than 1000 threads and everything is extremely slow. Creating or
>> > unloading a core even without documents takes about 20 minutes. Searching
>> > is more or less good, but storing also takes a lot.
>> > Is there some configuration I missed or that I did wrong? There aren't
>> many
>> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine
>> has
>> > 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2.
>> > Index is uppdated every 30 seconds/10 000 documents.
>> > I haven't checked the number of threads before the update, because I
>> didn't
>> > have to, it was working just fine. Any suggestion will be highly
>> > appreciated, thank you in advance.
>> >
>> > Regards,
>> > Atanas
>>
Re: Tomcat creates a thread for each SOLR core
Posted by Atanas Atanasov <at...@gmail.com>.
Thanks for the quick responses,
I have allocated 1GB min and 6 GB max memory to Java. The cache settings
are the default ones (maybe this is a good point to start).
All cores share the same schema and config.
I'll try setting the
loadOnStartup=*false* transient=*true *options for each core and see what
will happen.
Those are the exceptions from the log files:
SEVERE: Servlet.service() for servlet [default] in context with path
[/solrt] threw exception
java.lang.IllegalStateException: Cannot call sendError() after the response
has been committed
at
org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:450)
at
org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:695)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:383)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
AND
SEVERE: null:ClientAbortException: java.net.SocketException: Software
caused connection abort: socket write error
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:371)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:333)
at
org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
at sun.nio.cs.StreamEncoder.flush(Unknown Source)
at java.io.OutputStreamWriter.flush(Unknown Source)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:648)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:375)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Software caused connection abort:
socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
at
org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:799)
at org.apache.coyote.Response.action(Response.java:174)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:366)
... 24 more
On Thu, Apr 10, 2014 at 9:51 AM, Alexandre Rafalovitch
<ar...@gmail.com>wrote:
> Are you using all those cores at once? If not, there is a recent
> settings to allow solr to load cores on demand.
>
> If you are using them all, perhaps you need to look into splitting
> them to different machines (horizontal scaling).
>
> What about your caches? How many additional structures you have
> configured for each core? How much memory you allocated to the Java
> process. You are probably running out of memory and thrashing with a
> swap. I am not even sure Java process can access that much memory in
> one process. You might be better off running multiple Tomcat/Solr
> instances on the same machine with different subsets of cores.
>
> Regards,
> Alex.
> P.s. This is general advice, I don't know the specific issues around
> that version of Solr/Tomcat.
> Personal website: http://www.outerthoughts.com/
> Current project: http://www.solr-start.com/ - Accelerating your Solr
> proficiency
>
>
> On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov <at...@gmail.com>
> wrote:
> > Hi, guys,
> >
> > I need some help. After updating to SOLR 4.4 the tomcat process is
> > consuming about 2GBs of memory, the CPU usage is about 40% after the
> start
> > for about 10 minutes. However, the bigger problem is, I have about 1000
> > cores and seems that for each core a thread is created. The process has
> > more than 1000 threads and everything is extremely slow. Creating or
> > unloading a core even without documents takes about 20 minutes. Searching
> > is more or less good, but storing also takes a lot.
> > Is there some configuration I missed or that I did wrong? There aren't
> many
> > calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine
> has
> > 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2.
> > Index is uppdated every 30 seconds/10 000 documents.
> > I haven't checked the number of threads before the update, because I
> didn't
> > have to, it was working just fine. Any suggestion will be highly
> > appreciated, thank you in advance.
> >
> > Regards,
> > Atanas
>
Re: Tomcat creates a thread for each SOLR core
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Are you using all those cores at once? If not, there is a recent
settings to allow solr to load cores on demand.
If you are using them all, perhaps you need to look into splitting
them to different machines (horizontal scaling).
What about your caches? How many additional structures you have
configured for each core? How much memory you allocated to the Java
process. You are probably running out of memory and thrashing with a
swap. I am not even sure Java process can access that much memory in
one process. You might be better off running multiple Tomcat/Solr
instances on the same machine with different subsets of cores.
Regards,
Alex.
P.s. This is general advice, I don't know the specific issues around
that version of Solr/Tomcat.
Personal website: http://www.outerthoughts.com/
Current project: http://www.solr-start.com/ - Accelerating your Solr proficiency
On Thu, Apr 10, 2014 at 1:40 PM, Atanas Atanasov <at...@gmail.com> wrote:
> Hi, guys,
>
> I need some help. After updating to SOLR 4.4 the tomcat process is
> consuming about 2GBs of memory, the CPU usage is about 40% after the start
> for about 10 minutes. However, the bigger problem is, I have about 1000
> cores and seems that for each core a thread is created. The process has
> more than 1000 threads and everything is extremely slow. Creating or
> unloading a core even without documents takes about 20 minutes. Searching
> is more or less good, but storing also takes a lot.
> Is there some configuration I missed or that I did wrong? There aren't many
> calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine has
> 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2.
> Index is uppdated every 30 seconds/10 000 documents.
> I haven't checked the number of threads before the update, because I didn't
> have to, it was working just fine. Any suggestion will be highly
> appreciated, thank you in advance.
>
> Regards,
> Atanas
Re: Tomcat creates a thread for each SOLR core
Posted by Aman Tandon <am...@gmail.com>.
I guess this is definitely due to the firstsearcher defined in
solrconfig.xml, you must make some tweaks in that I hope it will help. We
are using the same typo which you just mentioned here but we are using the
indexing server separately and replicating data to our other two server so
that it won't harm any search performance.
Thanks
Aman Tandon
On Thu, Apr 10, 2014 at 12:10 PM, Atanas Atanasov <at...@gmail.com>wrote:
> Hi, guys,
>
> I need some help. After updating to SOLR 4.4 the tomcat process is
> consuming about 2GBs of memory, the CPU usage is about 40% after the start
> for about 10 minutes. However, the bigger problem is, I have about 1000
> cores and seems that for each core a thread is created. The process has
> more than 1000 threads and everything is extremely slow. Creating or
> unloading a core even without documents takes about 20 minutes. Searching
> is more or less good, but storing also takes a lot.
> Is there some configuration I missed or that I did wrong? There aren't many
> calls, I use 64 bit tomcat 7, SOLR 4.4, latest 64 bit Java. The machine has
> 24 GBs of RAM, a CPU with 16 cores and is running Windows Server 2008 R2.
> Index is uppdated every 30 seconds/10 000 documents.
> I haven't checked the number of threads before the update, because I didn't
> have to, it was working just fine. Any suggestion will be highly
> appreciated, thank you in advance.
>
> Regards,
> Atanas
>