You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Furkan KAMACI <fu...@gmail.com> on 2013/08/25 22:41:59 UTC
Dropping Caches of Machine That Solr Runs At
Sometimes Physical Memory usage of Solr is over %99 and this may cause
problems. Do you run such kind of a command periodically:
sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
to force dropping caches of machine that Solr runs at and avoid problems?
Re: Dropping Caches of Machine That Solr Runs At
Posted by Furkan KAMACI <fu...@gmail.com>.
EOF exception seems like a generic exception for me. I should find the
underlying problem within my infrastructure.
26 Ağustos 2013 Pazartesi tarihinde Walter Underwood <wu...@wunderwood.org>
adlı kullanıcı şöyle yazdı:
> We use Amazon EC2 machines with 34GB of memory (m2.2xlarge). The Solr
heap is 8GB. We have several cores, totaling about 14GB on disk. This
configuration allows 100% of the indexes to be in file buffers.
>
> wunder
>
> On Aug 26, 2013, at 9:57 AM, Furkan KAMACI wrote:
>
>> Hi Walter;
>>
>> You said you are caching your documents. What is average Physical Memory
>> usage of your Solr Nodes?
>>
>>
>> 2013/8/26 Walter Underwood <wu...@wunderwood.org>
>>
>>> It looks lik that error happens when reading XML from an HTTP request.
The
>>> XML ends too soon. This should be unrelated to file buffers.
>>>
>>> wunder
>>>
>>> On Aug 26, 2013, at 9:17 AM, Furkan KAMACI wrote:
>>>
>>>> It has a 48 GB of RAM and index size is nearly 100 GB at each node. I
>>> have
>>>> CentOS 6.4. While indexing I got that error and I am suspicious about
>>> that
>>>> it is because of high percentage of Physical Memory usage.
>>>>
>>>> ERROR - 2013-08-21 22:01:30.979; org.apache.solr.common.SolrException;
>>>> java.lang.RuntimeException: [was class
org.eclipse.jetty.io.EofException]
>>>> early EOF
>>>> at
>>>>
>>>
com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
>>>> at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
>>>> at
>>>>
>>>
com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
>>>> at
com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
>>>> at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:393)
>>>> at
>>>>
>>>
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:245)
>>>> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
>>>> at
>>>>
>>>
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>>>> at
>>>>
>>>
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>>>> at
>>>>
>>>
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1812)
>>>> at
>>>>
>>>
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
>>>> at
>>>>
>>>
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
>>>> at
>>>>
>>>
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
>>>> at
>>>>
>>>
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
>>>> at
>>>>
>>>
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
>>>> at
>>>>
>>>
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>>>> at
>>>>
>>>
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
>>>> at
>>>>
>>>
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>>>> at
>>>>
>>>
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
>>>> at
>>>
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
>>>> at
>>>>
>>>
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>>>> at
>>>>
>>>
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
>>>> at
>>>>
>>>
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>>> at
>>>>
>>>
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>>> at
>>>>
>>>
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>>>> at
>>>>
>>> org.eclipse.jetty.server.handler.--
> Walter Underwood
> wunder@wunderwood.org
>
>
>
>
Re: Dropping Caches of Machine That Solr Runs At
Posted by Walter Underwood <wu...@wunderwood.org>.
We use Amazon EC2 machines with 34GB of memory (m2.2xlarge). The Solr heap is 8GB. We have several cores, totaling about 14GB on disk. This configuration allows 100% of the indexes to be in file buffers.
wunder
On Aug 26, 2013, at 9:57 AM, Furkan KAMACI wrote:
> Hi Walter;
>
> You said you are caching your documents. What is average Physical Memory
> usage of your Solr Nodes?
>
>
> 2013/8/26 Walter Underwood <wu...@wunderwood.org>
>
>> It looks lik that error happens when reading XML from an HTTP request. The
>> XML ends too soon. This should be unrelated to file buffers.
>>
>> wunder
>>
>> On Aug 26, 2013, at 9:17 AM, Furkan KAMACI wrote:
>>
>>> It has a 48 GB of RAM and index size is nearly 100 GB at each node. I
>> have
>>> CentOS 6.4. While indexing I got that error and I am suspicious about
>> that
>>> it is because of high percentage of Physical Memory usage.
>>>
>>> ERROR - 2013-08-21 22:01:30.979; org.apache.solr.common.SolrException;
>>> java.lang.RuntimeException: [was class org.eclipse.jetty.io.EofException]
>>> early EOF
>>> at
>>>
>> com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
>>> at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
>>> at
>>>
>> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
>>> at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
>>> at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:393)
>>> at
>>>
>> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:245)
>>> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
>>> at
>>>
>> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>>> at
>>>
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>>> at
>>>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1812)
>>> at
>>>
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
>>> at
>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
>>> at
>>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
>>> at
>>>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
>>> at
>>>
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
>>> at
>>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>>> at
>>>
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
>>> at
>>>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>>> at
>>>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
>>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
>>> at
>>>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>>> at
>>>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
>>> at
>>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>>> at
>>>
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>>> at
>>>
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>>> at
>>>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>>> at org.eclipse.jetty.server.Server.handle(Server.java:365)
>>> at
>>>
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
>>> at
>>>
>> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
>>> at
>>>
>> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
>>> at
>>>
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
>>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
>>> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>>> at
>>>
>> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
>>> at
>>>
>> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
>>> at
>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>>> at
>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>>> at java.lang.Thread.run(Thread.java:722)
>>> Caused by: org.eclipse.jetty.io.EofException: early EOF
>>> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:65)
>>> at java.io.InputStream.read(InputStream.java:101)
>>> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
>>> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
>>> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
>>> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
>>> at
>>>
>> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
>>> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
>>> at
>>>
>> com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
>>> at
>>>
>> com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
>>> at
>>>
>> com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
>>> at
>>>
>> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
>>> ... 36 more
>>>
>>>
>>>
>>> 2013/8/26 Walter Underwood <wu...@wunderwood.org>
>>>
>>>> What is the precise error? What kind of machine?
>>>>
>>>> File buffers are a robust part of the OS. Unix has had file buffer
>> caching
>>>> for decades.
>>>>
>>>> wunder
>>>>
>>>> On Aug 26, 2013, at 1:37 AM, Furkan KAMACI wrote:
>>>>
>>>>> Hi Walter;
>>>>>
>>>>> You are right about performance. However when I index documents on a
>>>>> machine that has a high percentage of Physical Memory usage I get EOF
>>>>> errors?
>>>>>
>>>>>
>>>>> 2013/8/26 Walter Underwood <wu...@wunderwood.org>
>>>>>
>>>>>> On Aug 25, 2013, at 1:41 PM, Furkan KAMACI wrote:
>>>>>>
>>>>>>> Sometimes Physical Memory usage of Solr is over %99 and this may
>> cause
>>>>>>> problems. Do you run such kind of a command periodically:
>>>>>>>
>>>>>>> sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
>>>>>>>
>>>>>>> to force dropping caches of machine that Solr runs at and avoid
>>>> problems?
>>>>>>
>>>>>>
>>>>>> This is a terrible idea. The OS automatically manages the file
>> buffers.
>>>>>> When they are all used, that is a good thing, because it reduced disk
>>>> IO.
>>>>>>
>>>>>> After this, no files will be cached in RAM. Every single read from a
>>>> file
>>>>>> will have to go to disk. This will cause very slow performance until
>> the
>>>>>> files are recached.
>>>>>>
>>>>>> Recently, I did exactly the opposite to improve performance in our
>> Solr
>>>>>> installation. Before starting the Solr process, a script reads every
>>>> file
>>>>>> in the index so that it will already be in file buffers. This avoids
>>>>>> several minutes of high disk IO and slow performance after startup.
>>>>>>
>>>>>> wunder
>>>>>> Search Guy, Chegg.com
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Walter Underwood
>>>> wunder@wunderwood.org
>>>>
>>>>
>>>>
>>>>
>>
>> --
>> Walter Underwood
>> wunder@wunderwood.org
>>
>>
>>
>>
--
Walter Underwood
wunder@wunderwood.org
Re: Dropping Caches of Machine That Solr Runs At
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi Walter;
You said you are caching your documents. What is average Physical Memory
usage of your Solr Nodes?
2013/8/26 Walter Underwood <wu...@wunderwood.org>
> It looks lik that error happens when reading XML from an HTTP request. The
> XML ends too soon. This should be unrelated to file buffers.
>
> wunder
>
> On Aug 26, 2013, at 9:17 AM, Furkan KAMACI wrote:
>
> > It has a 48 GB of RAM and index size is nearly 100 GB at each node. I
> have
> > CentOS 6.4. While indexing I got that error and I am suspicious about
> that
> > it is because of high percentage of Physical Memory usage.
> >
> > ERROR - 2013-08-21 22:01:30.979; org.apache.solr.common.SolrException;
> > java.lang.RuntimeException: [was class org.eclipse.jetty.io.EofException]
> > early EOF
> > at
> >
> com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
> > at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
> > at
> >
> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
> > at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
> > at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:393)
> > at
> >
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:245)
> > at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
> > at
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> > at
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1812)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> > at org.eclipse.jetty.server.Server.handle(Server.java:365)
> > at
> >
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
> > at
> >
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> > at
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
> > at
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
> > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
> > at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> > at
> >
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> > at
> >
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> > at java.lang.Thread.run(Thread.java:722)
> > Caused by: org.eclipse.jetty.io.EofException: early EOF
> > at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:65)
> > at java.io.InputStream.read(InputStream.java:101)
> > at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
> > at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
> > at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
> > at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
> > at
> >
> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> > at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
> > at
> >
> com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
> > at
> >
> com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
> > at
> >
> com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
> > at
> >
> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
> > ... 36 more
> >
> >
> >
> > 2013/8/26 Walter Underwood <wu...@wunderwood.org>
> >
> >> What is the precise error? What kind of machine?
> >>
> >> File buffers are a robust part of the OS. Unix has had file buffer
> caching
> >> for decades.
> >>
> >> wunder
> >>
> >> On Aug 26, 2013, at 1:37 AM, Furkan KAMACI wrote:
> >>
> >>> Hi Walter;
> >>>
> >>> You are right about performance. However when I index documents on a
> >>> machine that has a high percentage of Physical Memory usage I get EOF
> >>> errors?
> >>>
> >>>
> >>> 2013/8/26 Walter Underwood <wu...@wunderwood.org>
> >>>
> >>>> On Aug 25, 2013, at 1:41 PM, Furkan KAMACI wrote:
> >>>>
> >>>>> Sometimes Physical Memory usage of Solr is over %99 and this may
> cause
> >>>>> problems. Do you run such kind of a command periodically:
> >>>>>
> >>>>> sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
> >>>>>
> >>>>> to force dropping caches of machine that Solr runs at and avoid
> >> problems?
> >>>>
> >>>>
> >>>> This is a terrible idea. The OS automatically manages the file
> buffers.
> >>>> When they are all used, that is a good thing, because it reduced disk
> >> IO.
> >>>>
> >>>> After this, no files will be cached in RAM. Every single read from a
> >> file
> >>>> will have to go to disk. This will cause very slow performance until
> the
> >>>> files are recached.
> >>>>
> >>>> Recently, I did exactly the opposite to improve performance in our
> Solr
> >>>> installation. Before starting the Solr process, a script reads every
> >> file
> >>>> in the index so that it will already be in file buffers. This avoids
> >>>> several minutes of high disk IO and slow performance after startup.
> >>>>
> >>>> wunder
> >>>> Search Guy, Chegg.com
> >>>>
> >>>>
> >>>>
> >>
> >> --
> >> Walter Underwood
> >> wunder@wunderwood.org
> >>
> >>
> >>
> >>
>
> --
> Walter Underwood
> wunder@wunderwood.org
>
>
>
>
Re: Dropping Caches of Machine That Solr Runs At
Posted by Walter Underwood <wu...@wunderwood.org>.
It looks lik that error happens when reading XML from an HTTP request. The XML ends too soon. This should be unrelated to file buffers.
wunder
On Aug 26, 2013, at 9:17 AM, Furkan KAMACI wrote:
> It has a 48 GB of RAM and index size is nearly 100 GB at each node. I have
> CentOS 6.4. While indexing I got that error and I am suspicious about that
> it is because of high percentage of Physical Memory usage.
>
> ERROR - 2013-08-21 22:01:30.979; org.apache.solr.common.SolrException;
> java.lang.RuntimeException: [was class org.eclipse.jetty.io.EofException]
> early EOF
> at
> com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
> at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
> at
> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
> at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
> at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:393)
> at
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:245)
> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1812)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:365)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
> at
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.eclipse.jetty.io.EofException: early EOF
> at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:65)
> at java.io.InputStream.read(InputStream.java:101)
> at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
> at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
> at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
> at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
> at
> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
> at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
> at
> com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
> at
> com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
> at
> com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
> at
> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
> ... 36 more
>
>
>
> 2013/8/26 Walter Underwood <wu...@wunderwood.org>
>
>> What is the precise error? What kind of machine?
>>
>> File buffers are a robust part of the OS. Unix has had file buffer caching
>> for decades.
>>
>> wunder
>>
>> On Aug 26, 2013, at 1:37 AM, Furkan KAMACI wrote:
>>
>>> Hi Walter;
>>>
>>> You are right about performance. However when I index documents on a
>>> machine that has a high percentage of Physical Memory usage I get EOF
>>> errors?
>>>
>>>
>>> 2013/8/26 Walter Underwood <wu...@wunderwood.org>
>>>
>>>> On Aug 25, 2013, at 1:41 PM, Furkan KAMACI wrote:
>>>>
>>>>> Sometimes Physical Memory usage of Solr is over %99 and this may cause
>>>>> problems. Do you run such kind of a command periodically:
>>>>>
>>>>> sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
>>>>>
>>>>> to force dropping caches of machine that Solr runs at and avoid
>> problems?
>>>>
>>>>
>>>> This is a terrible idea. The OS automatically manages the file buffers.
>>>> When they are all used, that is a good thing, because it reduced disk
>> IO.
>>>>
>>>> After this, no files will be cached in RAM. Every single read from a
>> file
>>>> will have to go to disk. This will cause very slow performance until the
>>>> files are recached.
>>>>
>>>> Recently, I did exactly the opposite to improve performance in our Solr
>>>> installation. Before starting the Solr process, a script reads every
>> file
>>>> in the index so that it will already be in file buffers. This avoids
>>>> several minutes of high disk IO and slow performance after startup.
>>>>
>>>> wunder
>>>> Search Guy, Chegg.com
>>>>
>>>>
>>>>
>>
>> --
>> Walter Underwood
>> wunder@wunderwood.org
>>
>>
>>
>>
--
Walter Underwood
wunder@wunderwood.org
Re: Dropping Caches of Machine That Solr Runs At
Posted by Furkan KAMACI <fu...@gmail.com>.
It has a 48 GB of RAM and index size is nearly 100 GB at each node. I have
CentOS 6.4. While indexing I got that error and I am suspicious about that
it is because of high percentage of Physical Memory usage.
ERROR - 2013-08-21 22:01:30.979; org.apache.solr.common.SolrException;
java.lang.RuntimeException: [was class org.eclipse.jetty.io.EofException]
early EOF
at
com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18)
at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731)
at
com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657)
at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809)
at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:393)
at
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:245)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1812)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:365)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.eclipse.jetty.io.EofException: early EOF
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:65)
at java.io.InputStream.read(InputStream.java:101)
at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
at com.ctc.wstx.io.MergedReader.read(MergedReader.java:101)
at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
at
com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
at
com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
at
com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
at
com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
at
com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649)
... 36 more
2013/8/26 Walter Underwood <wu...@wunderwood.org>
> What is the precise error? What kind of machine?
>
> File buffers are a robust part of the OS. Unix has had file buffer caching
> for decades.
>
> wunder
>
> On Aug 26, 2013, at 1:37 AM, Furkan KAMACI wrote:
>
> > Hi Walter;
> >
> > You are right about performance. However when I index documents on a
> > machine that has a high percentage of Physical Memory usage I get EOF
> > errors?
> >
> >
> > 2013/8/26 Walter Underwood <wu...@wunderwood.org>
> >
> >> On Aug 25, 2013, at 1:41 PM, Furkan KAMACI wrote:
> >>
> >>> Sometimes Physical Memory usage of Solr is over %99 and this may cause
> >>> problems. Do you run such kind of a command periodically:
> >>>
> >>> sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
> >>>
> >>> to force dropping caches of machine that Solr runs at and avoid
> problems?
> >>
> >>
> >> This is a terrible idea. The OS automatically manages the file buffers.
> >> When they are all used, that is a good thing, because it reduced disk
> IO.
> >>
> >> After this, no files will be cached in RAM. Every single read from a
> file
> >> will have to go to disk. This will cause very slow performance until the
> >> files are recached.
> >>
> >> Recently, I did exactly the opposite to improve performance in our Solr
> >> installation. Before starting the Solr process, a script reads every
> file
> >> in the index so that it will already be in file buffers. This avoids
> >> several minutes of high disk IO and slow performance after startup.
> >>
> >> wunder
> >> Search Guy, Chegg.com
> >>
> >>
> >>
>
> --
> Walter Underwood
> wunder@wunderwood.org
>
>
>
>
Re: Dropping Caches of Machine That Solr Runs At
Posted by Walter Underwood <wu...@wunderwood.org>.
What is the precise error? What kind of machine?
File buffers are a robust part of the OS. Unix has had file buffer caching for decades.
wunder
On Aug 26, 2013, at 1:37 AM, Furkan KAMACI wrote:
> Hi Walter;
>
> You are right about performance. However when I index documents on a
> machine that has a high percentage of Physical Memory usage I get EOF
> errors?
>
>
> 2013/8/26 Walter Underwood <wu...@wunderwood.org>
>
>> On Aug 25, 2013, at 1:41 PM, Furkan KAMACI wrote:
>>
>>> Sometimes Physical Memory usage of Solr is over %99 and this may cause
>>> problems. Do you run such kind of a command periodically:
>>>
>>> sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
>>>
>>> to force dropping caches of machine that Solr runs at and avoid problems?
>>
>>
>> This is a terrible idea. The OS automatically manages the file buffers.
>> When they are all used, that is a good thing, because it reduced disk IO.
>>
>> After this, no files will be cached in RAM. Every single read from a file
>> will have to go to disk. This will cause very slow performance until the
>> files are recached.
>>
>> Recently, I did exactly the opposite to improve performance in our Solr
>> installation. Before starting the Solr process, a script reads every file
>> in the index so that it will already be in file buffers. This avoids
>> several minutes of high disk IO and slow performance after startup.
>>
>> wunder
>> Search Guy, Chegg.com
>>
>>
>>
--
Walter Underwood
wunder@wunderwood.org
Re: Dropping Caches of Machine That Solr Runs At
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi Walter;
You are right about performance. However when I index documents on a
machine that has a high percentage of Physical Memory usage I get EOF
errors?
2013/8/26 Walter Underwood <wu...@wunderwood.org>
> On Aug 25, 2013, at 1:41 PM, Furkan KAMACI wrote:
>
> > Sometimes Physical Memory usage of Solr is over %99 and this may cause
> > problems. Do you run such kind of a command periodically:
> >
> > sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
> >
> > to force dropping caches of machine that Solr runs at and avoid problems?
>
>
> This is a terrible idea. The OS automatically manages the file buffers.
> When they are all used, that is a good thing, because it reduced disk IO.
>
> After this, no files will be cached in RAM. Every single read from a file
> will have to go to disk. This will cause very slow performance until the
> files are recached.
>
> Recently, I did exactly the opposite to improve performance in our Solr
> installation. Before starting the Solr process, a script reads every file
> in the index so that it will already be in file buffers. This avoids
> several minutes of high disk IO and slow performance after startup.
>
> wunder
> Search Guy, Chegg.com
>
>
>
Re: Dropping Caches of Machine That Solr Runs At
Posted by Walter Underwood <wu...@wunderwood.org>.
On Aug 25, 2013, at 1:41 PM, Furkan KAMACI wrote:
> Sometimes Physical Memory usage of Solr is over %99 and this may cause
> problems. Do you run such kind of a command periodically:
>
> sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
>
> to force dropping caches of machine that Solr runs at and avoid problems?
This is a terrible idea. The OS automatically manages the file buffers. When they are all used, that is a good thing, because it reduced disk IO.
After this, no files will be cached in RAM. Every single read from a file will have to go to disk. This will cause very slow performance until the files are recached.
Recently, I did exactly the opposite to improve performance in our Solr installation. Before starting the Solr process, a script reads every file in the index so that it will already be in file buffers. This avoids several minutes of high disk IO and slow performance after startup.
wunder
Search Guy, Chegg.com
Re: Dropping Caches of Machine That Solr Runs At
Posted by Furkan KAMACI <fu...@gmail.com>.
One of my Solr Nodes at SolrCloud (4.2.1) was down for a long time. I
restarted Solr and after recovery its Physical Memory usage is 99.5% and
does not decrease. Thats why I asked that question (I don't know is it
usual that Physical Memory did not decreased for 3 days. Why my CentOS 6.4
does not drop caches after a time later and why recovery resulted with such
a high Physical Memory usage.)
2013/8/25 Furkan KAMACI <fu...@gmail.com>
> Sometimes Physical Memory usage of Solr is over %99 and this may cause
> problems. Do you run such kind of a command periodically:
>
> sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
>
> to force dropping caches of machine that Solr runs at and avoid problems?
>