Solr I/O increases over time

[Daniel Cukier <da...@gmail.com>]

I am running around eight solr servers (version 3.5) instances behind a
Load Balancer. All servers are identical and the LB is weighted by number
connections. The servers have around 4M documents and receive a constant
flow of queries. When the solr server starts, it works fine. But after some
time running, it starts to take longer respond to queries, and the server
I/O goes crazy to 100%. Look at the New Relic graphic:

[image: enter image description here]

If the servers behaves well in the beginning, I it starts to fail after
some time? Then if I restart the server, it gets back to low I/O for same
time and this repeats over and over.
Daniel Cukier

Re: Solr I/O increases over time

[Shawn Heisey <ap...@elyograg.org>]

On 1/23/2015 3:52 PM, Daniel Cukier wrote:
> I am running around eight solr servers (version 3.5) instances behind a
> Load Balancer. All servers are identical and the LB is weighted by number
> connections. The servers have around 4M documents and receive a constant
> flow of queries. When the solr server starts, it works fine. But after some
> time running, it starts to take longer respond to queries, and the server
> I/O goes crazy to 100%. Look at the New Relic graphic:
>
> [image: enter image description here]
>
> If the servers behaves well in the beginning, I it starts to fail after
> some time? Then if I restart the server, it gets back to low I/O for same
> time and this repeats over and over.

The mailing list eats almost all attachments.  We can't see your image. 
You can use http://apaste.info for images (up to 1MB) and text, or pick
another hosting provider, and include the URL in your reply.

Most performance problems like this are memory related.  The high I/O
you mentioned definitely sounds like it could be a situation where you
don't have enough RAM available for OS disk cache.  When the OS cannot
cache the index effectively, queries will result in a large amount of
real disk I/O.  If there's enough memory for caching, queries will be
entirely or mostly handled from RAM, which is *MUCH* faster than the disk.

http://wiki.apache.org/solr/SolrPerformanceProblems

Thanks,
Shawn

RE: Solr I/O increases over time

[Toke Eskildsen <te...@statsbiblioteket.dk>]

Daniel Cukier [danicuki@gmail.com] wrote:
> The servers have around 4M documents and receive a constant
> flow of queries. When the solr server starts, it works fine. But after some
> time running, it starts to take longer respond to queries, and the server
> I/O goes crazy to 100%. Look at the New Relic graphic:

Two possible explanations:

1) You update the indexes and you commit too frequently, causing the Solrs to have overlapping upstarting searchers, each requiring IO. If you have many warmup queries, this excerberates the issue.

2) Your Xms and Xmx are different, with your Xmx being high, relative to physical memory on the server. At the beginning the allocated memory for the JVM will be low, leaving a lot of memory for disk caching. As you issue requests, heap memory allocation grows, leaving less memory for disk cache.

3) You have very little memory free for disk cache and a fairly large swap space. For some setups that will cause the OS to swap all the time.

- Toke Eskildsen

Re: Solr I/O increases over time

[Arcadius Ahouansou <ar...@menelic.com>]

On 23 January 2015 at 22:52, Daniel Cukier <da...@gmail.com> wrote:

> I am running around eight solr servers (version 3.5) instances behind a
> Load Balancer. All servers are identical and the LB is weighted by number
> connections. The servers have around 4M documents and receive a constant
> flow of queries. When the solr server starts, it works fine. But after some
> time running, it starts to take longer respond to queries, and the server
> I/O goes crazy to 100%. Look at the New Relic graphic:
>
> [image: enter image description here]
>
> If the servers behaves well in the beginning, I it starts to fail after
> some time? Then if I restart the server, it gets back to low I/O for same
> time and this repeats over and over.
> Daniel Cukier
>


Hello Daniel.
Your I/O issue may be due to clients not always closing/releasing
connections to the Solr Server, or not having a timeout set on the clients
I would suggest that when the incident occurs, you run netstat on the Solr
boxes.
Chances are,  a lot of connections would be in CLOSE_WAIT or TIME_WAIT
state.

Thanks.

Arcadius.

-- 
Arcadius Ahouansou
Menelic Ltd | Information is Power
M: 07908761999
W: www.menelic.com
---