Large import making solr unresponsive

[Brent Mills <bm...@uship.com>]

This is an issue we've only been running into lately so I'm not sure what to make of it.  We have 2 cores on a solr machine right now, one of them is about 10k documents, the other is about 1.5mil.  None of the documents are very large, only about 30 short attributes.  We also have about 10 requests/sec hitting the smaller core and less on the larger one.  Whenever we try to do a full import on the smaller one everything is fine, the response times stay the same during the whole 30 seconds it takes to run the indexer.  The cpu also stays fairly low.

When we run a full import on the larger one the response times on all cores tank from about 10ms to over 8 seconds.  We have a 4 core machine (VM) and I've noticed 1 core stays pegged the entire time which is understandable since the DIH as I understand it is single threaded.  Also, from what I can tell there is no disk, network, or memory pressure (8gb) either and the other procs do virtually nothing.  Also the responses from solr still come back with a <10ms qtime.  My best guess at this point is tomcat is having issues when the single proc gets pegged but I'm at a loss on how to further diagnose this to a tomcat issue or something weird that solr is doing.

Has anyone run into this before or have ideas about what might be happening?

Re: Large import making solr unresponsive

[Otis Gospodnetic <ot...@gmail.com>]

Hi Brent,

You said "from what I can tell there is no disk, network, or memory pressure "
- maybe you can share what and how you checked this? (see my signature for
a tool that can help with this)

I'm asking because the above is in conflict with "responses from solr still
come back with a <10ms qtime", which indicate search itself was fast, but
either disk or network were slow.  Try with rows=<big number here> and
rows=0 and that will give you an idea where to look.

Otis
--
SOLR Performance Monitoring - http://sematext.com/spm/index.html





On Mon, Dec 17, 2012 at 1:04 AM, Brent Mills <bm...@uship.com> wrote:

> This is an issue we've only been running into lately so I'm not sure what
> to make of it.  We have 2 cores on a solr machine right now, one of them is
> about 10k documents, the other is about 1.5mil.  None of the documents are
> very large, only about 30 short attributes.  We also have about 10
> requests/sec hitting the smaller core and less on the larger one.  Whenever
> we try to do a full import on the smaller one everything is fine, the
> response times stay the same during the whole 30 seconds it takes to run
> the indexer.  The cpu also stays fairly low.
>
> When we run a full import on the larger one the response times on all
> cores tank from about 10ms to over 8 seconds.  We have a 4 core machine
> (VM) and I've noticed 1 core stays pegged the entire time which is
> understandable since the DIH as I understand it is single threaded.  Also,
> from what I can tell there is no disk, network, or memory pressure (8gb)
> either and the other procs do virtually nothing.  Also the responses from
> solr still come back with a <10ms qtime.  My best guess at this point is
> tomcat is having issues when the single proc gets pegged but I'm at a loss
> on how to further diagnose this to a tomcat issue or something weird that
> solr is doing.
>
> Has anyone run into this before or have ideas about what might be
> happening?
>