You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Markus Fischer <ma...@fischer.name> on 2010/05/03 15:06:41 UTC
Commit takes 1 to 2 minutes, CPU usage affects other apps
Hi,
we recently began having trouble with our Solr 1.4 instance. We've about
850k documents in the index which is about 1.2GB in size; the JVM which
runs tomcat/solr (no other apps are deployed) has been given 2GB.
We've a forum and run a process every minute which indexes the new
messages. The number of messages updated are from 0 to 20 messages
average. The commit takes about 1 or two minutes, but usually when it
finished a few seconds later the next batch of documents is processed
and the story starts again.
So actually it's like Solr is running commits all day long and CPU usage
ranges from 80% to 120%.
This continuous CPU usage caused ill effects on other services running
on the same machine.
Our environment is being providing by a company purely using VMWare
infrastructure, the Solr index itself is on an NSF for which we get some
33MB/s throughput.
So, an easy solution would be to just put more resources into it, e.g. a
separate machine. But before I make the decision I'd like to find out
whether the app behaves properly under this circumstances or if its
possible to shorten the commit time down to a few seconds so the CPU is
not drained that long.
thanks for any pointers,
- Markus
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by Erick Erickson <er...@gmail.com>.
The mail servers are often not too friendly with attachments, so people
either inline configs or put them on a server and post the URL.
HTH
Erick
On Wed, May 5, 2010 at 12:06 PM, Markus Fischer <ma...@fischer.name> wrote:
> Hi,
>
> On 05.05.2010 03:49, Chris Hostetter wrote:
> >
> > : Are you accidentally building the spellchecker database on each commit?
> > ...
> > : > This could also be caused by performing an optimize after the commit,
> or it
> > : > could be caused by auto warming the caches, or a combination of both.
> >
> > The heart of the matter being: it's pretty much impossible to guess what
> > is taking up all this time (and eating up all that CPU) w/o seeing your
> > configs, and having a better idea of how you are timing this "1 to 2
> > minutes" ... is this what your client sending the commit reports? what
> > exactly is the command it's executing?
>
> The measurement was done outside our Solr client which sends the update
> and then the commit to the handler. I also see the update-URL call in
> the Tomcat Manager taking up that amount of time.
>
> > what do your Solr logs say about the commit, and the subsequent
> > newSearcher?
>
> How can I get such logs? I was asking for in the second mail do this
> thread; basically it's the same problem I'm having, as I've no clue were
> it costs that time.
>
> I'll get the configs asap; and yes, we're using a spellchecker too. Is
> this list fine with attachments? Or should I just paste the XML stuff
> inline? Or use pastebin?
>
> thanks for the help!
>
> - Markus
>
[Solved] Re: Commit takes 1 to 2 minutes, CPU usage affects other
apps
Posted by Markus Fischer <ma...@fischer.name>.
Hi,
On 07.05.2010 22:47, Chris Hostetter wrote:
> so it's the full request time, and would be inclusive of any postCommit
> event handlers -- that's important to know. the logs will help clear up
> wether the underlying "commit" is really taking up a large amount of time
> or if it's some postCommit even (like spellcheck index building, or
> snapshooting, etc...)
I finally was able to solve the issue and you hit the nail pretty hard:
the spellchecker was the culprit. I assume it was a) an oversight or b)
ignorance on our part about the spellchecker.
I wasn't aware that building the spellchecker index took so long (2 or
more minutes for around 800.000 documents); we disabled buildOnCommit
and only build the spellchecker once every night.
>
> :> what do your Solr logs say about the commit, and the subsequent
> :> newSearcher?
> :
> : How can I get such logs? I was asking for in the second mail do this
>
> it depends on what servlet container you use to run Solr, and how it's
> configured. In the simple the "java -jar start.jar" jetty setup used for
> the Solr example, jetty dumps them to STDOUT in your console. But most
> servlet containers will write log messages to a file someplace (and even
> jetty will log to a file if it's configured to do so -- most production
> instance are)
The start.jar was incredible helpful to dissect things.
Thanks Chris and everyone who helped me!
- Markus
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by Chris Hostetter <ho...@fucit.org>.
: The measurement was done outside our Solr client which sends the update
: and then the commit to the handler. I also see the update-URL call in
: the Tomcat Manager taking up that amount of time.
so it's the full request time, and would be inclusive of any postCommit
event handlers -- that's important to know. the logs will help clear up
wether the underlying "commit" is really taking up a large amount of time
or if it's some postCommit even (like spellcheck index building, or
snapshooting, etc...)
: > what do your Solr logs say about the commit, and the subsequent
: > newSearcher?
:
: How can I get such logs? I was asking for in the second mail do this
it depends on what servlet container you use to run Solr, and how it's
configured. In the simple the "java -jar start.jar" jetty setup used for
the Solr example, jetty dumps them to STDOUT in your console. But most
servlet containers will write log messages to a file someplace (and even
jetty will log to a file if it's configured to do so -- most production
instance are)
-Hoss
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by Markus Fischer <ma...@fischer.name>.
Hi,
On 05.05.2010 03:49, Chris Hostetter wrote:
>
> : Are you accidentally building the spellchecker database on each commit?
> ...
> : > This could also be caused by performing an optimize after the commit, or it
> : > could be caused by auto warming the caches, or a combination of both.
>
> The heart of the matter being: it's pretty much impossible to guess what
> is taking up all this time (and eating up all that CPU) w/o seeing your
> configs, and having a better idea of how you are timing this "1 to 2
> minutes" ... is this what your client sending the commit reports? what
> exactly is the command it's executing?
The measurement was done outside our Solr client which sends the update
and then the commit to the handler. I also see the update-URL call in
the Tomcat Manager taking up that amount of time.
> what do your Solr logs say about the commit, and the subsequent
> newSearcher?
How can I get such logs? I was asking for in the second mail do this
thread; basically it's the same problem I'm having, as I've no clue were
it costs that time.
I'll get the configs asap; and yes, we're using a spellchecker too. Is
this list fine with attachments? Or should I just paste the XML stuff
inline? Or use pastebin?
thanks for the help!
- Markus
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by Chris Hostetter <ho...@fucit.org>.
: Are you accidentally building the spellchecker database on each commit?
...
: > This could also be caused by performing an optimize after the commit, or it
: > could be caused by auto warming the caches, or a combination of both.
The heart of the matter being: it's pretty much impossible to guess what
is taking up all this time (and eating up all that CPU) w/o seeing your
configs, and having a better idea of how you are timing this "1 to 2
minutes" ... is this what your client sending the commit reports? what
exactly is the command it's executing?
what do your Solr logs say about the commit, and the subsequent
newSearcher?
-Hoss
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by Lance Norskog <go...@gmail.com>.
Are you accidentally building the spellchecker database on each commit?
An option is to use the MergePolicy stuff to avoid merging during
normal commits, but I failed to understand the interactions of
configuration numbers. It's a bit of a jungle in there.
On Tue, May 4, 2010 at 5:43 AM, <cb...@job.com> wrote:
> Hi,
>
> This could also be caused by performing an optimize after the commit, or it
> could be caused by auto warming the caches, or a combination of both.
>
> If you are using the Data Import Handler the default for a delta import is
> commit and optimize, which caused us a similar problem except we were
> optimizing a 7 million document, 23Gb index with every delta import which
> was taking over 10 minutes. As soon as we added optimize=false to the
> command updates took a few seconds. You can always add separate calls to
> perform the optimize when it's convenient for you.
>
> To see if the problem is auto warming take a look at the warm up time for
> the searcher. If this is the cause you will need to consider lowering the
> autowarmCount for your caches.
>
>
> Colin.
>
>> -----Original Message-----
>> From: Markus Fischer [mailto:markus@fischer.name]
>> Sent: Tuesday, May 04, 2010 6:22 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
>>
>> On 04.05.2010 11:01, Peter Sturge wrote:
>> > It might be worth checking the VMWare environment - if you're using
>> the
>> > VMWare scsi vmdk and it's shared across multiple VMs and there's a
>> lot of
>> > disk contention (i.e. multiple VMs are all busy reading/writing
>> to/from the
>> > same disk channel), this can really slow down I/O operations.
>>
>> Ok, thanks, I'll try to get the information from my hoster.
>>
>> I noticed that the commiting seems to be constant in time: it doesn't
>> matter whether I'm updating only one document or 50 (usually it won't
>> be
>> more). Maybe these numbers are so low anyway to cause any real impact
>> ...
>>
>> - Markus
>
>
>
>
>
--
Lance Norskog
goksron@gmail.com
RE: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by cb...@job.com.
Hi,
This could also be caused by performing an optimize after the commit, or it
could be caused by auto warming the caches, or a combination of both.
If you are using the Data Import Handler the default for a delta import is
commit and optimize, which caused us a similar problem except we were
optimizing a 7 million document, 23Gb index with every delta import which
was taking over 10 minutes. As soon as we added optimize=false to the
command updates took a few seconds. You can always add separate calls to
perform the optimize when it's convenient for you.
To see if the problem is auto warming take a look at the warm up time for
the searcher. If this is the cause you will need to consider lowering the
autowarmCount for your caches.
Colin.
> -----Original Message-----
> From: Markus Fischer [mailto:markus@fischer.name]
> Sent: Tuesday, May 04, 2010 6:22 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
>
> On 04.05.2010 11:01, Peter Sturge wrote:
> > It might be worth checking the VMWare environment - if you're using
> the
> > VMWare scsi vmdk and it's shared across multiple VMs and there's a
> lot of
> > disk contention (i.e. multiple VMs are all busy reading/writing
> to/from the
> > same disk channel), this can really slow down I/O operations.
>
> Ok, thanks, I'll try to get the information from my hoster.
>
> I noticed that the commiting seems to be constant in time: it doesn't
> matter whether I'm updating only one document or 50 (usually it won't
> be
> more). Maybe these numbers are so low anyway to cause any real impact
> ...
>
> - Markus
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by Markus Fischer <ma...@fischer.name>.
On 04.05.2010 11:01, Peter Sturge wrote:
> It might be worth checking the VMWare environment - if you're using the
> VMWare scsi vmdk and it's shared across multiple VMs and there's a lot of
> disk contention (i.e. multiple VMs are all busy reading/writing to/from the
> same disk channel), this can really slow down I/O operations.
Ok, thanks, I'll try to get the information from my hoster.
I noticed that the commiting seems to be constant in time: it doesn't
matter whether I'm updating only one document or 50 (usually it won't be
more). Maybe these numbers are so low anyway to cause any real impact ...
- Markus
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by Peter Sturge <pe...@googlemail.com>.
It might be worth checking the VMWare environment - if you're using the
VMWare scsi vmdk and it's shared across multiple VMs and there's a lot of
disk contention (i.e. multiple VMs are all busy reading/writing to/from the
same disk channel), this can really slow down I/O operations.
On Tue, May 4, 2010 at 8:52 AM, Markus Fischer <ma...@fischer.name> wrote:
> Hi,
>
>
> On 04.05.2010 03:24, Mark Miller wrote:
>
>> On 5/3/10 9:06 AM, Markus Fischer wrote:
>>
>>> we recently began having trouble with our Solr 1.4 instance. We've about
>>> 850k documents in the index which is about 1.2GB in size; the JVM which
>>> runs tomcat/solr (no other apps are deployed) has been given 2GB.
>>>
>>> We've a forum and run a process every minute which indexes the new
>>> messages. The number of messages updated are from 0 to 20 messages
>>> average. The commit takes about 1 or two minutes, but usually when it
>>> finished a few seconds later the next batch of documents is processed
>>> and the story starts again.
>>>
>>> Our environment is being providing by a company purely using VMWare
>>> infrastructure, the Solr index itself is on an NSF for which we get some
>>> 33MB/s throughput.
>>>
>>
>> That is certainly not a normal commit time for an index of that size.
>>
>> Note that Solr 1.4 can have issues when working on NFS, but I don't know
>> that it would have anything to do with this.
>>
>> Are you using the simple lock factory rather than the default native
>> lock factory? (as you should do when running on NFS)
>>
>
> I've switched the lockType to "simple" but didn't see any timing
> difference; it's still somewhat between one or two minutes.
>
> In my last test case I tested with the indexing having been updated with
> only a single document.
>
> I'm not very familiar with getting more debug information or similar out of
> Solr; is there a way to enable something to find out what's actually doing
> and what costs much time?
>
> thanks so far,
> - Markus
>
>
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by Markus Fischer <ma...@fischer.name>.
Hi,
On 04.05.2010 03:24, Mark Miller wrote:
> On 5/3/10 9:06 AM, Markus Fischer wrote:
>> we recently began having trouble with our Solr 1.4 instance. We've about
>> 850k documents in the index which is about 1.2GB in size; the JVM which
>> runs tomcat/solr (no other apps are deployed) has been given 2GB.
>>
>> We've a forum and run a process every minute which indexes the new
>> messages. The number of messages updated are from 0 to 20 messages
>> average. The commit takes about 1 or two minutes, but usually when it
>> finished a few seconds later the next batch of documents is processed
>> and the story starts again.
>>
>> Our environment is being providing by a company purely using VMWare
>> infrastructure, the Solr index itself is on an NSF for which we get some
>> 33MB/s throughput.
>
> That is certainly not a normal commit time for an index of that size.
>
> Note that Solr 1.4 can have issues when working on NFS, but I don't know
> that it would have anything to do with this.
>
> Are you using the simple lock factory rather than the default native
> lock factory? (as you should do when running on NFS)
I've switched the lockType to "simple" but didn't see any timing
difference; it's still somewhat between one or two minutes.
In my last test case I tested with the indexing having been updated with
only a single document.
I'm not very familiar with getting more debug information or similar out
of Solr; is there a way to enable something to find out what's actually
doing and what costs much time?
thanks so far,
- Markus
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
Posted by Mark Miller <ma...@gmail.com>.
On 5/3/10 9:06 AM, Markus Fischer wrote:
> Hi,
>
> we recently began having trouble with our Solr 1.4 instance. We've about
> 850k documents in the index which is about 1.2GB in size; the JVM which
> runs tomcat/solr (no other apps are deployed) has been given 2GB.
>
> We've a forum and run a process every minute which indexes the new
> messages. The number of messages updated are from 0 to 20 messages
> average. The commit takes about 1 or two minutes, but usually when it
> finished a few seconds later the next batch of documents is processed
> and the story starts again.
>
> So actually it's like Solr is running commits all day long and CPU usage
> ranges from 80% to 120%.
>
> This continuous CPU usage caused ill effects on other services running
> on the same machine.
>
> Our environment is being providing by a company purely using VMWare
> infrastructure, the Solr index itself is on an NSF for which we get some
> 33MB/s throughput.
>
> So, an easy solution would be to just put more resources into it, e.g. a
> separate machine. But before I make the decision I'd like to find out
> whether the app behaves properly under this circumstances or if its
> possible to shorten the commit time down to a few seconds so the CPU is
> not drained that long.
>
> thanks for any pointers,
>
> - Markus
>
That is certainly not a normal commit time for an index of that size.
Note that Solr 1.4 can have issues when working on NFS, but I don't know
that it would have anything to do with this.
Are you using the simple lock factory rather than the default native
lock factory? (as you should do when running on NFS)
--
- Mark
http://www.lucidimagination.com