You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by kiwi clive <ki...@yahoo.com> on 2012/11/07 11:58:33 UTC
Lucene 3.6.0 high CPU usage
Hi Guys,
Having upgraded from lucene2.3.2 to lucene3.6.0 and jumping through a few hoops, we note that the CPU usage on the new version is perhaps 20-30% higher. The i/o wait appears about the same on both versions.
Can anyone suggest why this should be so or is the newer lucene version just more CPU hungry? If not, any tuning suggestions to reduce CPU usage would be gratefully received !
Thanks
Clive
Re: Lucene 3.6.0 high CPU usage
Posted by kiwi clive <ki...@yahoo.com>.
Hi Ian,
Yes I/O ->CPU, but read on...
The throughput is about the same with similar i/o and 20% higher CPU. We have plenty of CPU and I would rather be cpu bound, thean i/o bound.
However, we have increased the number of indexing threads in our application. On Lucene 2.3.2, this gave no performance improvement as the I/O just increased. However, lucene 3.6.0 ramps up the CPU (user) usage while I/O only increases marginally.
Our throughput has trebled !! - amazing improvement, and all in CPU space.
So it appears there have been a lot of performance improvements made in the new version that I was not previously using, the Ferrari was in first gear and now we are taking advantage of these features.
This is truly phenominal step-change in throughput, thank you Lucene developers !
Clive
________________________________
From: Ian Lea <ia...@gmail.com>
To: java-user@lucene.apache.org; kiwi clive <ki...@yahoo.com>
Sent: Friday, November 9, 2012 10:04 AM
Subject: Re: Lucene 3.6.0 high CPU usage
Are you getting the same, improved or worse performance/throughput?
Has the bottleneck switched from IO to CPU?
--
Ian.
On Thu, Nov 8, 2012 at 12:40 PM, kiwi clive <ki...@yahoo.com> wrote:
> Having played with merge parameters and various index parameters, it seems possible to change the I/O usage at the cost of the number of index files. However, it does appear that this version of lucene is using more CPU. Is there any reason for this ? is it normal ? We can push a large amount of documents through the indexer and it seams to work admirably although I would be happier if the load average came down.
>
> Any lucene devs out there who could shed some light on this behaviour ?
>
>
> Thanks,
> Clive
>
>
> ________________________________
> From: kiwi clive <ki...@yahoo.com>
> To: "java-user@lucene.apache.org" <ja...@lucene.apache.org>
> Sent: Wednesday, November 7, 2012 10:58 AM
> Subject: Lucene 3.6.0 high CPU usage
>
> Hi Guys,
>
> Having upgraded from lucene2.3.2 to lucene3.6.0 and jumping through a few hoops, we note that the CPU usage on the new version is perhaps 20-30% higher. The i/o wait appears about the same on both versions.
>
> Can anyone suggest why this should be so or is the newer lucene version just more CPU hungry? If not, any tuning suggestions to reduce CPU usage would be gratefully received !
>
> Thanks
> Clive
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 3.6.0 high CPU usage
Posted by Ian Lea <ia...@gmail.com>.
Are you getting the same, improved or worse performance/throughput?
Has the bottleneck switched from IO to CPU?
--
Ian.
On Thu, Nov 8, 2012 at 12:40 PM, kiwi clive <ki...@yahoo.com> wrote:
> Having played with merge parameters and various index parameters, it seems possible to change the I/O usage at the cost of the number of index files. However, it does appear that this version of lucene is using more CPU. Is there any reason for this ? is it normal ? We can push a large amount of documents through the indexer and it seams to work admirably although I would be happier if the load average came down.
>
> Any lucene devs out there who could shed some light on this behaviour ?
>
>
> Thanks,
> Clive
>
>
> ________________________________
> From: kiwi clive <ki...@yahoo.com>
> To: "java-user@lucene.apache.org" <ja...@lucene.apache.org>
> Sent: Wednesday, November 7, 2012 10:58 AM
> Subject: Lucene 3.6.0 high CPU usage
>
> Hi Guys,
>
> Having upgraded from lucene2.3.2 to lucene3.6.0 and jumping through a few hoops, we note that the CPU usage on the new version is perhaps 20-30% higher. The i/o wait appears about the same on both versions.
>
> Can anyone suggest why this should be so or is the newer lucene version just more CPU hungry? If not, any tuning suggestions to reduce CPU usage would be gratefully received !
>
> Thanks
> Clive
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 3.6.0 high CPU usage
Posted by kiwi clive <ki...@yahoo.com>.
Having played with merge parameters and various index parameters, it seems possible to change the I/O usage at the cost of the number of index files. However, it does appear that this version of lucene is using more CPU. Is there any reason for this ? is it normal ? We can push a large amount of documents through the indexer and it seams to work admirably although I would be happier if the load average came down.
Any lucene devs out there who could shed some light on this behaviour ?
Thanks,
Clive
________________________________
From: kiwi clive <ki...@yahoo.com>
To: "java-user@lucene.apache.org" <ja...@lucene.apache.org>
Sent: Wednesday, November 7, 2012 10:58 AM
Subject: Lucene 3.6.0 high CPU usage
Hi Guys,
Having upgraded from lucene2.3.2 to lucene3.6.0 and jumping through a few hoops, we note that the CPU usage on the new version is perhaps 20-30% higher. The i/o wait appears about the same on both versions.
Can anyone suggest why this should be so or is the newer lucene version just more CPU hungry? If not, any tuning suggestions to reduce CPU usage would be gratefully received !
Thanks
Clive