You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Aviran <am...@infosciences.com> on 2004/07/12 16:27:04 UTC

Lucene Search has poor cpu utilization on a 4-CPU machine

Hi all,
First let me explain what I found out. I'm running Lucene on a 4 CPU server.
While doing some stress tests I've noticed (by doing full thread dump) that
searching threads are blocked on the method: public FieldInfo fieldInfo(int
fieldNumber) This causes for a significant cpu idle time. 
I noticed that the class org.apache.lucene.index.FieldInfos uses private
class members Vector byNumber and Hashtable byName, both of which are
synchronized objects. By changing the Vector byNumber to ArrayList byNumber
I was able to get 110% improvement in performance (number of searches per
second).
 
My question is: do the fields byNumber and byName have to be synchronized
and what can happen if I'll change them to be ArrayList and HashMap which
are not synchronized ? Can this corrupt the index or the integrity of the
results?

Thanks,
Aviran



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Lucene Search has poor cpu utilization on a 4-CPU machine

Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
Doug Cutting wrote:

> I noticed that the class org.apache.lucene.index.FieldInfos uses private
>
>> class members Vector byNumber and Hashtable byName, both of which are
>> synchronized objects. By changing the Vector byNumber to ArrayList 
>> byNumber
>> I was able to get 110% improvement in performance (number of searches 
>> per
>> second).
>
>
> That's impressive! Good job finding a bottleneck!

Wow... thats awesome.

We have all dual XEONs with Hyperthreading and kernel 2.6 so I imagine 
in this situation we'd see an improvement too.

I wonder if we could break this out into a patch for legacy Lucene 
users. I'd like to see the stacktrace too.

We're using a lot of synchronized code (Hashtable, Vector, etc) so I'm 
willing to bet this is happening in other places.

>> My question is: do the fields byNumber and byName have to be 
>> synchronized
>> and what can happen if I'll change them to be ArrayList and HashMap 
>> which
>> are not synchronized ? Can this corrupt the index or the integrity of 
>> the
>> results?
>
>
> I think that is a safe change. FieldInfos is only modifed by 
> DocumentWriter and SegmentMerger, and there is no possibility of other 
> threads accessing those instances. Please submit a patch to the 
> developer mailing list.
>
That would be great!

Kevin

-- 

Please reply using PGP.

    http://peerfear.org/pubkey.asc    
    
    NewsMonster - http://www.newsmonster.org/
    
Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965
       AIM/YIM - sfburtonator,  Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
  IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Lucene Search has poor cpu utilization on a 4-CPU machine

Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
Doug Cutting wrote:

> Aviran wrote:
>
>> I changed the Lucene 1.4 final source code and yes this is the source
>> version I changed.
>
>
> Note that this patch won't produce the a speedup on earlier releases, 
> since their was another multi-thread bottleneck higher up the stack 
> that was only recently removed, revealing this lower-level bottleneck.
>
> The other patch was:
>
> http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg07873.html
>
> Both are required to see the speedup.
>
Thanks...

> Also, is there any reason folks cannot use 1.4 final now?

No... just that I'm trying to be conservative... I'm probably going to 
look at just migrating to 1.4 ASAP but we're close to a milestone...

Kevin

-- 

Please reply using PGP.

    http://peerfear.org/pubkey.asc    
    
    NewsMonster - http://www.newsmonster.org/
    
Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965
       AIM/YIM - sfburtonator,  Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
  IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Lucene Search has poor cpu utilization on a 4-CPU machine

Posted by Doug Cutting <cu...@apache.org>.
Aviran wrote:
> I changed the Lucene 1.4 final source code and yes this is the source
> version I changed.

Note that this patch won't produce the a speedup on earlier releases, 
since their was another multi-thread bottleneck higher up the stack that 
was only recently removed, revealing this lower-level bottleneck.

The other patch was:

http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg07873.html

Both are required to see the speedup.

Also, is there any reason folks cannot use 1.4 final now?

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

RE: Lucene Search has poor cpu utilization on a 4-CPU machine

Posted by Aviran <am...@infosciences.com>.
I changed the Lucene 1.4 final source code and yes this is the source
version I changed.

Aviran

-----Original Message-----
From: Kevin A. Burton [mailto:burton@newsmonster.org] 
Sent: Monday, July 12, 2004 9:42 PM
To: Lucene Users List
Subject: Re: Lucene Search has poor cpu utilization on a 4-CPU machine


Aviran wrote:

>Bug 30058 posted
>
>  
>
Which of course is here:

http://issues.apache.org/bugzilla/show_bug.cgi?id=30058

Is this the source of the revision you modified?

http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg06116.html

Also what version of Lucene?

Kevin

-- 

Please reply using PGP.

    http://peerfear.org/pubkey.asc    
    
    NewsMonster - http://www.newsmonster.org/
    
Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965
       AIM/YIM - sfburtonator,  Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
  IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Lucene Search has poor cpu utilization on a 4-CPU machine

Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
Aviran wrote:

>Bug 30058 posted
>
>  
>
Which of course is here:

http://issues.apache.org/bugzilla/show_bug.cgi?id=30058

Is this the source of the revision you modified?

http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg06116.html

Also what version of Lucene?

Kevin

-- 

Please reply using PGP.

    http://peerfear.org/pubkey.asc    
    
    NewsMonster - http://www.newsmonster.org/
    
Kevin A. Burton, Location - San Francisco, CA, Cell - 415.595.9965
       AIM/YIM - sfburtonator,  Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
  IRC - freenode.net #infoanarchy | #p2p-hackers | #newsmonster


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

RE: Lucene Search has poor cpu utilization on a 4-CPU machine

Posted by Aviran <am...@infosciences.com>.
Bug 30058 posted

Aviran

-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org] 
Sent: Monday, July 12, 2004 1:38 PM
To: Lucene Users List
Subject: Re: Lucene Search has poor cpu utilization on a 4-CPU machine


Aviran wrote:
> I use Lucene 1.4 final
> 
> Here is the thread dump for one blocked thread (If you want a full 
> thread dump for all threads I can do that too)

Thanks.  I think I get the point.  I recently removed a synchronization 
point higher in the stack, so that now this one shows up!

Whether or not you submit a patch, please file a bug report in Bugzilla 
with your proposed change, so that we don't lose track of this issue.

Thanks,

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Lucene Search has poor cpu utilization on a 4-CPU machine

Posted by Doug Cutting <cu...@apache.org>.
Aviran wrote:
> I use Lucene 1.4 final
> 
> Here is the thread dump for one blocked thread (If you want a full thread
> dump for all threads I can do that too)

Thanks.  I think I get the point.  I recently removed a synchronization 
point higher in the stack, so that now this one shows up!

Whether or not you submit a patch, please file a bug report in Bugzilla 
with your proposed change, so that we don't lose track of this issue.

Thanks,

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

RE: Lucene Search has poor cpu utilization on a 4-CPU machine

Posted by Aviran <am...@infosciences.com>.
I use Lucene 1.4 final

Here is the thread dump for one blocked thread (If you want a full thread
dump for all threads I can do that too)

"Thread-32" daemon prio=1 tid=0x082334c0 nid=0xa66 waiting for monitor entry
[4f385000..4f38687c]
        at java.util.Vector.elementAt(Vector.java:430)
        - waiting to lock <0x452b93a8> (a java.util.Vector)
        at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:155)
        at org.apache.lucene.index.FieldInfos.fieldName(FieldInfos.java:151)
        at
org.apache.lucene.index.SegmentTermEnum.readTerm(SegmentTermEnum.java:149)
        at
org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:115)
        at
org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:143)
        at
org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:137)
        at
org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:51)
        at
org.apache.lucene.index.IndexReader.termDocs(IndexReader.java:364)
        at
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:59)
        at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java
:165)
        at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java
:165)
        at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:154)
        at
gov.gsa.search.SearcherByPageAndSortedField.search(SearcherByPageAndSortedFi
eld.java:317)
        at
gov.gsa.search.SearcherByPageAndSortedField.search(SearcherByPageAndSortedFi
eld.java:203)
        at
gov.gsa.search.grants.SearchGrants.searchByPageAndSortedField(SearchGrants.j
ava:308)
        at
gov.gsa.search.grants.SearchServlet.searchByIndex(SearchServlet.java:1541)
        at
gov.gsa.search.grants.SearchServlet.getResults(SearchServlet.java:1325)
        at gov.gsa.search.grants.SearchServlet.doGet(SearchServlet.java:500)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:247)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:193)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:256)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
        at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
        at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
        at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180
)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
        at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.
java:171)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172
)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
        at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:174)
        at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
        at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
        at
org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
        at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:261)
        at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:360)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:604)
        at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:562)
        at
org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:679)
        at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:619)
        at java.lang.Thread.run(Thread.java:534)


And how do I submit a patch to the developer mailing list? Just attach a
changed java file to an email?

Aviran

-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org] 
Sent: Monday, July 12, 2004 12:42 PM
To: Lucene Users List
Subject: Re: Lucene Search has poor cpu utilization on a 4-CPU machine


Aviran wrote:
> First let me explain what I found out. I'm running Lucene on a 4 CPU 
> server. While doing some stress tests I've noticed (by doing full 
> thread dump) that searching threads are blocked on the method: public 
> FieldInfo fieldInfo(int
> fieldNumber) This causes for a significant cpu idle time. 

What version of Lucene are you running?  Also, can you please send the 
stack traces of the blocked threads, or at least a description of them? 
  I'd be interested to see what context this happens in.  In particular, 
which IndexReader and Searcher/Scorer/Weight methods does it happen under?

> I noticed that the class org.apache.lucene.index.FieldInfos uses 
> private class members Vector byNumber and Hashtable byName, both of 
> which are synchronized objects. By changing the Vector byNumber to 
> ArrayList byNumber I was able to get 110% improvement in performance 
> (number of searches per second).

That's impressive!  Good job finding a bottleneck!

> My question is: do the fields byNumber and byName have to be 
> synchronized and what can happen if I'll change them to be ArrayList 
> and HashMap which are not synchronized ? Can this corrupt the index or 
> the integrity of the results?

I think that is a safe change.  FieldInfos is only modifed by 
DocumentWriter and SegmentMerger, and there is no possibility of other 
threads accessing those instances.  Please submit a patch to the 
developer mailing list.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Lucene Search has poor cpu utilization on a 4-CPU machine

Posted by Doug Cutting <cu...@apache.org>.
Aviran wrote:
> First let me explain what I found out. I'm running Lucene on a 4 CPU server.
> While doing some stress tests I've noticed (by doing full thread dump) that
> searching threads are blocked on the method: public FieldInfo fieldInfo(int
> fieldNumber) This causes for a significant cpu idle time. 

What version of Lucene are you running?  Also, can you please send the 
stack traces of the blocked threads, or at least a description of them? 
  I'd be interested to see what context this happens in.  In particular, 
which IndexReader and Searcher/Scorer/Weight methods does it happen under?

> I noticed that the class org.apache.lucene.index.FieldInfos uses private
> class members Vector byNumber and Hashtable byName, both of which are
> synchronized objects. By changing the Vector byNumber to ArrayList byNumber
> I was able to get 110% improvement in performance (number of searches per
> second).

That's impressive!  Good job finding a bottleneck!

> My question is: do the fields byNumber and byName have to be synchronized
> and what can happen if I'll change them to be ArrayList and HashMap which
> are not synchronized ? Can this corrupt the index or the integrity of the
> results?

I think that is a safe change.  FieldInfos is only modifed by 
DocumentWriter and SegmentMerger, and there is no possibility of other 
threads accessing those instances.  Please submit a patch to the 
developer mailing list.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org