You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Harry Yu <50...@qq.com> on 2014/09/15 09:10:37 UTC
Speed up searching in multiple-thread?
Dear members at Lucene project,
I'm developing poi search application using lucene 4.8 . Recently, I met a trouble that the performance of IndexSearcher.search is bad in multiple-thread environment. According the test results, I found that if thread number is 1, the response time of searching is only 17ms. But that would be 400ms when I increased threads to 30.
In our application, the size of index is 882M, cpu is 2.4GHZ i5, and test machine is mac pro. I read How to Make Searching Faster in Lucene-java wiki. And there are only singleton instance of IndexWriter, IndexReader and IndexSearcher.
Please give me some tips or solutions to speeding up searching in multiple-thread. Thanks.
Best regards,
Harry Yu
回复： Speed up searching in multiple-thread?
Posted by Harry Yu <50...@qq.com>.
Hi Michael,
Thanks for your reply. Yes, core i5 is 2 real (4 with hyperthreading) cores. And
is this the core reason? Are my test methods wrong?
In my test environment, I used JMeter to simulate 30 threads, and access my search
service. I recorded per-search time and analyzed them.
Best Regards,
Harry Yu
------------------ 原始邮件 ------------------
发件人: "Michael McCandless";<lu...@mikemccandless.com>;
发送时间: 2014年9月15日(星期一) 下午3:48
收件人: "Lucene Users"<ja...@lucene.apache.org>;
主题: Re: Speed up searching in multiple-thread?
If you run 30 search threads on a core i5 it's expected there will be
big slowdowns in the per-query latency since core i5 only has 2 real
(4 with hyperthreading) cores?
Mike McCandless
http://blog.mikemccandless.com
On Mon, Sep 15, 2014 at 3:10 AM, Harry Yu <50...@qq.com> wrote:
> Dear members at Lucene project,
>
> I'm developing poi search application using lucene 4.8 . Recently, I met a trouble that the performance of IndexSearcher.search is bad in multiple-thread environment. According the test results, I found that if thread number is 1, the response time of searching is only 17ms. But that would be 400ms when I increased threads to 30.
>
>
> In our application, the size of index is 882M, cpu is 2.4GHZ i5, and test machine is mac pro. I read How to Make Searching Faster in Lucene-java wiki. And there are only singleton instance of IndexWriter, IndexReader and IndexSearcher.
>
>
> Please give me some tips or solutions to speeding up searching in multiple-thread. Thanks.
>
>
> Best regards,
>
>
> Harry Yu
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Speed up searching in multiple-thread?
Posted by Michael McCandless <lu...@mikemccandless.com>.
If you run 30 search threads on a core i5 it's expected there will be
big slowdowns in the per-query latency since core i5 only has 2 real
(4 with hyperthreading) cores?
Mike McCandless
http://blog.mikemccandless.com
On Mon, Sep 15, 2014 at 3:10 AM, Harry Yu <50...@qq.com> wrote:
> Dear members at Lucene project,
>
> I'm developing poi search application using lucene 4.8 . Recently, I met a trouble that the performance of IndexSearcher.search is bad in multiple-thread environment. According the test results, I found that if thread number is 1, the response time of searching is only 17ms. But that would be 400ms when I increased threads to 30.
>
>
> In our application, the size of index is 882M, cpu is 2.4GHZ i5, and test machine is mac pro. I read How to Make Searching Faster in Lucene-java wiki. And there are only singleton instance of IndexWriter, IndexReader and IndexSearcher.
>
>
> Please give me some tips or solutions to speeding up searching in multiple-thread. Thanks.
>
>
> Best regards,
>
>
> Harry Yu
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 回复： Speed up searching in multiple-thread?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2014-09-15 at 11:41 +0200, Harry Yu wrote:
> 17ms / searches is the whole process of search service, include
> accessing complete data form db, calling REST service etc.
Try looking at QTime in solr.log and compare it with your measured
response times, to see if it is Solr or your other services that are the
bottleneck.
- Toke Eskildsen, State and University Library, Denmark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
回复： Speed up searching in multiple-thread?
Posted by Harry Yu <50...@qq.com>.
Hi Toke,
Thanks for your reply. I expect the outcome of my application would be 150~200 searches / per.
Now I'm looking for methods to optimize my application, and want to improve the performance of the throughout and response time.
17ms / searches is the whole process of search service, include accessing complete data form db, calling REST service etc.
Regards,
Harry Yu
------------------ 原始邮件 ------------------
发件人: "Toke Eskildsen";<te...@statsbiblioteket.dk>;
发送时间: 2014年9月15日(星期一) 下午4:47
收件人: "java-user@lucene.apache.org"<ja...@lucene.apache.org>;
主题: Re: Speed up searching in multiple-thread?
On Mon, 2014-09-15 at 09:10 +0200, Harry Yu wrote:
> I'm developing poi search application using lucene 4.8 . Recently, I
> met a trouble that the performance of IndexSearcher.search is bad in
> multiple-thread environment. According the test results, I found that
> if thread number is 1, the response time of searching is only 17ms.
> But that would be 400ms when I increased threads to 30.
With a single thread you have 1000ms/s / 17ms ~= 59 searches/second.
For 30 threads you have 1000ms/s / 400ms * 30 = 75 searches/second
As you have a dual-core machine, the ideal outcome would be 2*59 = 118
searches/second. Accounting for fluctuations in measuring, more GC and a
bit of congestion, it does not seem unreasonable with 75.
(17ms/search seems like quite a long time for such a small index, but
that is independent of the threading issue)
What was the expected outcome of your test?
- Toke Eskildsen, State and University Library, Denmark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Speed up searching in multiple-thread?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2014-09-15 at 09:10 +0200, Harry Yu wrote:
> I'm developing poi search application using lucene 4.8 . Recently, I
> met a trouble that the performance of IndexSearcher.search is bad in
> multiple-thread environment. According the test results, I found that
> if thread number is 1, the response time of searching is only 17ms.
> But that would be 400ms when I increased threads to 30.
With a single thread you have 1000ms/s / 17ms ~= 59 searches/second.
For 30 threads you have 1000ms/s / 400ms * 30 = 75 searches/second
As you have a dual-core machine, the ideal outcome would be 2*59 = 118
searches/second. Accounting for fluctuations in measuring, more GC and a
bit of congestion, it does not seem unreasonable with 75.
(17ms/search seems like quite a long time for such a small index, but
that is independent of the threading issue)
What was the expected outcome of your test?
- Toke Eskildsen, State and University Library, Denmark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org