You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by YouPeng Yang <yy...@gmail.com> on 2014/07/16 11:03:36 UTC
weird drastic query latency during performance testing and DIH import
delay after performance testing
Hi
I build my SolrCloud using Solr 4.6.0 (java version:1.7.0_45). In my
cloud,I have a collection with 30 shard,and each shard has one replica.
each core of the shard contains nearly 50 million docs that is 15GB in
size,so does the replica.
Before applying my cloud in the real world,I do a performance test with
JMeter 2.11.
The scenario of the my test is simple:100 threads sending requests for 20
seconds ,and these requests are only sent to a specific core of a specific
shard.the request is similar to the following :
http://IP:port/solr/tv_201407/select?q=*:*&fq=BEGINTIME:["2014-06-01
00:00:00"+TO+*]+AND+(CONTACT:${user})+AND (TV_STATE:"00")&shards=tv_201407
&rows=2000&sort=BEGINTIME+desc.
I encountered the drastic query latency during performance testing and
DIH import delay after performance testing.Please help me. I have tested
several times and get the same problem and can not handle it by myself.Any
suggestion will be apprecaited.
The following steps describes what I have done .
Step 1: Before the test,the DIH import job is very fast.As the statistics
[1], the DIH importing takes only 1s for 10 docs.
[1]-----------------------------------
Indexing completed. Added/Updated: 10 documents. Deleted 0 documents.
(Duration: 01s)
Requests: 1 (1/s), Fetched: 10 (10/s), Skipped: 0, Processed: 10 (10/s)
Started: less than a minute ago
---------------------------------------
Step 2: Then ,Doing the test under the caches are cleaned. The summery
statistics data is as [2]. Although I have clean the caches,I never think
the query latency becomes so drastic that it cannot be acceptable in my
real application.
The red font describes the latency of the query performance test on the
core tv_201407 of the shard tv_201407 .
So would you experts can give some hints about the drastic query latency
?
[2]-----------------------------------
[solr@solr2 test]$ ../bin/jmeter.sh -n -t solrCoudKala20140401.jmx -l
logfile_solrCloud_20.jtl
Creating summariser <aggregate>
Created the tree successfully using solrCoudKala20140401.jmx
Starting the test @ Wed Jul 16 15:59:28 CST 2014 (1405497568104)
Waiting for possible shutdown message on port 4445
aggregate + 1 in 8.1s = 0.1/s Avg: 8070 Min: 8070 Max: 8070 Err:
0 (0.00%) Active: 100 Started: 100 Finished: 0
aggregate + 103 in 13.4s = 7.7/s Avg: 8027 Min: 4191 Max: 8434 Err:
0 (0.00%) Active: 97 Started: 100 Finished: 3
aggregate = 104 in 13.4s = 7.7/s Avg: 8027 Min: 4191 Max: 8434 Err:
0 (0.00%)
aggregate + 96 in 7s = 14.5/s Avg: 6160 Min: 5295 Max: 6625 Err:
0 (0.00%) Active: 0 Started: 100 Finished: 100
aggregate = 200 in 15s = 13.6/s Avg: 7131 Min: 4191 Max: 8434 Err:
0 (0.00%)
Tidying up ... @ Wed Jul 16 15:59:43 CST 2014 (1405497583461)
... end of run
[solr@solr2 test]$
---------------------------------------
Step 3:To be continued,after the test,I do the DIH importing job again
using the same import expresion.However the performance of the DIH becomes
so unacceptable.
to import the 10 docs takes 2 m 15 s [3]!
Having noticing that ,solr can fetched the 10 docs fast,the processing is
slow.
[3]-----------------------------------
*Indexing completed. Added/Updated: 10 documents. Deleted 0 documents.
(Duration: 2m 15s)*
Requests: 1 (0/s), Fetched: 10 (0/s), Skipped: 0, Processed: 10 (0/s)
Started: about an hour ago
---------------------------------------
By the way. jvm gc goes normal,and there is no long full gc during the
test. the load of my system(rhel 6.5) are also normal.
Regards
Re: weird drastic query latency during performance testing and DIH
import delay after performance testing
Posted by Erick Erickson <er...@gmail.com>.
This is very strange. I have no idea why DIH is
taking so long. What happens if you execute the DIH
query in some SQL front end? It's possible that DIH
is just taking forever to execute the SQL query because
of how it's written.
I'm having trouble following the query results, but again
this is super-slow. How many docs do you have in
your index? How much memory have you allocated
for your JVM? Your query is actually pretty simple, so I have
no clue whatsoever why your response times are
so slow. But this is such bad performance that something
major is wrong.
I'd approach it one problem at a time. Understand what's
happening with your queries, _then_ tackle DIH (or the other
way around)...
Best,
Erick
On Wed, Jul 16, 2014 at 2:03 AM, YouPeng Yang <yy...@gmail.com>
wrote:
> Hi
> I build my SolrCloud using Solr 4.6.0 (java version:1.7.0_45). In my
> cloud,I have a collection with 30 shard,and each shard has one replica.
> each core of the shard contains nearly 50 million docs that is 15GB in
> size,so does the replica.
> Before applying my cloud in the real world,I do a performance test with
> JMeter 2.11.
> The scenario of the my test is simple:100 threads sending requests for 20
> seconds ,and these requests are only sent to a specific core of a specific
> shard.the request is similar to the following :
> http://IP:port/solr/tv_201407/select?q=*:*&fq=BEGINTIME:["2014-06-01
> 00:00:00"+TO+*]+AND+(CONTACT:${user})+AND (TV_STATE:"00")&shards=tv_201407
> &rows=2000&sort=BEGINTIME+desc.
>
> I encountered the drastic query latency during performance testing and
> DIH import delay after performance testing.Please help me. I have tested
> several times and get the same problem and can not handle it by myself.Any
> suggestion will be apprecaited.
>
> The following steps describes what I have done .
>
> Step 1: Before the test,the DIH import job is very fast.As the statistics
> [1], the DIH importing takes only 1s for 10 docs.
> [1]-----------------------------------
> Indexing completed. Added/Updated: 10 documents. Deleted 0 documents.
> (Duration: 01s)
> Requests: 1 (1/s), Fetched: 10 (10/s), Skipped: 0, Processed: 10 (10/s)
> Started: less than a minute ago
> ---------------------------------------
>
> Step 2: Then ,Doing the test under the caches are cleaned. The summery
> statistics data is as [2]. Although I have clean the caches,I never think
> the query latency becomes so drastic that it cannot be acceptable in my
> real application.
> The red font describes the latency of the query performance test on the
> core tv_201407 of the shard tv_201407 .
>
> So would you experts can give some hints about the drastic query latency
> ?
>
> [2]-----------------------------------
> [solr@solr2 test]$ ../bin/jmeter.sh -n -t solrCoudKala20140401.jmx -l
> logfile_solrCloud_20.jtl
> Creating summariser <aggregate>
> Created the tree successfully using solrCoudKala20140401.jmx
> Starting the test @ Wed Jul 16 15:59:28 CST 2014 (1405497568104)
> Waiting for possible shutdown message on port 4445
> aggregate + 1 in 8.1s = 0.1/s Avg: 8070 Min: 8070 Max: 8070
> Err:
> 0 (0.00%) Active: 100 Started: 100 Finished: 0
> aggregate + 103 in 13.4s = 7.7/s Avg: 8027 Min: 4191 Max: 8434
> Err:
> 0 (0.00%) Active: 97 Started: 100 Finished: 3
> aggregate = 104 in 13.4s = 7.7/s Avg: 8027 Min: 4191 Max: 8434
> Err:
> 0 (0.00%)
> aggregate + 96 in 7s = 14.5/s Avg: 6160 Min: 5295 Max: 6625
> Err:
> 0 (0.00%) Active: 0 Started: 100 Finished: 100
> aggregate = 200 in 15s = 13.6/s Avg: 7131 Min: 4191 Max: 8434
> Err:
> 0 (0.00%)
> Tidying up ... @ Wed Jul 16 15:59:43 CST 2014 (1405497583461)
> ... end of run
> [solr@solr2 test]$
> ---------------------------------------
> Step 3:To be continued,after the test,I do the DIH importing job again
> using the same import expresion.However the performance of the DIH becomes
> so unacceptable.
> to import the 10 docs takes 2 m 15 s [3]!
> Having noticing that ,solr can fetched the 10 docs fast,the processing is
> slow.
>
> [3]-----------------------------------
> *Indexing completed. Added/Updated: 10 documents. Deleted 0 documents.
> (Duration: 2m 15s)*
> Requests: 1 (0/s), Fetched: 10 (0/s), Skipped: 0, Processed: 10 (0/s)
> Started: about an hour ago
> ---------------------------------------
>
> By the way. jvm gc goes normal,and there is no long full gc during the
> test. the load of my system(rhel 6.5) are also normal.
>
> Regards
>