You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by ku3ia <de...@gmail.com> on 2011/12/15 23:00:49 UTC

Poor performance on distributed search

Hi, all!

I have a problem with distributed search. I downloaded one shard from my
production. It has:
* ~29M docs
* 11 fields
* ~105M terms
* size of shard is: 13GB
On production there are near 30 the same shards. I split this shard to 4
more smaller shards, so now I have:
small shard1:
docs: 6.2M
terms: 27.2M
size: 2.89GB
small shard2:
docs: 6.3M
terms: 28.7M
size: 2.98GB
small shard3:
docs: 7.9M
terms: 32.8M
size: 3.60GB
small shard4:
docs: 8.2M
terms: 32.6M
size: 3.70GB

My machine confguration:
ABIT AX-78
AMD Athlon 64 X2 5200+
DDR2 Kingston 2x2G+2x1G = 6G
WDC WD2500JS (System here)
WDC WD20EARS (6 partitions = 30 GB for shards at begin of drive, and other
empty, all partitions are well aligned)
GNU/Linux Debian Squeeze
Tomcat 6.0.32 with JAVA_OPTS:
JAVA_OPTS="$JAVA_OPTS -XX:+DisableExplicitGC -server \
    -XX:PermSize=512M -XX:MaxPermSize=512M -Xmx4096M -Xms4096M
-XX:NewSize=128M -XX:MaxNewSize=128M \
    -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled \
    -XX:CMSInitiatingOccupancyFraction=50 -XX:GCTimeRatio=9
-XX:MinHeapFreeRatio=25 -XX:MaxHeapFreeRatio=25 \
    -verbose:gc -XX:+PrintGCTimeStamps -Xloggc:$CATALINA_HOME/logs/gc.log"
Solr 3.5

I configured 4 cores and start Tomcat. I write a bash script. It's runing
during 300 seconds and sending every 6 seconds queries like
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(assistants)&rows=2000&start=0&fl=*,score&qt=requestShards
where qt=requestShards is my 4 shards. After test I have the results:

Elapsed time: 299 secs
--- solr ---
Queries processed: 21 << this is full response file
Queries cancelled: 29 << this is number of killed curls
Average QTime is: 59645.6 ms 
Average RTime is: 59.7619 sec(s) << this is average time difference between
start and stop the curl. There is a part of script:
# >>dcs=`date +%s`
# >>curl ${url} -s -H 'Content-type:text/xml; charset=utf-8' >
${F_DATADIR}/$dest.fdata
# >>dce=`date +%s`
# >>dcd=$(echo "$dce - $dcs" | bc)
Size of data-dir is: 3346766 bytes << this is response dir size

I'm using nmon to to monitor R/W disk speed, and I was surprised that read
speed of my shards volumes WDC20EAR's drive was nearly 3 MB/s when script is
working. After this I run benchmark test from disk utility. Here is results:
Minimum read rate: 53.2MB/s
Maximum Read rate: 126.4 MB/s
Average Read rate: 95.8 MB/s

But from the other side I tested queries like
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(assistants)&rows=2000&start=0&fl=*,score
results is:
Elapsed time: 299 secs
--- solr ---
Queries processed: 50
Queries cancelled: 0
Average QTime is: 139.76 ms
Average RTime is: 2.2 sec(s)
Size of data-dir is: 6819259 bytes

and quesries like
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(assistants)&rows=2000&start=0&fl=*,score&shards=127.0.0.1:8080/solr/shard1
and result is:
Elapsed time: 299 secs
--- solr ---
Queries processed: 49
Queries cancelled: 1
Average QTime is: 1878.37 ms
Average RTime is: 1.95918 sec(s)
Size of data-dir is: 4274099 bytes
So we see the results are the same.

My big question is: why is so slow drive read speed when Solr is working? 
Thanks for any replies

P.S. And maybe my general problem is too much terms in shard, for example,
query 
http://127.0.0.1:8080/solr/shard1/terms?terms.fl=field1
shows:
<lst name="field1">
<int name="a">58641</int>
<int name="the">45022</int>
<int name="i">36339</int>
<int name="s">35637</int>
<int name="d">34247</int>
<int name="m">33869</int>
<int name="b">28961</int>
<int name="r">28147</int>
<int name="e">27654</int>
<int name="n">26940</int>
</lst>

Thanks.

--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3590028.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Default Search UI not working

Posted by Martin Koch <ma...@issuu.com>.
Have you looked here http://wiki.apache.org/solr/VelocityResponseWriter  ?

/Martin


On Mon, Dec 19, 2011 at 12:44 PM, remi tassing <ta...@yahoo.com>wrote:

> Hello guys,
> the default search UI doesn't work for me.
> http://localhost:8983/solr/browse gives me an HTTP 404 error.
> I'm using Solr-1.4. Any idea how to fix this?
> Remi

Re: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
Any ideas?



--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p4106968.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
Yonik Seeley-2-2 wrote
> On Wed, Dec 28, 2011 at 5:47 AM, ku3ia &lt;

> demesg@

> &gt; wrote:
>> So, based on p.2) and on my previous researches, I conclude, that the
>> more
>> documents I want to retrieve, the slower is search and main problem is
>> the
>> cycle in writeDocs method. Am I right? Can you advice something in this
>> situation?
> 
> For the first phase in a distributed search, Solr must return the top
> N ids (in your case 200).  It currently does this by loading stored
> fields, which is slow.  A better approach is to store the "id" field
> as a column stride field.
> 
> https://issues.apache.org/jira/browse/SOLR-2753
> 
> -Yonik
> http://www.lucidimagination.com

Hi all again.
I have upgraded Solr version to 4.6.0 and want to test DocValues fields in
action.
My schema is: 4 shards of 48M of docs each. All shards are on the same
machine. I have turned on DV for "id" field in my schema.xml:

<field name="RecordID" type="uid" indexed="true" stored="true"
required="true" docValues="true"/>
...
<uniqueKey>RecordID</uniqueKey>
...
<fieldType name="uid" class="solr.StrField" sortMissingLast="true"
docValuesFormat="Disk"/>

I had created 30 queries like,
q=(a OR b) OR (c OR "d e" OR f), where "a" through "f" are random words

These queries were curled each other, not concurrently. After that I turned
off doc values, rebuild all four shards and runs queries again.

At results, a haven't seen any difference between stored ID field and
docValues and one more moment, when I set stored=false and docValues=true
for "id" field I had an error when running distributed search:

ERROR - 2013-12-12 16:59:45.508; org.apache.solr.common.SolrException;
java.lang.NullPointerException
        at
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1004)
        at
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:750)
        at
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:729)
        at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:312)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1863)
        at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:710)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
        at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
        at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
        at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
        at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
        at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
        at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
        at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
        at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
        at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
        at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
        at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
        at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)

This error is on QueryComponent on merge Ids:
resultIds.put(shardDoc.id.toString(), shardDoc);

So seems it try to use stored "id", not docvalue? I tried different codecs:
Direct, Disk, Memory, Lucene45. Results are nearby the same.

Can you advise me, may be I'm wrong in my configs or I'm not understand when
usage of CSF is faster than traditional stored fields.

Thanks.




--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p4106866.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Dec 28, 2011 at 5:47 AM, ku3ia <de...@gmail.com> wrote:
> So, based on p.2) and on my previous researches, I conclude, that the more
> documents I want to retrieve, the slower is search and main problem is the
> cycle in writeDocs method. Am I right? Can you advice something in this
> situation?

For the first phase in a distributed search, Solr must return the top
N ids (in your case 200).  It currently does this by loading stored
fields, which is slow.  A better approach is to store the "id" field
as a column stride field.

https://issues.apache.org/jira/browse/SOLR-2753

-Yonik
http://www.lucidimagination.com

RE: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
Hi all.
Due to my code review, I discovered next things:
1) as I wrote before, seems there is a low disk read speed;
2) at ~/solr-3.5/solr/core/src/java/org/apache/solr/response/XMLWriter.java
and in the same classes there is a writeDocList => writeDocs method, which
contains a cycle for of all docs;
3) as Michael Ryan wrote, this method uses SolrIndexSearcher.doc(int i,
Set<String> fields), which as I understand returns data from cache, or from
index;
4) I found a patch SOLR-1961
(https://issues.apache.org/jira/browse/SOLR-1961), but, seems it uses
RAMDirectoryFactory, so I can't apply it;

So, based on p.2) and on my previous researches, I conclude, that the more
documents I want to retrieve, the slower is search and main problem is the
cycle in writeDocs method. Am I right? Can you advice something in this
situation?

Thanks.


--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3616192.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
Hi!
Today I'd added loginfo to Solr here:
~/solr-3.5/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java

to method
  private void writeResponse(SolrQueryResponse solrRsp, ServletResponse
response,
                             QueryResponseWriter responseWriter,
SolrQueryRequest solrReq, Method reqMethod)
          throws IOException {

at

String charset = ContentStreamBase.getCharsetFromContentType(ct);
Writer out = (charset == null || charset.equalsIgnoreCase("UTF-8"))
            ? new OutputStreamWriter(response.getOutputStream(), UTF8)
            : new OutputStreamWriter(response.getOutputStream(), charset);
>>log here
out = new FastWriter(out);
>> log here
responseWriter.write(out, solrReq, solrRsp);
>> log here
out.flush();
>> log here

and what I have:
Dec 21, 2011 8:44:26 AM org.apache.solr.servlet.SolrDispatchFilter
writeResponse
WARNING: webapp=/solr path=/select/
params={fl=RecordID,score&start=0&q=mobile&rows=2000} hits=18887 status=0
QTime=2846 OutputStreamWriterTime=1
Dec 21, 2011 8:44:26 AM org.apache.solr.servlet.SolrDispatchFilter
writeResponse
WARNING: webapp=/solr path=/select/
params={fl=RecordID,score&start=0&q=mobile&rows=2000} hits=18887 status=0
QTime=2846 FastWriterTime=0
Dec 21, 2011 8:45:41 AM org.apache.solr.servlet.SolrDispatchFilter
writeResponse
WARNING: webapp=/solr path=/select/
params={fl=RecordID,score&start=0&q=mobile&rows=2000} hits=18887 status=0
QTime=2846 responseWriterTime=74207
Dec 21, 2011 8:45:41 AM org.apache.solr.servlet.SolrDispatchFilter
writeResponse
WARNING: webapp=/solr path=/select/
params={fl=RecordID,score&start=0&q=mobile&rows=2000} hits=18887 status=0
QTime=2846 outFlushTime=0

This is the first query, so Tomcat was started and I'd run this query. This
is on production only on one shard:
* ~29M docs 
* 11 fields 
* ~105M terms 
* size of shard is: 13GB

Have you any ideas, why so long or it is normal read speed time for
responseWriter of first query???

P.S. If I change a query patameter, for example on any keyword the
situations is similar.

Thanks.

--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3605074.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
tomas.zerolo wrote
> 
> But then the results would be wrong? Suppose the documents are not evenly
> distributed (wrt the sort criterium) across all the shards. In an extreme
> case, just imagine all 2000 top-most documents are on shard 3. You would
> get
> the 500 top-most (from shard 3) and some other you don't want (from the
> other shards). You wouldn't even know.
> 
> What SOLR is doig here is planning for the worst case.
> 
> Now if it could just do some piece-wise "merge sort" of sorts, that would
> be
> better.
> 
Yeah, it's true. But from the other side, the second month I'm trying to
resolve this problem on my huge data, so I'm attempting to raise any ideas,
maybe, even these ideas are incorrect.

Thanks.


--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3600648.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by Tomas Zerolo <to...@axelspringer.de>.
On Mon, Dec 19, 2011 at 01:32:22PM -0800, ku3ia wrote:
> >>Uhm, either I misunderstand your question or you're doing 
> >>a lot of extra work for nothing.... 
> 
> >>The whole point of sharding it exactly to collect the top N docs 
> >>from each shard and merge them into a single result [...]

> >>>>P.S. Is any mechanism, for example, to get top 100 rows from each shard,
> only merge it, sort by defined at query filed or score and pull result to
> the user?
> >>Uhm, either I misunderstand your question
> For example I have 4 shards. Finally, I need 2000 docs. Now, when I'm using
> &shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> Solr gets 2000 docs from each shard (shard1,2,3,4, summary we have 8000
> docs) merge and sort it, for example, by default field (score), and returns
> me only 2000 rows (not all 8000), which I specified at request.
> So, my question was about, is any mechanism in Solr, which gets not 2000
> rows from each shard, and say, If I specified 2000 docs at request, Solr
> calculates how much shards I have (four shards), divides total rows onto
> shards (2000/4=500) and sends to each shard queries with rows=500, but not
> rows=2000, so finally, summary after merging and sorting I'll have 2000 rows
> (maybe less), but not 8000... That was my question.

But then the results would be wrong? Suppose the documents are not evenly
distributed (wrt the sort criterium) across all the shards. In an extreme
case, just imagine all 2000 top-most documents are on shard 3. You would get
the 500 top-most (from shard 3) and some other you don't want (from the
other shards). You wouldn't even know.

What SOLR is doig here is planning for the worst case.

Now if it could just do some piece-wise "merge sort" of sorts, that would be
better.

-- 
Tomás Zerolo
Axel Springer AG
Axel Springer media Systems
BILD Produktionssysteme
Axel-Springer-Straße 65
10888 Berlin
Tel.: +49 (30) 2591-72875
tomas.zerolo@axelspringer.de
www.axelspringer.de

RE: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
project2501 wrote
> 
> I see what you are asking. This is an interesting question. It seems 
> inefficient for Solr to apply the
> requested rows to all shards only to discard most of the results on merge.
> That would consume lots of resources not used in the final result set.
> 
Yeah, like Erick says this is extra working. Maybe, when I asked to get 10
rows from my shards it is insensibly (10x30 or 10x4), but when I asked 2000
we have a big trouble. Finally, we have a lot of needless data.


Michael Ryan wrote
> 
> I had a similar requirement in my project, where a user might ask for up
> to 3000 results. What I did was change SolrIndexSearcher.doc(int, Set) to
> retrieve the unique key from the field cache instead of retrieving it as a
> stored field from disk. This resulted in a massive speed improvement for
> these requests (like 10x if I recall correctly).
> 
> -Michael
> 
Thanks Michel, I'll try it.

--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3599752.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by Chris Hostetter <ho...@fucit.org>.
: For example I have 4 shards. Finally, I need 2000 docs. Now, when I'm using
: &shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
: Solr gets 2000 docs from each shard (shard1,2,3,4, summary we have 8000
: docs) merge and sort it, for example, by default field (score), and returns
: me only 2000 rows (not all 8000), which I specified at request.
: So, my question was about, is any mechanism in Solr, which gets not 2000
: rows from each shard, and say, If I specified 2000 docs at request, Solr
: calculates how much shards I have (four shards), divides total rows onto
: shards (2000/4=500) and sends to each shard queries with rows=500, but not
: rows=2000, so finally, summary after merging and sorting I'll have 2000 rows
: (maybe less), but not 8000... That was my question.


Solr has to over-request to each of the shards in order to give you the 
"top" results from all shards (ie: sorted by score, or by whatever sort 
criteria you specify).

If Solr didn't do this, you could easily wind up in a situation where 
if you asked for the "top" 200 results across 10 shards, you would get the 
top 20 from *each* shard, but the top 200 across the whole index might 
really all be from a single shard.  Consider the case of an index sharded 
by date, and then you want to sort the documents by date -- or sorted by 
score, but the topic you are searching for is really preveland in "recent" 
documnts (ie: maybe it's collection of news articles sharded by decade, 
and you are searching for "twitter" which didn't exist 10 years ago) ... 
you would get terrible results if solr just divided rows/shard and asked 
each shard for that number of results.



-Hoss

RE: Poor performance on distributed search

Posted by Chris Hostetter <ho...@fucit.org>.
: I had a similar requirement in my project, where a user might ask for up 
: to 3000 results. What I did was change SolrIndexSearcher.doc(int, Set) 
: to retrieve the unique key from the field cache instead of retrieving it 
: as a stored field from disk. This resulted in a massive speed 
: improvement for these requests (like 10x if I recall correctly).

There was another discussion about this idea for phrase #1 of distributed 
search a while back...

http://osdir.com/ml/solr-user.lucene.apache.org/2011-07/msg00812.html

...but no one worked up a generalized patch - part of the issue is that it 
doesn't make sense in all cases, because the overhead of building the 
FieldCache for a unique key may be worse the the overhead of reading teh 
stored fields -- it depends on the pain points of the particular instance.  
(and as ryan points out: the new per field codec work + the memory codec 
might aleviate some of hte pain w/o needing any code changes here)

If folks are intersted in working up a patch to make this 
optional/configurable that would be awesome.



-Hoss

RE: Poor performance on distributed search

Posted by Michael Ryan <mr...@moreover.com>.
I had a similar requirement in my project, where a user might ask for up to 3000 results. What I did was change SolrIndexSearcher.doc(int, Set) to retrieve the unique key from the field cache instead of retrieving it as a stored field from disk. This resulted in a massive speed improvement for these requests (like 10x if I recall correctly).

-Michael

Re: Poor performance on distributed search

Posted by Darren Govoni <da...@ontrenet.com>.
I see what you are asking. This is an interesting question. It seems 
inefficient for Solr to apply the
requested rows to all shards only to discard most of the results on merge.
That would consume lots of resources not used in the final result set.

On 12/19/2011 04:32 PM, ku3ia wrote:
>>> Uhm, either I misunderstand your question or you're doing
>>> a lot of extra work for nothing....
>>> The whole point of sharding it exactly to collect the top N docs
>> >from each shard and merge them into a single result. So if
>>> you want 10 docs, just specify rows=10. Solr will query all
>>> the shards, get the top 10 docs from each and then
>>> merge them into a final list 10 items long. Both the initial
>>> fetch and the final merge are based on the
>>> sort criteria are respected.
>>> Score is the default "sort". If you specify other sort criteria,
>>> i.e. a field, then that sort is respected by the merge process.
>>> So why do you have this 2,000 requirement in the first
>>> place? This really sounds like an XY problem.
> As I wrote it is a minimum for me. I can't change it. Final response must
> has top 2K docs from all shards by query, so I specify rows=2000. Yeah, it
> collects top N docs from each shard. In my case N=2000, so on production I
> have 2000x30=60K, and on my own machine 2000x4=8K docs. Its true, this is an
> extra work, but in other case, seems it's only way to get top 2K docs from
> all shards, am I right?
>
>>>>> P.S. Is any mechanism, for example, to get top 100 rows from each shard,
> only merge it, sort by defined at query filed or score and pull result to
> the user?
>>> Uhm, either I misunderstand your question
> For example I have 4 shards. Finally, I need 2000 docs. Now, when I'm using
> &shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> Solr gets 2000 docs from each shard (shard1,2,3,4, summary we have 8000
> docs) merge and sort it, for example, by default field (score), and returns
> me only 2000 rows (not all 8000), which I specified at request.
> So, my question was about, is any mechanism in Solr, which gets not 2000
> rows from each shard, and say, If I specified 2000 docs at request, Solr
> calculates how much shards I have (four shards), divides total rows onto
> shards (2000/4=500) and sends to each shard queries with rows=500, but not
> rows=2000, so finally, summary after merging and sorting I'll have 2000 rows
> (maybe less), but not 8000... That was my question.
>
> Thanks.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3599636.html
> Sent from the Solr - User mailing list archive at Nabble.com.


Re: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
>>Uhm, either I misunderstand your question or you're doing 
>>a lot of extra work for nothing.... 

>>The whole point of sharding it exactly to collect the top N docs 
>>from each shard and merge them into a single result. So if 
>>you want 10 docs, just specify rows=10. Solr will query all 
>>the shards, get the top 10 docs from each and then 
>>merge them into a final list 10 items long. Both the initial 
>>fetch and the final merge are based on the 
>>sort criteria are respected. 

>>Score is the default "sort". If you specify other sort criteria, 
>>i.e. a field, then that sort is respected by the merge process. 

>>So why do you have this 2,000 requirement in the first 
>>place? This really sounds like an XY problem.
As I wrote it is a minimum for me. I can't change it. Final response must
has top 2K docs from all shards by query, so I specify rows=2000. Yeah, it
collects top N docs from each shard. In my case N=2000, so on production I
have 2000x30=60K, and on my own machine 2000x4=8K docs. Its true, this is an
extra work, but in other case, seems it's only way to get top 2K docs from
all shards, am I right?

>>>>P.S. Is any mechanism, for example, to get top 100 rows from each shard,
only merge it, sort by defined at query filed or score and pull result to
the user?
>>Uhm, either I misunderstand your question
For example I have 4 shards. Finally, I need 2000 docs. Now, when I'm using
&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
Solr gets 2000 docs from each shard (shard1,2,3,4, summary we have 8000
docs) merge and sort it, for example, by default field (score), and returns
me only 2000 rows (not all 8000), which I specified at request.
So, my question was about, is any mechanism in Solr, which gets not 2000
rows from each shard, and say, If I specified 2000 docs at request, Solr
calculates how much shards I have (four shards), divides total rows onto
shards (2000/4=500) and sends to each shard queries with rows=500, but not
rows=2000, so finally, summary after merging and sorting I'll have 2000 rows
(maybe less), but not 8000... That was my question.

Thanks.

--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3599636.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by Chris Hostetter <ho...@fucit.org>.
: So why do you have this 2,000 requirement in the first
: place? This really sounds like an XY problem.

I would really suggest re-visiting this question.  No sinle user is going 
to look at 2000 docs on a single page, and in your previous email you said 
there was a requirement to ask solr for 2000 docs so that you could 
"process" them -- suggesting that these aren't human generated queries 
being returned to a human, it sounds more likely that you are going to 
feed the results into some other system.

what is the other system?  what type of processing are you doing?  would 
it be more efficient to do that processing as a Solr plugin thta could run 
on the individual shards?

-Hoss

Re: Poor performance on distributed search

Posted by Erick Erickson <er...@gmail.com>.
Uhm, either I misunderstand your question or you're doing
a lot of extra work for nothing....

The whole point of sharding it exactly to collect the top N docs
from each shard and merge them into a single result. So if
you want 10 docs, just specify rows=10. Solr will query all
the shards, get the top 10 docs from each and then
merge them into a final list 10 items long. Both the initial
fetch and the final merge are based on the
sort criteria are respected.

Score is the default "sort". If you specify other sort criteria,
i.e. a field, then that sort is respected by the merge process.

So why do you have this 2,000 requirement in the first
place? This really sounds like an XY problem.


Best
Erick

On Mon, Dec 19, 2011 at 4:35 AM, ku3ia <de...@gmail.com> wrote:
> Hi, Erick. Thanks for your advice.
>>>Here's another test. Add &debugQuery=on to your query and post the
> results.
> Here is for 2K rows:
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">53153</int>
> <lst name="params">
> <str name="debugQuery">on</str>
> <str name="fl">*,score</str>
> <str name="shards">
> 127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> </str>
> <str name="ident">true</str>
> <str name="start">0</str>
> <str name="q">(mainstreaming)</str>
> <str name="rows">2000</str>
> </lst>
> </lst>
> <result name="response" numFound="2305" start="0" maxScore="4.657284">
>>>>Here 2K docs<<<
> </result>
> <lst name="debug">
> <str name="rawquerystring">(mainstreaming)</str>
> <str name="querystring">(mainstreaming)</str>
> <str name="parsedquery">ArticleText:mainstream</str>
> <str name="parsedquery_toString">ArticleText:mainstream</str>
> <str name="QParser">LuceneQParser</str>
> <lst name="timing">
> <double name="time">67797.0</double>
> <lst name="prepare">
> <double name="time">73.0</double>
> <lst name="org.apache.solr.handler.component.QueryComponent">
> <double name="time">72.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.FacetComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.HighlightComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.StatsComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.DebugComponent">
> <double name="time">0.0</double>
> </lst>
> </lst>
> <lst name="process">
> <double name="time">67724.0</double>
> <lst name="org.apache.solr.handler.component.QueryComponent">
> <double name="time">66607.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.FacetComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.HighlightComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.StatsComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.DebugComponent">
> <double name="time">1115.0</double>
> </lst>
> </lst>
> </lst>
> <lst name="explain">
> ...
> </lst>
> </lst>
> </response>
>
> And this is for 10:
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">3626</int>
> <lst name="params">
> <str name="debugQuery">on</str>
> <str name="fl">*,score</str>
> <str name="shards">
> 127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> </str>
> <str name="ident">true</str>
> <str name="start">0</str>
> <str name="q">(mainstreaming)</str>
> <str name="rows">10</str>
> </lst>
> </lst>
> <result name="response" numFound="2305" start="0" maxScore="4.657284">
>>>>Here 10 docs<<<
> </result>
> <lst name="debug">
> <str name="rawquerystring">(mainstreaming)</str>
> <str name="querystring">(mainstreaming)</str>
> <str name="parsedquery">ArticleText:mainstream</str>
> <str name="parsedquery_toString">ArticleText:mainstream</str>
> <str name="QParser">LuceneQParser</str>
> <lst name="timing">
> <double name="time">566.0</double>
> <lst name="prepare">
> <double name="time">17.0</double>
> <lst name="org.apache.solr.handler.component.QueryComponent">
> <double name="time">17.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.FacetComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.HighlightComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.StatsComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.DebugComponent">
> <double name="time">0.0</double>
> </lst>
> </lst>
> <lst name="process">
> <double name="time">549.0</double>
> <lst name="org.apache.solr.handler.component.QueryComponent">
> <double name="time">353.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.FacetComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.HighlightComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.StatsComponent">
> <double name="time">0.0</double>
> </lst>
> <lst name="org.apache.solr.handler.component.DebugComponent">
> <double name="time">196.0</double>
> </lst>
> </lst>
> </lst>
> <lst name="explain">
> ...
> </lst>
> </lst>
> </response>
>
>>>Also, I really have a hard time seeing what advantage you get from
>>>putting all those shards on the same machine, you're just creating
>>>extra work.
> Yeah, on my production I have 5 servers and 6 shards (big shards) on each.
> But I tried to use only one shard for each server (summary five shards) but
> results wasn't fine.
>
>>>Although there's one other possibility: By returning 2,000 rows, you
>>>require that each shard assemble a list of the top 2,000 documents
>>>and then they are collated into a single packet, so you're asking
>>>the system to do a lot of list processing.
> So, as I understand, my main problem is to get 2000 rows from each shard?
>
> P.S. Is any mechanism, for example, to get top 100 rows from each shard,
> only merge it, sort by defined at query filed or score and pull result to
> the user?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3597893.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Default Search UI not working

Posted by Chris Hostetter <ho...@fucit.org>.
: Subject: Default Search UI not working
: In-Reply-To: <13...@n3.nabble.com>

http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists

When starting a new discussion on a mailing list, please do not reply to 
an existing message, instead start a fresh email.  Even if you change the 
subject line of your email, other mail headers still track which thread 
you replied to and your question is "hidden" in that thread and gets less 
attention.   It makes following discussions in the mailing list archives 
particularly difficult.



-Hoss

Default Search UI not working

Posted by remi tassing <ta...@yahoo.com>.
Hello guys,
the default search UI doesn't work for me. http://localhost:8983/solr/browse gives me an HTTP 404 error.
I'm using Solr-1.4. Any idea how to fix this?
Remi

Re: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
Hi, Erick. Thanks for your advice.
>>Here's another test. Add &debugQuery=on to your query and post the
results.
Here is for 2K rows:
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">53153</int>
<lst name="params">
<str name="debugQuery">on</str>
<str name="fl">*,score</str>
<str name="shards">
127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
</str>
<str name="ident">true</str>
<str name="start">0</str>
<str name="q">(mainstreaming)</str>
<str name="rows">2000</str>
</lst>
</lst>
<result name="response" numFound="2305" start="0" maxScore="4.657284">
>>>Here 2K docs<<<
</result>
<lst name="debug">
<str name="rawquerystring">(mainstreaming)</str>
<str name="querystring">(mainstreaming)</str>
<str name="parsedquery">ArticleText:mainstream</str>
<str name="parsedquery_toString">ArticleText:mainstream</str>
<str name="QParser">LuceneQParser</str>
<lst name="timing">
<double name="time">67797.0</double>
<lst name="prepare">
<double name="time">73.0</double>
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">72.0</double>
</lst>
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">0.0</double>
</lst>
</lst>
<lst name="process">
<double name="time">67724.0</double>
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">66607.0</double>
</lst>
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">1115.0</double>
</lst>
</lst>
</lst>
<lst name="explain">
...
</lst>
</lst>
</response>

And this is for 10:
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">3626</int>
<lst name="params">
<str name="debugQuery">on</str>
<str name="fl">*,score</str>
<str name="shards">
127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
</str>
<str name="ident">true</str>
<str name="start">0</str>
<str name="q">(mainstreaming)</str>
<str name="rows">10</str>
</lst>
</lst>
<result name="response" numFound="2305" start="0" maxScore="4.657284">
>>>Here 10 docs<<<
</result>
<lst name="debug">
<str name="rawquerystring">(mainstreaming)</str>
<str name="querystring">(mainstreaming)</str>
<str name="parsedquery">ArticleText:mainstream</str>
<str name="parsedquery_toString">ArticleText:mainstream</str>
<str name="QParser">LuceneQParser</str>
<lst name="timing">
<double name="time">566.0</double>
<lst name="prepare">
<double name="time">17.0</double>
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">17.0</double>
</lst>
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">0.0</double>
</lst>
</lst>
<lst name="process">
<double name="time">549.0</double>
<lst name="org.apache.solr.handler.component.QueryComponent">
<double name="time">353.0</double>
</lst>
<lst name="org.apache.solr.handler.component.FacetComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.MoreLikeThisComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.HighlightComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.StatsComponent">
<double name="time">0.0</double>
</lst>
<lst name="org.apache.solr.handler.component.DebugComponent">
<double name="time">196.0</double>
</lst>
</lst>
</lst>
<lst name="explain">
...
</lst>
</lst>
</response>

>>Also, I really have a hard time seeing what advantage you get from 
>>putting all those shards on the same machine, you're just creating 
>>extra work.
Yeah, on my production I have 5 servers and 6 shards (big shards) on each.
But I tried to use only one shard for each server (summary five shards) but
results wasn't fine.

>>Although there's one other possibility: By returning 2,000 rows, you 
>>require that each shard assemble a list of the top 2,000 documents 
>>and then they are collated into a single packet, so you're asking 
>>the system to do a lot of list processing.
So, as I understand, my main problem is to get 2000 rows from each shard?

P.S. Is any mechanism, for example, to get top 100 rows from each shard,
only merge it, sort by defined at query filed or score and pull result to
the user?

--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3597893.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by Erick Erickson <er...@gmail.com>.
Here's another test. Add &debugQuery=on to your query and post the results.
I believe you'll find that the QTime parameter returned in the packet
is quite small. That is the amount of time spent in the query, NOT
the time spent reading the docs form disk to return. And if that number
remains small whether you search 10 or 2,000 docs, then your
problem is just that you're reading 2,000 documents from disk
and returning them. And that has nothing to do with sharding, it's
just how things work....

Although there's one other possibility: By returning 2,000 rows, you
require that each shard assemble a list of the top 2,000 documents
and then they are collated into a single packet, so you're asking
the system to do a lot of list processing.

Also, I really have a hard time seeing what advantage you get from
putting all those shards on the same machine, you're just creating
extra work.

Best
Erick

On Sat, Dec 17, 2011 at 4:11 PM, ku3ia <de...@gmail.com> wrote:
> Hi, Erick!
>
>>>Right, are you falling afoul of the recursive shard thing? That is,
>>>if you shards point back to itself. As far as I understand, your
>>>shards parameter in your request handler shouldn't point back
>>>to itself....
> No, my request handler don't points itself cause default is false. But I
> tested today, using classic shards parameter. Here is results:
>
> successfully started at 22:25:58
> Queries interval is: 10 queries per minute
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(jurisdictions)&rows=2000&start=0&fl=*,score&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> ...
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(mainstreaming)&rows=2000&start=0&fl=*,score&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> utility successfully stopped at 22:30:58
> Elapsed time: 300 secs
> --- solr ---
> Queries processed: 11
> Queries cancelled: 39
> Average QTime is: 55966.8 ms
> Average RTime is: 56.2727 sec(s)
> Size of data-dir is: 496726 bytes
>
> And test with RecordID only:
> successfully started at 22:17:16
> Queries interval is: 10 queries per minute
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(simulating)&rows=2000&start=0&fl=RecordID&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> ...
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(multimillionaire's)&rows=2000&start=0&fl=RecordID&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
> utility successfully stopped at 22:22:16
> Elapsed time: 300 secs
> --- solr ---
> Queries processed: 23
> Queries cancelled: 27
> Average QTime is: 13991.8 ms
> Average RTime is: 14.1739 sec(s)
> Size of data-dir is: 223580 bytes
>
> And read speed is the same.
>
> On production I create a RAMDisk (mount -t tmpfs -o size=xxxM tmpfs
> /mnt/tmpfs/) and put my shards onto it. The QTime/RTime was magnificent. But
> the problem is, I dont have much RAM for it(
>
> Can it be my huge index, terms or maybe it's to much queries per minute, or
> row count or I made something wrong in my configs? And, can you please watch
> drive speed on your indexes? Is it the same?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3594683.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
Hi, Erick!

>>Right, are you falling afoul of the recursive shard thing? That is, 
>>if you shards point back to itself. As far as I understand, your 
>>shards parameter in your request handler shouldn't point back 
>>to itself.... 
No, my request handler don't points itself cause default is false. But I
tested today, using classic shards parameter. Here is results:

successfully started at 22:25:58
Queries interval is: 10 queries per minute
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(jurisdictions)&rows=2000&start=0&fl=*,score&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
...
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(mainstreaming)&rows=2000&start=0&fl=*,score&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
utility successfully stopped at 22:30:58
Elapsed time: 300 secs
--- solr ---
Queries processed: 11
Queries cancelled: 39
Average QTime is: 55966.8 ms
Average RTime is: 56.2727 sec(s)
Size of data-dir is: 496726 bytes

And test with RecordID only:
successfully started at 22:17:16
Queries interval is: 10 queries per minute
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(simulating)&rows=2000&start=0&fl=RecordID&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
...
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(multimillionaire's)&rows=2000&start=0&fl=RecordID&shards=127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4
utility successfully stopped at 22:22:16
Elapsed time: 300 secs
--- solr ---
Queries processed: 23
Queries cancelled: 27
Average QTime is: 13991.8 ms
Average RTime is: 14.1739 sec(s)
Size of data-dir is: 223580 bytes

And read speed is the same.

On production I create a RAMDisk (mount -t tmpfs -o size=xxxM tmpfs
/mnt/tmpfs/) and put my shards onto it. The QTime/RTime was magnificent. But
the problem is, I dont have much RAM for it(

Can it be my huge index, terms or maybe it's to much queries per minute, or
row count or I made something wrong in my configs? And, can you please watch
drive speed on your indexes? Is it the same?

--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3594683.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by Erick Erickson <er...@gmail.com>.
Right, are you falling afoul of the recursive shard thing? That is,
if you shards point back to itself. As far as I understand, your
shards parameter in your request handler shouldn't point back
to itself....

But I'm guessing here.

Best
Erick

On Fri, Dec 16, 2011 at 4:27 PM, ku3ia <de...@gmail.com> wrote:
>>> OK, so your speed differences are pretty much dependent upon whether you
> specify
>>> rows=2000 or rows=10, right? Why do you need 2,000 rows?
> Yes, big difference is 10 v. 2K records. Limit of 2K rows is setted by
> manager and I can't decrease it. It is a minimum row count needed to process
> data.
>
>>> Or is the root question why there's such a difference when you specify
>>> qt=requestShards? In which case I'm curious to see that request
>>> handler definition...
>
>  <requestHandler name="requestShards" class="solr.SearchHandler"
> default="false">
>     <lst name="defaults">
>       <str name="echoParams">explicit</str>
>       <int name="rows">10</int>
>       <str
> name="shards">127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4</str>
>     </lst>
>    </requestHandler>
>
> This request handler is defined at shard1's solrconfig.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3592734.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
>> OK, so your speed differences are pretty much dependent upon whether you
specify 
>> rows=2000 or rows=10, right? Why do you need 2,000 rows? 
Yes, big difference is 10 v. 2K records. Limit of 2K rows is setted by
manager and I can't decrease it. It is a minimum row count needed to process
data.

>> Or is the root question why there's such a difference when you specify 
>> qt=requestShards? In which case I'm curious to see that request 
>> handler definition... 

  <requestHandler name="requestShards" class="solr.SearchHandler"
default="false">
     <lst name="defaults">
       <str name="echoParams">explicit</str>
       <int name="rows">10</int>
       <str
name="shards">127.0.0.1:8080/solr/shard1,127.0.0.1:8080/solr/shard2,127.0.0.1:8080/solr/shard3,127.0.0.1:8080/solr/shard4</str>
     </lst>
    </requestHandler>

This request handler is defined at shard1's solrconfig.

--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3592734.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by Erick Erickson <er...@gmail.com>.
OK, so your speed differences are pretty much dependent upon whether you specify
rows=2000 or rows=10, right? Why do you need 2,000 rows?

Or is the root question why there's such a difference when you specify
qt=requestShards? In which case I'm curious to see that request
handler definition...

Best
Erick

On Fri, Dec 16, 2011 at 1:38 PM, ku3ia <de...@gmail.com> wrote:
> Hi, Erick, thanks for your reply
>
> Yeah, you are right - document cache is default, but I tried to decrease and
> increase values but I didn't get the desired result.
>
> I tried the tests. Here are results:
>
>>>1> try with "&rows=10"
> successfully started at 19:48:34
> Queries interval is: 10 queries per minute
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(gulping)&rows=10&start=0&fl=*,score&qt=requestShards
> ...
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(tabors)&rows=10&start=0&fl=*,score&qt=requestShards
> utility successfully stopped at 19:53:33
> Elapsed time: 299 secs
> --- solr ---
> Queries processed: 50
> Queries cancelled: 0
> Average QTime is: 764 ms
> Average RTime is: 0.68 sec(s)
> Size of data-dir is: 235784 bytes
>
>>>2> try with "&fl=id" assuming "id" is your <uniqueKey>
> successfully started at 19:56:23
> Queries interval is: 10 queries per minute
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(psyche's)&rows=2000&start=0&fl=RecordID&qt=requestShards
> ...
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(betook)&rows=2000&start=0&fl=RecordID&qt=requestShards
> utility successfully stopped at 20:01:24
> Elapsed time: 301 secs
> --- solr ---
> Queries processed: 15
> Queries cancelled: 35
> Average QTime is: 52775.7 ms
> Average RTime is: 53.2667 sec(s)
> Size of data-dir is: 212978 bytes
>
> In first test disk usage by nmon: ~30-40% and in the second - 100%. Drive
> read speed starting from 3-5 MB/s and falls to 500-700 KB/s in both tests.
>
> Have you any ideas?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3592364.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by ku3ia <de...@gmail.com>.
Hi, Erick, thanks for your reply

Yeah, you are right - document cache is default, but I tried to decrease and
increase values but I didn't get the desired result.

I tried the tests. Here are results:

>>1> try with "&rows=10"
successfully started at 19:48:34
Queries interval is: 10 queries per minute
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(gulping)&rows=10&start=0&fl=*,score&qt=requestShards
...
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(tabors)&rows=10&start=0&fl=*,score&qt=requestShards
utility successfully stopped at 19:53:33
Elapsed time: 299 secs
--- solr ---
Queries processed: 50
Queries cancelled: 0
Average QTime is: 764 ms
Average RTime is: 0.68 sec(s)
Size of data-dir is: 235784 bytes

>>2> try with "&fl=id" assuming "id" is your <uniqueKey> 
successfully started at 19:56:23
Queries interval is: 10 queries per minute
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(psyche's)&rows=2000&start=0&fl=RecordID&qt=requestShards
...
http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(betook)&rows=2000&start=0&fl=RecordID&qt=requestShards
utility successfully stopped at 20:01:24
Elapsed time: 301 secs
--- solr ---
Queries processed: 15
Queries cancelled: 35
Average QTime is: 52775.7 ms
Average RTime is: 53.2667 sec(s)
Size of data-dir is: 212978 bytes

In first test disk usage by nmon: ~30-40% and in the second - 100%. Drive
read speed starting from 3-5 MB/s and falls to 500-700 KB/s in both tests.

Have you any ideas?

--
View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3592364.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Poor performance on distributed search

Posted by Erick Erickson <er...@gmail.com>.
The thing that jumps out at me is "&rows=2000". If you documentCache in
solrconfig.xml is still the defaults, it only holds 512. So you're running
all over your disk gathering up the fields to return, especially since
you also specified "fl=*,score". And if you have large fields stored, you're
doing an awful lot of disk reading.....

simple tests to see if this is on the right track, try these, singly
and in combination.

1> try with "&rows=10"
2> try with "&fl=id" assuming "id" is your <uniqueKey>

Best
Erick

On Thu, Dec 15, 2011 at 5:00 PM, ku3ia <de...@gmail.com> wrote:
> Hi, all!
>
> I have a problem with distributed search. I downloaded one shard from my
> production. It has:
> * ~29M docs
> * 11 fields
> * ~105M terms
> * size of shard is: 13GB
> On production there are near 30 the same shards. I split this shard to 4
> more smaller shards, so now I have:
> small shard1:
> docs: 6.2M
> terms: 27.2M
> size: 2.89GB
> small shard2:
> docs: 6.3M
> terms: 28.7M
> size: 2.98GB
> small shard3:
> docs: 7.9M
> terms: 32.8M
> size: 3.60GB
> small shard4:
> docs: 8.2M
> terms: 32.6M
> size: 3.70GB
>
> My machine confguration:
> ABIT AX-78
> AMD Athlon 64 X2 5200+
> DDR2 Kingston 2x2G+2x1G = 6G
> WDC WD2500JS (System here)
> WDC WD20EARS (6 partitions = 30 GB for shards at begin of drive, and other
> empty, all partitions are well aligned)
> GNU/Linux Debian Squeeze
> Tomcat 6.0.32 with JAVA_OPTS:
> JAVA_OPTS="$JAVA_OPTS -XX:+DisableExplicitGC -server \
>    -XX:PermSize=512M -XX:MaxPermSize=512M -Xmx4096M -Xms4096M
> -XX:NewSize=128M -XX:MaxNewSize=128M \
>    -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled \
>    -XX:CMSInitiatingOccupancyFraction=50 -XX:GCTimeRatio=9
> -XX:MinHeapFreeRatio=25 -XX:MaxHeapFreeRatio=25 \
>    -verbose:gc -XX:+PrintGCTimeStamps -Xloggc:$CATALINA_HOME/logs/gc.log"
> Solr 3.5
>
> I configured 4 cores and start Tomcat. I write a bash script. It's runing
> during 300 seconds and sending every 6 seconds queries like
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(assistants)&rows=2000&start=0&fl=*,score&qt=requestShards
> where qt=requestShards is my 4 shards. After test I have the results:
>
> Elapsed time: 299 secs
> --- solr ---
> Queries processed: 21 << this is full response file
> Queries cancelled: 29 << this is number of killed curls
> Average QTime is: 59645.6 ms
> Average RTime is: 59.7619 sec(s) << this is average time difference between
> start and stop the curl. There is a part of script:
> # >>dcs=`date +%s`
> # >>curl ${url} -s -H 'Content-type:text/xml; charset=utf-8' >
> ${F_DATADIR}/$dest.fdata
> # >>dce=`date +%s`
> # >>dcd=$(echo "$dce - $dcs" | bc)
> Size of data-dir is: 3346766 bytes << this is response dir size
>
> I'm using nmon to to monitor R/W disk speed, and I was surprised that read
> speed of my shards volumes WDC20EAR's drive was nearly 3 MB/s when script is
> working. After this I run benchmark test from disk utility. Here is results:
> Minimum read rate: 53.2MB/s
> Maximum Read rate: 126.4 MB/s
> Average Read rate: 95.8 MB/s
>
> But from the other side I tested queries like
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(assistants)&rows=2000&start=0&fl=*,score
> results is:
> Elapsed time: 299 secs
> --- solr ---
> Queries processed: 50
> Queries cancelled: 0
> Average QTime is: 139.76 ms
> Average RTime is: 2.2 sec(s)
> Size of data-dir is: 6819259 bytes
>
> and quesries like
> http://127.0.0.1:8080/solr/shard1/select/?ident=true&q=(assistants)&rows=2000&start=0&fl=*,score&shards=127.0.0.1:8080/solr/shard1
> and result is:
> Elapsed time: 299 secs
> --- solr ---
> Queries processed: 49
> Queries cancelled: 1
> Average QTime is: 1878.37 ms
> Average RTime is: 1.95918 sec(s)
> Size of data-dir is: 4274099 bytes
> So we see the results are the same.
>
> My big question is: why is so slow drive read speed when Solr is working?
> Thanks for any replies
>
> P.S. And maybe my general problem is too much terms in shard, for example,
> query
> http://127.0.0.1:8080/solr/shard1/terms?terms.fl=field1
> shows:
> <lst name="field1">
> <int name="a">58641</int>
> <int name="the">45022</int>
> <int name="i">36339</int>
> <int name="s">35637</int>
> <int name="d">34247</int>
> <int name="m">33869</int>
> <int name="b">28961</int>
> <int name="r">28147</int>
> <int name="e">27654</int>
> <int name="n">26940</int>
> </lst>
>
> Thanks.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Poor-performance-on-distributed-search-tp3590028p3590028.html
> Sent from the Solr - User mailing list archive at Nabble.com.