You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "XJ Wang (Created) (JIRA)" <ji...@apache.org> on 2012/02/07 02:40:59 UTC
[jira] [Created] (SOLR-3104) Bad performance with distributed
search when sort contains relevancy queries
Bad performance with distributed search when sort contains relevancy queries
----------------------------------------------------------------------------
Key: SOLR-3104
URL: https://issues.apache.org/jira/browse/SOLR-3104
Project: Solr
Issue Type: Improvement
Components: search
Affects Versions: 3.6
Reporter: XJ Wang
Priority: Critical
So I found this issue when trying out distributed search with solr 3.5 and noticed big performance degradation for some queries comparing to the single box search.
After some query analysis and comparison, it turns out that shard queries with "fsv=true" are much slower than the same queries w/o "fsv=true". Some examples are like 1200ms vs 200ms (start=0, rows=30, hits<100).
>From the discussions with Yonik Seeley on solr mailing list, it may due to fact that I'm using lot of relevancy queries in sorting. But Solr is not retrieving those sort values efficiently .
This is critical for us and prevents us from moving to distributed search. I believe users like our scenarios will also suffer from this issue. Any patch/idea is welcomed.
Quote from Yonik Seeley on solr-user mailing list:
"OK, so basically it's slow because functions with embedded relevancy
queries are "forward only" - if you request the value for a docid
previous to the last, we need to reboot the query (re-weight, ask for
the scorer, etc). This means that for your 30 documents, that will
require rebooting the query about 15 times (assuming that roughly half
of the time the next docid will be less than the previous one).
Unfortunately there's not much you can do externally... we need to
implement optimizations at the Solr level for this."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-3104) Bad performance with distributed
search when sort contains relevancy queries
Posted by "Yonik Seeley (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley resolved SOLR-3104.
--------------------------------
Resolution: Fixed
Fix Version/s: 4.0
Committed to trunk.
> Bad performance with distributed search when sort contains relevancy queries
> ----------------------------------------------------------------------------
>
> Key: SOLR-3104
> URL: https://issues.apache.org/jira/browse/SOLR-3104
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 3.6
> Reporter: XJ Wang
> Priority: Critical
> Fix For: 4.0
>
> Attachments: SOLR-3104.patch
>
>
> So I found this issue when trying out distributed search with solr 3.5 and noticed big performance degradation for some queries comparing to the single box search.
> After some query analysis and comparison, it turns out that shard queries with "fsv=true" are much slower than the same queries w/o "fsv=true". Some examples are like 1200ms vs 200ms (start=0, rows=30, hits<100).
> From the discussions with Yonik Seeley on solr mailing list, it may due to fact that I'm using lot of relevancy queries in sorting. But Solr is not retrieving those sort values efficiently .
> This is critical for us and prevents us from moving to distributed search. I believe users like our scenarios will also suffer from this issue. Any patch/idea is welcomed.
> Quote from Yonik Seeley on solr-user mailing list:
> "OK, so basically it's slow because functions with embedded relevancy
> queries are "forward only" - if you request the value for a docid
> previous to the last, we need to reboot the query (re-weight, ask for
> the scorer, etc). This means that for your 30 documents, that will
> require rebooting the query about 15 times (assuming that roughly half
> of the time the next docid will be less than the previous one).
> Unfortunately there's not much you can do externally... we need to
> implement optimizations at the Solr level for this."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3104) Bad performance with distributed
search when sort contains relevancy queries
Posted by "XJ Wang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204201#comment-13204201 ]
XJ Wang commented on SOLR-3104:
-------------------------------
Michael, Mark, yeah I ported it for 3.x (same with the patch uploaded here) and was just about to post it. You guys are fast. It'll be nice to commit this into 3.x as well.
> Bad performance with distributed search when sort contains relevancy queries
> ----------------------------------------------------------------------------
>
> Key: SOLR-3104
> URL: https://issues.apache.org/jira/browse/SOLR-3104
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 3.6
> Reporter: XJ Wang
> Priority: Critical
> Fix For: 4.0
>
> Attachments: SOLR-3104-3x.patch, SOLR-3104.patch
>
>
> So I found this issue when trying out distributed search with solr 3.5 and noticed big performance degradation for some queries comparing to the single box search.
> After some query analysis and comparison, it turns out that shard queries with "fsv=true" are much slower than the same queries w/o "fsv=true". Some examples are like 1200ms vs 200ms (start=0, rows=30, hits<100).
> From the discussions with Yonik Seeley on solr mailing list, it may due to fact that I'm using lot of relevancy queries in sorting. But Solr is not retrieving those sort values efficiently .
> This is critical for us and prevents us from moving to distributed search. I believe users like our scenarios will also suffer from this issue. Any patch/idea is welcomed.
> Quote from Yonik Seeley on solr-user mailing list:
> "OK, so basically it's slow because functions with embedded relevancy
> queries are "forward only" - if you request the value for a docid
> previous to the last, we need to reboot the query (re-weight, ask for
> the scorer, etc). This means that for your 30 documents, that will
> require rebooting the query about 15 times (assuming that roughly half
> of the time the next docid will be less than the previous one).
> Unfortunately there's not much you can do externally... we need to
> implement optimizations at the Solr level for this."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3104) Bad performance with distributed
search when sort contains relevancy queries
Posted by "Mark Miller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-3104:
------------------------------
Attachment: SOLR-3104-3x.patch
here is a 3x patch from my back merge
> Bad performance with distributed search when sort contains relevancy queries
> ----------------------------------------------------------------------------
>
> Key: SOLR-3104
> URL: https://issues.apache.org/jira/browse/SOLR-3104
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 3.6
> Reporter: XJ Wang
> Priority: Critical
> Fix For: 4.0
>
> Attachments: SOLR-3104-3x.patch, SOLR-3104.patch
>
>
> So I found this issue when trying out distributed search with solr 3.5 and noticed big performance degradation for some queries comparing to the single box search.
> After some query analysis and comparison, it turns out that shard queries with "fsv=true" are much slower than the same queries w/o "fsv=true". Some examples are like 1200ms vs 200ms (start=0, rows=30, hits<100).
> From the discussions with Yonik Seeley on solr mailing list, it may due to fact that I'm using lot of relevancy queries in sorting. But Solr is not retrieving those sort values efficiently .
> This is critical for us and prevents us from moving to distributed search. I believe users like our scenarios will also suffer from this issue. Any patch/idea is welcomed.
> Quote from Yonik Seeley on solr-user mailing list:
> "OK, so basically it's slow because functions with embedded relevancy
> queries are "forward only" - if you request the value for a docid
> previous to the last, we need to reboot the query (re-weight, ask for
> the scorer, etc). This means that for your 30 documents, that will
> require rebooting the query about 15 times (assuming that roughly half
> of the time the next docid will be less than the previous one).
> Unfortunately there's not much you can do externally... we need to
> implement optimizations at the Solr level for this."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-3104) Bad performance with distributed
search when sort contains relevancy queries
Posted by "Yonik Seeley (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley updated SOLR-3104:
-------------------------------
Attachment: SOLR-3104.patch
Here's a patch for trunk that should help the situation by sorting the ids first and getting the sort values in order.
> Bad performance with distributed search when sort contains relevancy queries
> ----------------------------------------------------------------------------
>
> Key: SOLR-3104
> URL: https://issues.apache.org/jira/browse/SOLR-3104
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 3.6
> Reporter: XJ Wang
> Priority: Critical
> Attachments: SOLR-3104.patch
>
>
> So I found this issue when trying out distributed search with solr 3.5 and noticed big performance degradation for some queries comparing to the single box search.
> After some query analysis and comparison, it turns out that shard queries with "fsv=true" are much slower than the same queries w/o "fsv=true". Some examples are like 1200ms vs 200ms (start=0, rows=30, hits<100).
> From the discussions with Yonik Seeley on solr mailing list, it may due to fact that I'm using lot of relevancy queries in sorting. But Solr is not retrieving those sort values efficiently .
> This is critical for us and prevents us from moving to distributed search. I believe users like our scenarios will also suffer from this issue. Any patch/idea is welcomed.
> Quote from Yonik Seeley on solr-user mailing list:
> "OK, so basically it's slow because functions with embedded relevancy
> queries are "forward only" - if you request the value for a docid
> previous to the last, we need to reboot the query (re-weight, ask for
> the scorer, etc). This means that for your 30 documents, that will
> require rebooting the query about 15 times (assuming that roughly half
> of the time the next docid will be less than the previous one).
> Unfortunately there's not much you can do externally... we need to
> implement optimizations at the Solr level for this."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3104) Bad performance with distributed
search when sort contains relevancy queries
Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203487#comment-13203487 ]
Michael McCandless commented on SOLR-3104:
------------------------------------------
Shouldn't we commit this fix on 3.x as well...? Is there any reason not to? Sounds like a nasty bug, and it sounds like XJ has applied the patch on 3.x successfully.
> Bad performance with distributed search when sort contains relevancy queries
> ----------------------------------------------------------------------------
>
> Key: SOLR-3104
> URL: https://issues.apache.org/jira/browse/SOLR-3104
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 3.6
> Reporter: XJ Wang
> Priority: Critical
> Fix For: 4.0
>
> Attachments: SOLR-3104.patch
>
>
> So I found this issue when trying out distributed search with solr 3.5 and noticed big performance degradation for some queries comparing to the single box search.
> After some query analysis and comparison, it turns out that shard queries with "fsv=true" are much slower than the same queries w/o "fsv=true". Some examples are like 1200ms vs 200ms (start=0, rows=30, hits<100).
> From the discussions with Yonik Seeley on solr mailing list, it may due to fact that I'm using lot of relevancy queries in sorting. But Solr is not retrieving those sort values efficiently .
> This is critical for us and prevents us from moving to distributed search. I believe users like our scenarios will also suffer from this issue. Any patch/idea is welcomed.
> Quote from Yonik Seeley on solr-user mailing list:
> "OK, so basically it's slow because functions with embedded relevancy
> queries are "forward only" - if you request the value for a docid
> previous to the last, we need to reboot the query (re-weight, ask for
> the scorer, etc). This means that for your 30 documents, that will
> require rebooting the query about 15 times (assuming that roughly half
> of the time the next docid will be less than the previous one).
> Unfortunately there's not much you can do externally... we need to
> implement optimizations at the Solr level for this."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3104) Bad performance with distributed
search when sort contains relevancy queries
Posted by "XJ Wang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203166#comment-13203166 ]
XJ Wang commented on SOLR-3104:
-------------------------------
Yonik, your fast response and help is much appreciated. I applied the patch and saw some big improvements. The slow query I mentioned in description went from 1200ms to 280ms! Thanks again.
> Bad performance with distributed search when sort contains relevancy queries
> ----------------------------------------------------------------------------
>
> Key: SOLR-3104
> URL: https://issues.apache.org/jira/browse/SOLR-3104
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 3.6
> Reporter: XJ Wang
> Priority: Critical
> Fix For: 4.0
>
> Attachments: SOLR-3104.patch
>
>
> So I found this issue when trying out distributed search with solr 3.5 and noticed big performance degradation for some queries comparing to the single box search.
> After some query analysis and comparison, it turns out that shard queries with "fsv=true" are much slower than the same queries w/o "fsv=true". Some examples are like 1200ms vs 200ms (start=0, rows=30, hits<100).
> From the discussions with Yonik Seeley on solr mailing list, it may due to fact that I'm using lot of relevancy queries in sorting. But Solr is not retrieving those sort values efficiently .
> This is critical for us and prevents us from moving to distributed search. I believe users like our scenarios will also suffer from this issue. Any patch/idea is welcomed.
> Quote from Yonik Seeley on solr-user mailing list:
> "OK, so basically it's slow because functions with embedded relevancy
> queries are "forward only" - if you request the value for a docid
> previous to the last, we need to reboot the query (re-weight, ask for
> the scorer, etc). This means that for your 30 documents, that will
> require rebooting the query about 15 times (assuming that roughly half
> of the time the next docid will be less than the previous one).
> Unfortunately there's not much you can do externally... we need to
> implement optimizations at the Solr level for this."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Reopened] (SOLR-3104) Bad performance with distributed
search when sort contains relevancy queries
Posted by "Mark Miller (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller reopened SOLR-3104:
-------------------------------
> Bad performance with distributed search when sort contains relevancy queries
> ----------------------------------------------------------------------------
>
> Key: SOLR-3104
> URL: https://issues.apache.org/jira/browse/SOLR-3104
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 3.6
> Reporter: XJ Wang
> Priority: Critical
> Fix For: 4.0
>
> Attachments: SOLR-3104-3x.patch, SOLR-3104.patch
>
>
> So I found this issue when trying out distributed search with solr 3.5 and noticed big performance degradation for some queries comparing to the single box search.
> After some query analysis and comparison, it turns out that shard queries with "fsv=true" are much slower than the same queries w/o "fsv=true". Some examples are like 1200ms vs 200ms (start=0, rows=30, hits<100).
> From the discussions with Yonik Seeley on solr mailing list, it may due to fact that I'm using lot of relevancy queries in sorting. But Solr is not retrieving those sort values efficiently .
> This is critical for us and prevents us from moving to distributed search. I believe users like our scenarios will also suffer from this issue. Any patch/idea is welcomed.
> Quote from Yonik Seeley on solr-user mailing list:
> "OK, so basically it's slow because functions with embedded relevancy
> queries are "forward only" - if you request the value for a docid
> previous to the last, we need to reboot the query (re-weight, ask for
> the scorer, etc). This means that for your 30 documents, that will
> require rebooting the query about 15 times (assuming that roughly half
> of the time the next docid will be less than the previous one).
> Unfortunately there's not much you can do externally... we need to
> implement optimizations at the Solr level for this."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-3104) Bad performance with distributed
search when sort contains relevancy queries
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203715#comment-13203715 ]
Mark Miller commented on SOLR-3104:
-----------------------------------
bq. Shouldn't we commit this fix on 3.x as well...?
+1 - though it doesn't apply cleanly to 3.x, seems easy enough to get this to work - I can probably take a crack at it.
Probably should add a changes entry as well.
> Bad performance with distributed search when sort contains relevancy queries
> ----------------------------------------------------------------------------
>
> Key: SOLR-3104
> URL: https://issues.apache.org/jira/browse/SOLR-3104
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 3.6
> Reporter: XJ Wang
> Priority: Critical
> Fix For: 4.0
>
> Attachments: SOLR-3104.patch
>
>
> So I found this issue when trying out distributed search with solr 3.5 and noticed big performance degradation for some queries comparing to the single box search.
> After some query analysis and comparison, it turns out that shard queries with "fsv=true" are much slower than the same queries w/o "fsv=true". Some examples are like 1200ms vs 200ms (start=0, rows=30, hits<100).
> From the discussions with Yonik Seeley on solr mailing list, it may due to fact that I'm using lot of relevancy queries in sorting. But Solr is not retrieving those sort values efficiently .
> This is critical for us and prevents us from moving to distributed search. I believe users like our scenarios will also suffer from this issue. Any patch/idea is welcomed.
> Quote from Yonik Seeley on solr-user mailing list:
> "OK, so basically it's slow because functions with embedded relevancy
> queries are "forward only" - if you request the value for a docid
> previous to the last, we need to reboot the query (re-weight, ask for
> the scorer, etc). This means that for your 30 documents, that will
> require rebooting the query about 15 times (assuming that roughly half
> of the time the next docid will be less than the previous one).
> Unfortunately there's not much you can do externally... we need to
> implement optimizations at the Solr level for this."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org