You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Houston Putman (Jira)" <ji...@apache.org> on 2025/07/15 17:54:00 UTC

[jira] [Comment Edited] (SOLR-17792) ParallelHttpShardHandler has massive performance issues.

    [ https://issues.apache.org/jira/browse/SOLR-17792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18007303#comment-18007303 ] 

Houston Putman edited comment on SOLR-17792 at 7/15/25 5:53 PM:
----------------------------------------------------------------

Copying from the PR:

So this broke {{DistributedDebugComponentTest.testTolerantSearch}} and {{{}TestJsonFacets.testTolerant{}}}. The reason is that originally {{HttpShardHandler.take()}} would return on an error, cancelling all other responses. Now the logic is that on an error, outstanding requests will be cancelled, but if any responses come in in the meantime, they will be processed. However, the code currently expects that the last response will have the error. So since the last response might not have the error anymore, we have to change the code that checks the responses for errors.

However, after fixing that and beasting the tests, there is a really weird error around cancelling requests. The {{DistributedDebugComponentTest.testTolerantSearch}} does a non-tolerant search then does a tolerant search. The error I described above was breaking the non-tolerant search. That is easily fixable. The second part of the test, testing tolerant search fails very occasionally (but only when the non-tolerant search is done first, when that is commented out, the tolerant search does not fail).

The tolerant search fails (occasionally) because all three shard requests fail instead of just 1 of the shard requests failing (because of a non-exisistant endpoint). the bad shard has the failure that the test expects, but the good shards both fail with {{java.io.IOException: cancel_stream_error/unexpected_data_frame}} meaning that the requests were cancelled, even thought the request is "tolerant". I did a lot of debugging here, and noticed that Solr is behaving correctly and we are not cancelling shard requests for tolerant solr requests. And the fact that if the "non-tolerant search" request case right before the tolerant search request is commented out, the failures stop, tell us that the cancellations from the non-tolerant request are bleeding into the tolerant request. This is bad. I also confirmed this by commenting out the line that actually cancels the HTTP requests: [https://github.com/apache/solr/blob/branch_9_9/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L570-L574]

This only happens on branch_9x (and presumable branch_9_9), not on main. So I believe it's a bug in Jetty 10, which Jetty 12 has solved. So we are probably fine just fixing this part on branch_9x and branch_9_9, and leaving the request cancellation enabled on main (10.x).

Amazingly, when beasting, there is a big difference in whether the non-existent endpoint is put first or last in the list of shards. The failure rate is much higher when the bad shard is the first listed rather the last one listed.

I've create another ticket (SOLR-17819) for the cancellation bug. But I'll create another PR to fix what this PR broke (the first part of this long comment).


was (Author: houston):
Copying from the PR:

So this broke {{DistributedDebugComponentTest.testTolerantSearch}} and {{{}TestJsonFacets.testTolerant{}}}. The reason is that originally {{HttpShardHandler.take()}} would return on an error, cancelling all other responses. Now the logic is that on an error, outstanding requests will be cancelled, but if any responses come in in the meantime, they will be processed. However, the code currently expects that the last response will have the error. So since the last response might not have the error anymore, we have to change the code that checks the responses for errors.

However, after fixing that and beasting the tests, there is a really weird error around cancelling requests. The {{DistributedDebugComponentTest.testTolerantSearch}} does a non-tolerant search then does a tolerant search. The error I described above was breaking the non-tolerant search. That is easily fixable. The second part of the test, testing tolerant search fails very occasionally (but only when the non-tolerant search is done first, when that is commented out, the tolerant search does not fail).

The tolerant search fails (occasionally) because all three shard requests fail instead of just 1 of the shard requests failing (because of a non-exisistant endpoint). the bad shard has the failure that the test expects, but the good shards both fail with {{java.io.IOException: cancel_stream_error/unexpected_data_frame}} meaning that the requests were cancelled, even thought the request is "tolerant". I did a lot of debugging here, and noticed that Solr is behaving correctly and we are not cancelling shard requests for tolerant solr requests. And the fact that if the "non-tolerant search" request case right before the tolerant search request is commented out, the failures stop, tell us that the cancellations from the non-tolerant request are bleeding into the tolerant request. This is bad. I also confirmed this by commenting out the line that actually cancels the HTTP requests: [https://github.com/apache/solr/blob/branch_9_9/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L570-L574]

This only happens on branch_9x (and presumable branch_9_9), not on main. So I believe it's a bug in Jetty 10, which Jetty 12 has solved. So we are probably fine just fixing this part on branch_9x and branch_9_9, and leaving the request cancellation enabled on main (10.x).

Amazingly, when beasting, there is a big difference in whether the non-existent endpoint is put first or last in the list of shards. The failure rate is much higher when the bad shard is the first listed rather the last one listed.

I'm going to create another ticket for the cancellation bug. But I'll create another PR to fix what this PR broke (the first part of this long comment).

> ParallelHttpShardHandler has massive performance issues.
> --------------------------------------------------------
>
>                 Key: SOLR-17792
>                 URL: https://issues.apache.org/jira/browse/SOLR-17792
>             Project: Solr
>          Issue Type: Bug
>    Affects Versions: 9.8
>            Reporter: Houston Putman
>            Assignee: Houston Putman
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 9.9
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> SOLR-17158 changed the way that the HttpShardHandler (And ParallelHttpShardHandler) did locking and concurrency. However, after upgrading, and running distributed queries (at a relatively slow rate), I noticed that there were 3 types of responses:
>  * QTimes between 3-6ms
>  * QTimes between 53-56 ms
>  * And requests that timed out
> Looking at the logic in HttpShardHandler, there is a poll(50ms) call that is very suspicious, and likely the reason for the jump between 3-6 ms and 53-56 ms. I would also assume that this change in concurrency logic is the reason that many requests started timing out. Changing to the HttpShardHandlerFactory from the ParallellHttpShardHandlerFactory fixed these issues.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org