You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Pierre Salagnac <pi...@gmail.com> on 2023/07/03 20:27:44 UTC
Re: Circuit breakers and update requests
Hi Jan,
As far as I know, Solr only supports circuit breaks for queries at the
moment.
We have a custom integration of circuit breakers for indexing (in Solr 8,
so that's not fully aligned with what in solr 9) with a custom
UpdateRequestProcessor. Basically, a new instance of every update processor
is created by calling the corresponding factory. We then check system stats
(CPU...) at the first handled doc.
Maybe there are other ways of integrating circuit breakers, but at least
doing so in an update processor was not intrusive in the code.
Probably it is worth it to have something standard in Solr. A "symetric"
class of SearchHandler to add CB integration could be UpdateRequestHandler.
Pierre
Le mar. 27 juin 2023 à 13:32, Jan Høydahl <ja...@cominvent.com> a écrit :
> Hi,
>
> Solr has CPU and Memory circuit breakers that will terminate Search
> Requests only.
> See
> https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html
> for docs.
>
> Example:
>
> <circuitBreaker class="solr.CircuitBreakerManager" enabled="true">
> <str name="memEnabled">true</str>
> <str name="memThreshold">75</str>
> <str name="cpuEnabled">true</str>
> <str name="cpuThreshold">75</str>
> </circuitBreaker>
>
>
> A Solr node typically gets overloaded by the combined update and query
> traffic, and
> I'm looking into enabling ciricuit breakers for update requests. For many
> workloads, pausing
> update traffic would resolve the situation, with the benefit of users not
> being affected by aborted
> queries.
>
> So ideally I'd want to be able to choose to enable/disable CB on
> update/query individually.
> Or better, to kill update requests on e.g. 80% threshold and search
> requests on 90% threshold.
>
> The current breaker impl for search <
> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L372:380>
> [1] is hardcoded into SearchHandler in such a way that if ANY
> of the configured breakers trips, search requests are aborted. Also the
> breakers seem to be generic
> in nature, named CPUCircuitBreaker <
> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java>
> and MemoryCircuitBreaker <
> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java>,
> so adding new UpdateCPUCircuitBreaker
> does not seem to be the intention here.
>
> I'm also unclear on the usefulness of having CB on the core level and not
> the node level. If you have
> 10 cores from 10 collections on a node, and only some have CBs while
> others do not, the node will
> still be overloaded unless the admin convinces every collection owner to
> implement the same CBs?
>
> So my question becomes - how do we enable CB for update requests into this
> mix in a clean way?
>
> Jan
Re: Circuit breakers and update requests
Posted by Jan Høydahl <ja...@cominvent.com>.
There was no feedback on the topic of global Circuit Breaker vs per-core.
So I opened https://issues.apache.org/jira/browse/SOLR-16974 to propose supporting Global CB. This is still on design level, please weigh in with your thoughts.
Jan
> 30. aug. 2023 kl. 22:02 skrev Jan Høydahl <ja...@cominvent.com>:
>
> Hi
>
> For those interested in this topic, here is the JIRA, which has a Pull Request for review: https://issues.apache.org/jira/browse/SOLR-16954
>
> Jan Høydahl
>
>> 3. jul. 2023 kl. 22:28 skrev Pierre Salagnac <pi...@gmail.com>:
>>
>> Hi Jan,
>>
>> As far as I know, Solr only supports circuit breaks for queries at the
>> moment.
>>
>> We have a custom integration of circuit breakers for indexing (in Solr 8,
>> so that's not fully aligned with what in solr 9) with a custom
>> UpdateRequestProcessor. Basically, a new instance of every update processor
>> is created by calling the corresponding factory. We then check system stats
>> (CPU...) at the first handled doc.
>> Maybe there are other ways of integrating circuit breakers, but at least
>> doing so in an update processor was not intrusive in the code.
>>
>> Probably it is worth it to have something standard in Solr. A "symetric"
>> class of SearchHandler to add CB integration could be UpdateRequestHandler.
>>
>> Pierre
>>
>>
>> Le mar. 27 juin 2023 à 13:32, Jan Høydahl <ja...@cominvent.com> a écrit :
>>
>>> Hi,
>>>
>>> Solr has CPU and Memory circuit breakers that will terminate Search
>>> Requests only.
>>> See
>>> https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html
>>> for docs.
>>>
>>> Example:
>>>
>>> <circuitBreaker class="solr.CircuitBreakerManager" enabled="true">
>>> <str name="memEnabled">true</str>
>>> <str name="memThreshold">75</str>
>>> <str name="cpuEnabled">true</str>
>>> <str name="cpuThreshold">75</str>
>>> </circuitBreaker>
>>>
>>>
>>> A Solr node typically gets overloaded by the combined update and query
>>> traffic, and
>>> I'm looking into enabling ciricuit breakers for update requests. For many
>>> workloads, pausing
>>> update traffic would resolve the situation, with the benefit of users not
>>> being affected by aborted
>>> queries.
>>>
>>> So ideally I'd want to be able to choose to enable/disable CB on
>>> update/query individually.
>>> Or better, to kill update requests on e.g. 80% threshold and search
>>> requests on 90% threshold.
>>>
>>> The current breaker impl for search <
>>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L372:380>
>>> [1] is hardcoded into SearchHandler in such a way that if ANY
>>> of the configured breakers trips, search requests are aborted. Also the
>>> breakers seem to be generic
>>> in nature, named CPUCircuitBreaker <
>>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java>
>>> and MemoryCircuitBreaker <
>>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java>,
>>> so adding new UpdateCPUCircuitBreaker
>>> does not seem to be the intention here.
>>>
>>> I'm also unclear on the usefulness of having CB on the core level and not
>>> the node level. If you have
>>> 10 cores from 10 collections on a node, and only some have CBs while
>>> others do not, the node will
>>> still be overloaded unless the admin convinces every collection owner to
>>> implement the same CBs?
>>>
>>> So my question becomes - how do we enable CB for update requests into this
>>> mix in a clean way?
>>>
>>> Jan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: Circuit breakers and update requests
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi
For those interested in this topic, here is the JIRA, which has a Pull Request for review: https://issues.apache.org/jira/browse/SOLR-16954
Jan Høydahl
> 3. jul. 2023 kl. 22:28 skrev Pierre Salagnac <pi...@gmail.com>:
>
> Hi Jan,
>
> As far as I know, Solr only supports circuit breaks for queries at the
> moment.
>
> We have a custom integration of circuit breakers for indexing (in Solr 8,
> so that's not fully aligned with what in solr 9) with a custom
> UpdateRequestProcessor. Basically, a new instance of every update processor
> is created by calling the corresponding factory. We then check system stats
> (CPU...) at the first handled doc.
> Maybe there are other ways of integrating circuit breakers, but at least
> doing so in an update processor was not intrusive in the code.
>
> Probably it is worth it to have something standard in Solr. A "symetric"
> class of SearchHandler to add CB integration could be UpdateRequestHandler.
>
> Pierre
>
>
> Le mar. 27 juin 2023 à 13:32, Jan Høydahl <ja...@cominvent.com> a écrit :
>
>> Hi,
>>
>> Solr has CPU and Memory circuit breakers that will terminate Search
>> Requests only.
>> See
>> https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html
>> for docs.
>>
>> Example:
>>
>> <circuitBreaker class="solr.CircuitBreakerManager" enabled="true">
>> <str name="memEnabled">true</str>
>> <str name="memThreshold">75</str>
>> <str name="cpuEnabled">true</str>
>> <str name="cpuThreshold">75</str>
>> </circuitBreaker>
>>
>>
>> A Solr node typically gets overloaded by the combined update and query
>> traffic, and
>> I'm looking into enabling ciricuit breakers for update requests. For many
>> workloads, pausing
>> update traffic would resolve the situation, with the benefit of users not
>> being affected by aborted
>> queries.
>>
>> So ideally I'd want to be able to choose to enable/disable CB on
>> update/query individually.
>> Or better, to kill update requests on e.g. 80% threshold and search
>> requests on 90% threshold.
>>
>> The current breaker impl for search <
>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L372:380>
>> [1] is hardcoded into SearchHandler in such a way that if ANY
>> of the configured breakers trips, search requests are aborted. Also the
>> breakers seem to be generic
>> in nature, named CPUCircuitBreaker <
>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/CPUCircuitBreaker.java>
>> and MemoryCircuitBreaker <
>> https://github.com/apache/solr/blob/branch_9_2/solr/core/src/java/org/apache/solr/util/circuitbreaker/MemoryCircuitBreaker.java>,
>> so adding new UpdateCPUCircuitBreaker
>> does not seem to be the intention here.
>>
>> I'm also unclear on the usefulness of having CB on the core level and not
>> the node level. If you have
>> 10 cores from 10 collections on a node, and only some have CBs while
>> others do not, the node will
>> still be overloaded unless the admin convinces every collection owner to
>> implement the same CBs?
>>
>> So my question becomes - how do we enable CB for update requests into this
>> mix in a clean way?
>>
>> Jan