[jira] [Updated] (SOLR-16487) Improve calculation of poll interval for replicas

["Justin Sweeney (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-16487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Sweeney updated SOLR-16487:
----------------------------------
    Summary: Improve calculation of poll interval for replicas  (was: Improve efficiency of pull replicas)

> Improve calculation of poll interval for replicas
> -------------------------------------------------
>
>                 Key: SOLR-16487
>                 URL: https://issues.apache.org/jira/browse/SOLR-16487
>             Project: Solr
>          Issue Type: Improvement
>          Components: replication (java)
>    Affects Versions: 9.0, 8.11.2
>            Reporter: Justin Sweeney
>            Priority: Minor
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently there are a few inefficiencies related to pull replicas that can cause increased resource usage on a Solr cluster, particularly if there are a large number of cores.
> The following are the inefficiencies identified:
>  # The [RecoveryStrategy.replicate()|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java#L219] method makes a call to commit to on the leader. This happens whenever a replica is reloaded. For PULL replicas in particular this isn't necessary since we can just pull down whatever the latest data is and rely on other mechanisms to be consistently committing the leader. (As an aside, it seems like forcing a commit on the leader might never be necessary, but for this I've limited it to focusing on PULL replicas).
>  # In a case where the leader has no data yet (index version is 0), then a non-leader replica will consistently delete and recreate its core due to this case in IndexFetcher: [https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L549.] This can cause unnecessary CPU usage until the leader has data indexed to it.
>  # The polling for replication is fairly simply, but can lead to polling too often. As an example if you had the following config for commits:
> {code:java}
> <autoCommit>
>     <maxTime>15000</maxTime>
>     <openSearcher>false</openSearcher>
> </autoCommit>
> <autoSoftCommit>
>     <maxTime>60000</maxTime>
> </autoSoftCommit> {code}
> The current logic would setup polling to be half of the autoCommit time, so poll every 7.5 seconds. However since a new searcher isn't opened, there will only be changes reflected every 60 seconds on the leader. We can make this logic a bit smarter knowing that the replication handler won't reflect changes until a new searcher is opened.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org