You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2024/08/07 03:51:00 UTC
[jira] [Commented] (SOLR-17102) VersionBucket not needed
[ https://issues.apache.org/jira/browse/SOLR-17102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871532#comment-17871532 ]
ASF subversion and git services commented on SOLR-17102:
--------------------------------------------------------
Commit 7ccf4d3c9af7a03dc5ffce8cceda11c360fbccd0 in solr's branch refs/heads/main from David Smiley
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=7ccf4d3c9af ]
SOLR-17102: Replaced VersionBucket array with locks on-demand (#2548)
The VersionBucket indexing lock mechanism was replaced with something just as fast yet that which consumes almost no memory, saving 1MB of memory per SolrCore.
Removed numVersionBuckets and versionBucketLockTimeoutMs.
Refactorings:
DistributedUpdateProcessor: some refactoring to balance locks clearly.
VersionInfo: moved locks out to new UpdateLocks.
OrderedExecutor uses generics now. Allows use of the ID directly (a BytesRef) instead of a hash.
> VersionBucket not needed
> ------------------------
>
> Key: SOLR-17102
> URL: https://issues.apache.org/jira/browse/SOLR-17102
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Reporter: David Smiley
> Assignee: David Smiley
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> SolrCloud ensures that updates for the same document ID are done in the correct order internally in the face of possible re-orders during replication / log replay. In order to ensure the updates are applied consecutively, a lock is held on a hash of the ID for the doc. A hash is used to limit the number of total locks because the locks are pre-created in advance for the core (numVersionBuckets == 65k by default). The memory is non-negligible with many cores, and it introduces the possibility of collisions, especially at lower bucket counts if you configure it much lower.
> Here I propose doing away with a pre-created hashed bucket strategy. Instead, I propose more simply creating and GC'ing a lock per update being processed, and using a ConcurrentHashMap to hold those in-flight. This strategy is already used in org.apache.solr.util.OrderedExecutor.SparseStripedLock, more or less.
> Doing this is more tractable now that VersionBucket only holds a lock, not a version anymore – SOLR-17036
> The biggest challenge is that the code calls for the ability to use a Condition to away/notify, which means the solution can't just re-use SparseStripedLock above nor be quite so simple.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org