[jira] [Commented] (SOLR-13932) Review directory locking for Blob interactions

["Ilan Ginzburg (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974191#comment-16974191 ] 

Ilan Ginzburg commented on SOLR-13932:
--------------------------------------

Commented code in {{ServerSideMetadata}} constructor regarding {{DirectoryReader.listCommits()}} mentions "But in blob context we serialize commits and pulls by proper locking therefore we should be good here". I believe this is no longer true (a previous Salesforce version of that code used to serialize updates). We do not serialize commits but let Lucene do it the way it wants (we serialize pushes to Blob).

OTOH, I don't see why we try to detect inconsistency using past files that are no longer part of the commit point. If we rely on such detections, we're at risk because files of past commit points can be deleted at any time. Our strategy must make sure (I believe it does) that local is always consistent, or local should be deleted and the Blob version adopted instead. The only time where local is "a source of truth" over blob is after indexing. Consistency in such a case is guaranteed by verifying that local was consistent with Blob before indexing started (blob being source of truth), and that Blob hasn't changed by the time indexing finished and got pushed back to Blob (allowing parallel indexing from given node/replica, this is done by the conditional update into ZK).

> Review directory locking for Blob interactions
> ----------------------------------------------
>
>                 Key: SOLR-13932
>                 URL: https://issues.apache.org/jira/browse/SOLR-13932
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Ilan Ginzburg
>            Priority: Major
>
> Review resolution of local index directory content vs Blob copy.
> There has been wrong understanding of following line acquiring a lock on index directory.
>  {{solrCore.getDirectoryFactory().get(indexDirPath, DirectoryFactory.DirContext.DEFAULT, solrCore.getSolrConfig().indexConfig.lockType);}}
> From Yonik:
> _A couple things about Directory locking.... the locks were only ever to prevent more than one IndexWriter from trying to modify the same index. The IndexWriter grabs a write lock once when it is created and does not release it until it is closed._ 
> _Directories are not locked on acquisition of the Directory from the DirectoryFactory. See the IndexWriter constructor, where the lock is explicitly grabbed._
> Review CorePushPull#pullUpdateFromBlob, ServerSideMetadata and other classes as relevant.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org