[jira] [Commented] (SOLR-14044) Support shard/collection deletion in shared storage

["Andy Vuong (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999472#comment-16999472 ] 

Andy Vuong commented on SOLR-14044:
-----------------------------------

Our current shared store deletion path involves the use of an asynchronous process manager, BlobDeleteManager, in which we enqueue eligible files for deletion and this has been scoped to one deletion flow, which is the “activation” and “deactivation” of segment files as indexing progresses locally on a node. 

More concretely: in the indexing path of a shared collection, we resolve the differences between our local index and source of truth (blob) and after we’re synced, track files missing locally (were formerly present and then likely deleted locally as indexing happens) to be deleted.

Eligible files to be deleted refer to the files that are being “tracked”, and have aged beyond a deletion threshold. That threshold is currently 30 seconds. A good heuristic hasn’t been explored yet but we haven’t seen an issue with this value so far in our load tests.  

BlobDeleteManager uses a thread pool of 5 deleter threads and a BlockingQueue, allowing up to 200 deletion tasks at a time.

> Support shard/collection deletion in shared storage
> ---------------------------------------------------
>
>                 Key: SOLR-14044
>                 URL: https://issues.apache.org/jira/browse/SOLR-14044
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud
>            Reporter: Andy Vuong
>            Priority: Major
>
> The Solr Cloud deletion APIs for collections and shards are not currently supported by shared storage but are an essential functionality required by the shared storage design. Deletion of objects from shared storage currently only happens in the indexing path (on pushes) and after the index file listings between the local solr process and external store have been resolved.
>  
> This task is to track supporting the delete shard/collection API commands and its scope does not include cleaning up so called “orphaned” index files from blob (i.e. files that are no longer referenced by any core.metadata file on the external store). This will be designed/covered in another subtask.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org