[jira] [Resolved] (SOLR-15051) Shared storage -- BlobDirectory (de-duping)

["David Smiley (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Smiley resolved SOLR-15051.
---------------------------------
    Resolution: Won't Fix

Maybe Bruno could put it on his fork as he crafted all or nearly all that exists.

This is debatable but I'm closing this now as won't-fix.  The title of the issue contains "de-duping" and the proposed way to achieve this specifically is (a) rather complicated, and (b) hasn't been started, and moreover (c) is an approach I no longer want to put any energy into because I think I'd rather go about things differently.  I instead favor an approach where only the leader writes to remote storage that is one place for the shard, not per-replica, and thus there is no duplication concern.  It would be some other issue/(s), perhaps SOLR-6237.

I really like what we've developed here so far; it doesn't have nor _need_ de-duping to be useful.  It's backend, BackupRepository, is an abstraction Solr already has with a growing number of implementations.  We could deliver just this by itself, and do so under a new JIRA issue.  I'm not sure of known bugs/gaps; ideally we'd have time to push it past a finish line and compare its performance to HdfsDirectory, which I'm very curious about.

> Shared storage -- BlobDirectory (de-duping)
> -------------------------------------------
>
>                 Key: SOLR-15051
>                 URL: https://issues.apache.org/jira/browse/SOLR-15051
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: David Smiley
>            Assignee: David Smiley
>            Priority: Major
>          Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> This proposal is a way to accomplish shared storage in SolrCloud with a few key characteristics: (A) using a Directory implementation, (B) delegates to a backing local file Directory as a kind of read/write cache (C) replicas have their own "space", (D) , de-duplication across replicas via reference counting, (E) uses ZK but separately from SolrCloud stuff.
> The Directory abstraction is a good one, and helps isolate shared storage from the rest of SolrCloud that doesn't care.  Using a backing normal file Directory is faster for reads and is simpler than Solr's HDFSDirectory's BlockCache.  Replicas having their own space solves the problem of multiple writers (e.g. of the same shard) trying to own and write to the same space, and it implies that any of Solr's replica types can be used along with what goes along with them like peer-to-peer replication (sometimes faster/cheaper than pulling from shared storage).  A de-duplication feature solves needless duplication of files across replicas and from parent shards (i.e. from shard splitting).  The de-duplication feature requires a place to cache directory listings so that they can be shared across replicas and atomically updated; this is handled via ZooKeeper.  Finally, some sort of Solr daemon / auto-scaling code should be added to implement "autoAddReplicas", especially to provide for a scenario where the leader is gone and can't be replicated from directly but we can access shared storage.
> For more about shared storage concepts, consider looking at the description in SOLR-13101 and the linked Google Doc.
> *[PROPOSAL DOC|https://docs.google.com/document/d/1kjQPK80sLiZJyRjek_Edhokfc5q9S3ISvFRM2_YeL8M/edit?usp=sharing]*



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org