You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Ilan Ginzburg <il...@gmail.com> on 2024/01/13 22:10:17 UTC
[DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud
I have created SIP-20
https://cwiki.apache.org/confluence/display/SOLR/SIP-20%3A+Separation+of+Compute+and+Storage+in+SolrCloud
In the next few days I will create a Jira + a branch that implements
the SIP proposal and that includes documentation on how to approach
that branch and what's in it.
This proposed contribution is based on work done at Salesforce these
last few years and currently running at scale in multiple regions.
Thanks,
Ilan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud
Posted by Ilan Ginzburg <il...@gmail.com>.
I have created a Jira, the branch with the current code and
documentation and updated the SIP-20
<https://cwiki.apache.org/confluence/display/SOLR/SIP-20%3A+Separation+of+Compute+and+Storage+in+SolrCloud>
page:
Jira: SOLR-17125 <https://issues.apache.org/jira/browse/SOLR-17125>
Branch: jira/solr-17125-zero-replicas
<https://github.com/apache/solr/tree/jira/solr-17125-zero-replicas>
Documentation (in the branch): ZERO-REPLICAS.adoc
<https://github.com/apache/solr/blob/jira/solr-17125-zero-replicas/solr/ZERO-REPLICAS.adoc>
This code is based on code running in production at large scale, but that
code underwent heavy cleanup, preparation and porting for sharing it here.
It is quite possible that regressions were introduced, the code as
presented *did not* run in production.
I'd be happy to answer any questions and if there's interest, I can host
live sessions for code walkthrough or discussions.
Ilan
On Tue, Jan 23, 2024 at 1:49 AM Ilan Ginzburg <il...@gmail.com> wrote:
> To clarify - since you mentioned durability, I'm assuming the commit that
>> happens on each update batch is a "hard commit"?
>>
> Yes.
>
> If the user made an explicit hard-commit or had an autoCommit configured,
>> would that be a no-op for ZERO collections?
>>
> Yes.
>
>
>> How do soft-commits behave - I imagine they're handled similarly to
>> whatever is done for PULL replicas currently?
>>
> Currently a hard commit is automatically added after the last document of
> a batch. Not sure what would happen if that wasn't the case and a soft
> commit was issued instead.
> In any case the soft commit would not propagate to other ZERO replicas
> than the leader that's for sure (I assume when you mention soft commit and
> PULL you mean soft commit on another replica?). The soft commit will also
> not write the files to the shared repository. So likely the batch would be
> acknowledged to the client but would be lost if a failure hit the node
> before the next (hard) commit (no transaction log, nodes are assumed
> stateless, so Zero replicas skip recovery on node startup).
>
> And that ZK read happens on each query? Or does the current strategy have
>> it check "every X seconds" like how PULL/TLOG replicas check for updates?
>>
> (about a replica checking with a ZK read if it's up to date). In the
> current implementation such a read is enqueued each time there's a query,
> but basically until that read has happened then another query arrived,
> there will not be a second read. So if for example a lot of queries arrive
> in the same second before the read happens, there would be a single read.
> Or if the cluster is loaded (many replicas get queries and enqueue such
> reads from ZK), the reads will not be executed right away and no new read
> will be enqueued until the previous one was done.
> Spacing such reads by some minimum delay is not a big change.
>
> Is that ZK check looking at the "shard term" stuff, or would Zero
>> replicas/collections store some additional state in ZK to manage this?
>
> It is not shard terms. Terms are being used when a follower replica does
> not manage to get an indexing update from the leader. Given Zero replicas
> do not send indexing updates to other replicas, terms are never increased
> and are therefore not used for this type of replica.
> The value that is read in ZK is basically the pointer to the current
> metadata file for the shard in the [Backup]Repository (it's called
> BackupRepository but the Zero implementation uses it as a general purpose
> repository). When the shard in the repository changes, the shard metadata
> file changes (you can think of it as listing the current stored commit
> point in the repository). The shard ZooKeeper node references the current
> valid shard metadata file in the repository. A node that has read that
> value (then the repository metadata file then all the segments) can check
> if the ZooKeeper node has changed, and if it has not, the node already has
> the current version of the shard.
>
> Ilan
>
> On Mon, Jan 22, 2024 at 10:20 PM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
>> Thanks for the detailed response! I've got a few follow-ups that I'll ask
>> in line, but I think I've got the core idea now, thanks!
>>
>> > With the implementation in its current state, an indexing batch (as sent
>> > by the client) is processed by the leader replica, committed to local
>> disk
>> > then written into the remote storage (S3) before SolrCloud acknowledges
>> the
>> > batch. Transaction log is not used and the commit is forced.
>>
>> To clarify - since you mentioned durability, I'm assuming the commit that
>> happens on each update batch is a "hard commit"?
>>
>> If the user made an explicit hard-commit or had an autoCommit configured,
>> would that be a no-op for ZERO collections?
>>
>> How do soft-commits behave - I imagine they're handled similarly to
>> whatever is done for PULL replicas currently?
>>
>> > Currently the (basic) approach is that a replica checks if it is up to
>> > date as long as it's getting queries. If it is already up to date, the
>> cost
>> > of that check is a ZooKeeper node read.
>>
>> And that ZK read happens on each query? Or does the current strategy have
>> it check "every X seconds" like how PULL/TLOG replicas check for updates?
>>
>> Is that ZK check looking at the "shard term" stuff, or would Zero
>> replicas/collections store some additional state in ZK to manage this?
>>
>> Best,
>>
>> Jason
>>
>> On Wed, Jan 17, 2024 at 8:05 PM Ilan Ginzburg <il...@gmail.com> wrote:
>>
>> > Hi Jason,
>> >
>> > Good questions!
>> > Note there is a related question on the users list, see thread Node
>> roles
>> > vs SIP-20 Separation of Compute and Storage
>> > <https://lists.apache.org/thread/ox7xbl1hd2j87ccvlyjho4kqqv2jnfmc>
>> >
>> > When I share the code (later this week in all likelihood) I will share a
>> > detailed write up. I'm open to discuss/present/write more as needed (and
>> > plan to attend Thursday 18/1 Meetup
>> > <
>> https://cwiki.apache.org/confluence/display/SOLR/2024-01-18+Meeting+notes
>> >
>> > to
>> > present the SIP).
>> >
>> > To your questions:
>> >
>> > - The name "Zero" is because "Shared" was too broad (and already
>> used)
>> > and "Blob" was already used as well in Solr code, and we didn't find
>> any
>> > better name (but the name can be changed, there are a couple thousand
>> > occurrences of it in different forms in the code, I know because
>> "Zero"
>> > is
>> > not the name we use internally at Salesforce for that replica type,
>> I've
>> > renamed everything for sharing here). The reason for "Zero" is that
>> > there
>> > is only one instance (zero copies) in persistent storage for each
>> shard,
>> > and also this evoques the (longer term) option to have zero (i.e. no)
>> > replicas for a shard on the cluster, pick a node, load and
>> materialize a
>> > replica when needed. But that's longer term.
>> > - Your understanding is correct, currently it's a kind of new type of
>> > collection given all replicas are ZERO (or none are) for a
>> collection.
>> > ZERO
>> > could have been a new type of shard, as one could imagine different
>> > shards
>> > having different types (but shards do not have types), or why not
>> also
>> > allow a given shard to have ZERO replicas as well as other types of
>> > replicas (this is not supported in the current implementation for
>> > simplicity and we didn't really felt the need). If collectively we
>> think
>> > that PULL + ZERO do make sense, why not. PULL would then be fetching
>> its
>> > content from ZERO replicas rather than directly from the shared
>> > storage. I
>> > don't see ZERO coexisting with NRT or TLOG in a given shard though.
>> > Currently as the implementation forces all replicas of all shards of
>> a
>> > "Zero" collection to be ZERO, there is a flag on that collection
>> saying
>> > it
>> > is a Zero collection.
>> > - How the compute behaves:
>> > With the implementation in its current state, an indexing batch (as
>> sent
>> > by the client) is processed by the leader replica, committed to local
>> > disk
>> > then written into the remote storage (S3) before SolrCloud
>> acknowledges
>> > the
>> > batch. Transaction log is not used and the commit is forced. This
>> slows
>> > down indexing, but a design guideline from the start was that the
>> > SolrCloud
>> > nodes are stateless (can restart with empty disk) to simplify
>> managing
>> > elasticity (and in a public cloud setting, Availability Zone
>> failures).
>> > The next evolution step of this code that we plan to start upstream
>> once
>> > the branch has been shared is to enable the transaction log for ZERO
>> > replicas, and have indexing behave more like a normal SolrCloud:
>> persist
>> > the transaction log before replying to a client, do not force a
>> commit.
>> > The
>> > transaction log would be changed to be on shared storage as well,
>> with a
>> > single copy for a shard, not one log per node/replica. All replicas
>> of
>> > the
>> > shard, when they're becoming leaders, would access the same
>> transaction
>> > log. Some of the logic used for implementing the shared storage for
>> the
>> > index segments will be reused.
>> > For serving queries, the non leader replicas (also of type ZERO)
>> update
>> > themselves from the shared storage directly. They behave mostly like
>> > PULL
>> > replicas (except the data doesn't come from the leader but from the
>> > shared
>> > storage), but can become leader because the shared storage is the
>> > "source
>> > of truth" and by reading all the data present there, any replica can
>> get
>> > itself up to date with all acknowledged updates to become leader
>> (there
>> > is
>> > protection so two replicas that think they're leader at the same
>> time do
>> > not overwrite each other, I can describe this on another occasion).
>> > Currently the (basic) approach is that a replica checks if it is up
>> to
>> > date as long as it's getting queries. If it is already up to date,
>> the
>> > cost
>> > of that check is a ZooKeeper node read. If it is not up to date, it
>> then
>> > fetches the updated content from the shared storage. Other strategies
>> > (check less often, check while not getting queries etc. are easy to
>> > implement). The updates are currently done asynchronously so do not
>> > delay
>> > the queries (that serve the "previous" content, like normal SolrCloud
>> > replication does).
>> >
>> > I'd be happy to discuss and explain this in more detail. Quickly
>> tomorrow
>> > then more once I've shared the source code branch and the design doc.
>> > I'd of course be happy to also follow up here with any questions, so
>> don't
>> > hesitate to ask!
>> >
>> > Ilan
>> >
>> >
>> >
>> > On Wed, Jan 17, 2024 at 8:30 PM Jason Gerlowski <ge...@gmail.com>
>> > wrote:
>> >
>> > > Hey Ilan,
>> > >
>> > > Thanks for putting together this writeup. I think I understand the
>> goal
>> > > conceptually, and it sounds like a good one for Solr! But I'm still
>> > having
>> > > trouble understanding how this all would actually work. So a few
>> > > questions, inline:
>> > >
>> > > > A fourth replica type called ZERO is introduced
>> > >
>> > > Why the name "Zero"? Is it conveying something about the design that
>> I'm
>> > > not picking up on?
>> > >
>> > > > At Collection creation time, it is possible to specify that the
>> > > collection exclusively uses replicas of type ZERO rather than being a
>> > > “normal” collection that uses NRT/TLOG/PULL.
>> > >
>> > > Am I correct in understanding this to mean that if "zero" is used, it
>> > must
>> > > be used for every replica in the collection? If so, it almost sounds
>> > like
>> > > this isn't a new type of replica but a new "collection type"
>> altogether?
>> > >
>> > > > This allows scaling compute (more queries, more indexing)
>> independently
>> > > of storage
>> > >
>> > > I think the biggest question I have is: how does the "compute" side of
>> > this
>> > > actually work?
>> > >
>> > > On the indexing side: what all happens in Solr before giving a
>> response
>> > > back to users? What happens on a commit? Are updates indexed only on
>> > the
>> > > leader (like TLOG/PULL) or on all replicas (like NRT), or some other
>> > > arrangement altogether?
>> > >
>> > > On the querying side: what situations cause index data to be pulled
>> from
>> > > the remote store?
>> > >
>> > > (These last questions might be a bit lengthy to get into via email,
>> but
>> > > they should probably be in the writeup? Not sure what's best
>> there...)
>> > >
>> > > Best,
>> > >
>> > > Jason
>> > >
>> > > On Sat, Jan 13, 2024 at 9:15 PM Ishan Chattopadhyaya <
>> > > ichattopadhyaya@gmail.com> wrote:
>> > >
>> > > > +1, thanks for the contribution Ilan! Looking forward to seeing this
>> > > coming
>> > > > to fruition.
>> > > >
>> > > > On Sun, 14 Jan 2024 at 03:40, Ilan Ginzburg <il...@gmail.com>
>> > wrote:
>> > > >
>> > > > > I have created SIP-20
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/SOLR/SIP-20%3A+Separation+of+Compute+and+Storage+in+SolrCloud
>> > > > >
>> > > > > In the next few days I will create a Jira + a branch that
>> implements
>> > > > > the SIP proposal and that includes documentation on how to
>> approach
>> > > > > that branch and what's in it.
>> > > > >
>> > > > > This proposed contribution is based on work done at Salesforce
>> these
>> > > > > last few years and currently running at scale in multiple regions.
>> > > > >
>> > > > > Thanks,
>> > > > > Ilan
>> > > > >
>> > > > >
>> ---------------------------------------------------------------------
>> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> > > > > For additional commands, e-mail: dev-help@solr.apache.org
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud
Posted by Ilan Ginzburg <il...@gmail.com>.
>
> To clarify - since you mentioned durability, I'm assuming the commit that
> happens on each update batch is a "hard commit"?
>
Yes.
If the user made an explicit hard-commit or had an autoCommit configured,
> would that be a no-op for ZERO collections?
>
Yes.
> How do soft-commits behave - I imagine they're handled similarly to
> whatever is done for PULL replicas currently?
>
Currently a hard commit is automatically added after the last document of a
batch. Not sure what would happen if that wasn't the case and a soft commit
was issued instead.
In any case the soft commit would not propagate to other ZERO replicas than
the leader that's for sure (I assume when you mention soft commit and PULL
you mean soft commit on another replica?). The soft commit will also not
write the files to the shared repository. So likely the batch would be
acknowledged to the client but would be lost if a failure hit the node
before the next (hard) commit (no transaction log, nodes are assumed
stateless, so Zero replicas skip recovery on node startup).
And that ZK read happens on each query? Or does the current strategy have
> it check "every X seconds" like how PULL/TLOG replicas check for updates?
>
(about a replica checking with a ZK read if it's up to date). In the
current implementation such a read is enqueued each time there's a query,
but basically until that read has happened then another query arrived,
there will not be a second read. So if for example a lot of queries arrive
in the same second before the read happens, there would be a single read.
Or if the cluster is loaded (many replicas get queries and enqueue such
reads from ZK), the reads will not be executed right away and no new read
will be enqueued until the previous one was done.
Spacing such reads by some minimum delay is not a big change.
Is that ZK check looking at the "shard term" stuff, or would Zero
> replicas/collections store some additional state in ZK to manage this?
It is not shard terms. Terms are being used when a follower replica does
not manage to get an indexing update from the leader. Given Zero replicas
do not send indexing updates to other replicas, terms are never increased
and are therefore not used for this type of replica.
The value that is read in ZK is basically the pointer to the current
metadata file for the shard in the [Backup]Repository (it's called
BackupRepository but the Zero implementation uses it as a general purpose
repository). When the shard in the repository changes, the shard metadata
file changes (you can think of it as listing the current stored commit
point in the repository). The shard ZooKeeper node references the current
valid shard metadata file in the repository. A node that has read that
value (then the repository metadata file then all the segments) can check
if the ZooKeeper node has changed, and if it has not, the node already has
the current version of the shard.
Ilan
On Mon, Jan 22, 2024 at 10:20 PM Jason Gerlowski <ge...@gmail.com>
wrote:
> Thanks for the detailed response! I've got a few follow-ups that I'll ask
> in line, but I think I've got the core idea now, thanks!
>
> > With the implementation in its current state, an indexing batch (as sent
> > by the client) is processed by the leader replica, committed to local
> disk
> > then written into the remote storage (S3) before SolrCloud acknowledges
> the
> > batch. Transaction log is not used and the commit is forced.
>
> To clarify - since you mentioned durability, I'm assuming the commit that
> happens on each update batch is a "hard commit"?
>
> If the user made an explicit hard-commit or had an autoCommit configured,
> would that be a no-op for ZERO collections?
>
> How do soft-commits behave - I imagine they're handled similarly to
> whatever is done for PULL replicas currently?
>
> > Currently the (basic) approach is that a replica checks if it is up to
> > date as long as it's getting queries. If it is already up to date, the
> cost
> > of that check is a ZooKeeper node read.
>
> And that ZK read happens on each query? Or does the current strategy have
> it check "every X seconds" like how PULL/TLOG replicas check for updates?
>
> Is that ZK check looking at the "shard term" stuff, or would Zero
> replicas/collections store some additional state in ZK to manage this?
>
> Best,
>
> Jason
>
> On Wed, Jan 17, 2024 at 8:05 PM Ilan Ginzburg <il...@gmail.com> wrote:
>
> > Hi Jason,
> >
> > Good questions!
> > Note there is a related question on the users list, see thread Node roles
> > vs SIP-20 Separation of Compute and Storage
> > <https://lists.apache.org/thread/ox7xbl1hd2j87ccvlyjho4kqqv2jnfmc>
> >
> > When I share the code (later this week in all likelihood) I will share a
> > detailed write up. I'm open to discuss/present/write more as needed (and
> > plan to attend Thursday 18/1 Meetup
> > <
> https://cwiki.apache.org/confluence/display/SOLR/2024-01-18+Meeting+notes>
> > to
> > present the SIP).
> >
> > To your questions:
> >
> > - The name "Zero" is because "Shared" was too broad (and already used)
> > and "Blob" was already used as well in Solr code, and we didn't find
> any
> > better name (but the name can be changed, there are a couple thousand
> > occurrences of it in different forms in the code, I know because
> "Zero"
> > is
> > not the name we use internally at Salesforce for that replica type,
> I've
> > renamed everything for sharing here). The reason for "Zero" is that
> > there
> > is only one instance (zero copies) in persistent storage for each
> shard,
> > and also this evoques the (longer term) option to have zero (i.e. no)
> > replicas for a shard on the cluster, pick a node, load and
> materialize a
> > replica when needed. But that's longer term.
> > - Your understanding is correct, currently it's a kind of new type of
> > collection given all replicas are ZERO (or none are) for a collection.
> > ZERO
> > could have been a new type of shard, as one could imagine different
> > shards
> > having different types (but shards do not have types), or why not also
> > allow a given shard to have ZERO replicas as well as other types of
> > replicas (this is not supported in the current implementation for
> > simplicity and we didn't really felt the need). If collectively we
> think
> > that PULL + ZERO do make sense, why not. PULL would then be fetching
> its
> > content from ZERO replicas rather than directly from the shared
> > storage. I
> > don't see ZERO coexisting with NRT or TLOG in a given shard though.
> > Currently as the implementation forces all replicas of all shards of a
> > "Zero" collection to be ZERO, there is a flag on that collection
> saying
> > it
> > is a Zero collection.
> > - How the compute behaves:
> > With the implementation in its current state, an indexing batch (as
> sent
> > by the client) is processed by the leader replica, committed to local
> > disk
> > then written into the remote storage (S3) before SolrCloud
> acknowledges
> > the
> > batch. Transaction log is not used and the commit is forced. This
> slows
> > down indexing, but a design guideline from the start was that the
> > SolrCloud
> > nodes are stateless (can restart with empty disk) to simplify managing
> > elasticity (and in a public cloud setting, Availability Zone
> failures).
> > The next evolution step of this code that we plan to start upstream
> once
> > the branch has been shared is to enable the transaction log for ZERO
> > replicas, and have indexing behave more like a normal SolrCloud:
> persist
> > the transaction log before replying to a client, do not force a
> commit.
> > The
> > transaction log would be changed to be on shared storage as well,
> with a
> > single copy for a shard, not one log per node/replica. All replicas of
> > the
> > shard, when they're becoming leaders, would access the same
> transaction
> > log. Some of the logic used for implementing the shared storage for
> the
> > index segments will be reused.
> > For serving queries, the non leader replicas (also of type ZERO)
> update
> > themselves from the shared storage directly. They behave mostly like
> > PULL
> > replicas (except the data doesn't come from the leader but from the
> > shared
> > storage), but can become leader because the shared storage is the
> > "source
> > of truth" and by reading all the data present there, any replica can
> get
> > itself up to date with all acknowledged updates to become leader
> (there
> > is
> > protection so two replicas that think they're leader at the same time
> do
> > not overwrite each other, I can describe this on another occasion).
> > Currently the (basic) approach is that a replica checks if it is up to
> > date as long as it's getting queries. If it is already up to date, the
> > cost
> > of that check is a ZooKeeper node read. If it is not up to date, it
> then
> > fetches the updated content from the shared storage. Other strategies
> > (check less often, check while not getting queries etc. are easy to
> > implement). The updates are currently done asynchronously so do not
> > delay
> > the queries (that serve the "previous" content, like normal SolrCloud
> > replication does).
> >
> > I'd be happy to discuss and explain this in more detail. Quickly tomorrow
> > then more once I've shared the source code branch and the design doc.
> > I'd of course be happy to also follow up here with any questions, so
> don't
> > hesitate to ask!
> >
> > Ilan
> >
> >
> >
> > On Wed, Jan 17, 2024 at 8:30 PM Jason Gerlowski <ge...@gmail.com>
> > wrote:
> >
> > > Hey Ilan,
> > >
> > > Thanks for putting together this writeup. I think I understand the
> goal
> > > conceptually, and it sounds like a good one for Solr! But I'm still
> > having
> > > trouble understanding how this all would actually work. So a few
> > > questions, inline:
> > >
> > > > A fourth replica type called ZERO is introduced
> > >
> > > Why the name "Zero"? Is it conveying something about the design that
> I'm
> > > not picking up on?
> > >
> > > > At Collection creation time, it is possible to specify that the
> > > collection exclusively uses replicas of type ZERO rather than being a
> > > “normal” collection that uses NRT/TLOG/PULL.
> > >
> > > Am I correct in understanding this to mean that if "zero" is used, it
> > must
> > > be used for every replica in the collection? If so, it almost sounds
> > like
> > > this isn't a new type of replica but a new "collection type"
> altogether?
> > >
> > > > This allows scaling compute (more queries, more indexing)
> independently
> > > of storage
> > >
> > > I think the biggest question I have is: how does the "compute" side of
> > this
> > > actually work?
> > >
> > > On the indexing side: what all happens in Solr before giving a response
> > > back to users? What happens on a commit? Are updates indexed only on
> > the
> > > leader (like TLOG/PULL) or on all replicas (like NRT), or some other
> > > arrangement altogether?
> > >
> > > On the querying side: what situations cause index data to be pulled
> from
> > > the remote store?
> > >
> > > (These last questions might be a bit lengthy to get into via email, but
> > > they should probably be in the writeup? Not sure what's best there...)
> > >
> > > Best,
> > >
> > > Jason
> > >
> > > On Sat, Jan 13, 2024 at 9:15 PM Ishan Chattopadhyaya <
> > > ichattopadhyaya@gmail.com> wrote:
> > >
> > > > +1, thanks for the contribution Ilan! Looking forward to seeing this
> > > coming
> > > > to fruition.
> > > >
> > > > On Sun, 14 Jan 2024 at 03:40, Ilan Ginzburg <il...@gmail.com>
> > wrote:
> > > >
> > > > > I have created SIP-20
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/SOLR/SIP-20%3A+Separation+of+Compute+and+Storage+in+SolrCloud
> > > > >
> > > > > In the next few days I will create a Jira + a branch that
> implements
> > > > > the SIP proposal and that includes documentation on how to approach
> > > > > that branch and what's in it.
> > > > >
> > > > > This proposed contribution is based on work done at Salesforce
> these
> > > > > last few years and currently running at scale in multiple regions.
> > > > >
> > > > > Thanks,
> > > > > Ilan
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud
Posted by Jason Gerlowski <ge...@gmail.com>.
Thanks for the detailed response! I've got a few follow-ups that I'll ask
in line, but I think I've got the core idea now, thanks!
> With the implementation in its current state, an indexing batch (as sent
> by the client) is processed by the leader replica, committed to local disk
> then written into the remote storage (S3) before SolrCloud acknowledges
the
> batch. Transaction log is not used and the commit is forced.
To clarify - since you mentioned durability, I'm assuming the commit that
happens on each update batch is a "hard commit"?
If the user made an explicit hard-commit or had an autoCommit configured,
would that be a no-op for ZERO collections?
How do soft-commits behave - I imagine they're handled similarly to
whatever is done for PULL replicas currently?
> Currently the (basic) approach is that a replica checks if it is up to
> date as long as it's getting queries. If it is already up to date, the
cost
> of that check is a ZooKeeper node read.
And that ZK read happens on each query? Or does the current strategy have
it check "every X seconds" like how PULL/TLOG replicas check for updates?
Is that ZK check looking at the "shard term" stuff, or would Zero
replicas/collections store some additional state in ZK to manage this?
Best,
Jason
On Wed, Jan 17, 2024 at 8:05 PM Ilan Ginzburg <il...@gmail.com> wrote:
> Hi Jason,
>
> Good questions!
> Note there is a related question on the users list, see thread Node roles
> vs SIP-20 Separation of Compute and Storage
> <https://lists.apache.org/thread/ox7xbl1hd2j87ccvlyjho4kqqv2jnfmc>
>
> When I share the code (later this week in all likelihood) I will share a
> detailed write up. I'm open to discuss/present/write more as needed (and
> plan to attend Thursday 18/1 Meetup
> <https://cwiki.apache.org/confluence/display/SOLR/2024-01-18+Meeting+notes>
> to
> present the SIP).
>
> To your questions:
>
> - The name "Zero" is because "Shared" was too broad (and already used)
> and "Blob" was already used as well in Solr code, and we didn't find any
> better name (but the name can be changed, there are a couple thousand
> occurrences of it in different forms in the code, I know because "Zero"
> is
> not the name we use internally at Salesforce for that replica type, I've
> renamed everything for sharing here). The reason for "Zero" is that
> there
> is only one instance (zero copies) in persistent storage for each shard,
> and also this evoques the (longer term) option to have zero (i.e. no)
> replicas for a shard on the cluster, pick a node, load and materialize a
> replica when needed. But that's longer term.
> - Your understanding is correct, currently it's a kind of new type of
> collection given all replicas are ZERO (or none are) for a collection.
> ZERO
> could have been a new type of shard, as one could imagine different
> shards
> having different types (but shards do not have types), or why not also
> allow a given shard to have ZERO replicas as well as other types of
> replicas (this is not supported in the current implementation for
> simplicity and we didn't really felt the need). If collectively we think
> that PULL + ZERO do make sense, why not. PULL would then be fetching its
> content from ZERO replicas rather than directly from the shared
> storage. I
> don't see ZERO coexisting with NRT or TLOG in a given shard though.
> Currently as the implementation forces all replicas of all shards of a
> "Zero" collection to be ZERO, there is a flag on that collection saying
> it
> is a Zero collection.
> - How the compute behaves:
> With the implementation in its current state, an indexing batch (as sent
> by the client) is processed by the leader replica, committed to local
> disk
> then written into the remote storage (S3) before SolrCloud acknowledges
> the
> batch. Transaction log is not used and the commit is forced. This slows
> down indexing, but a design guideline from the start was that the
> SolrCloud
> nodes are stateless (can restart with empty disk) to simplify managing
> elasticity (and in a public cloud setting, Availability Zone failures).
> The next evolution step of this code that we plan to start upstream once
> the branch has been shared is to enable the transaction log for ZERO
> replicas, and have indexing behave more like a normal SolrCloud: persist
> the transaction log before replying to a client, do not force a commit.
> The
> transaction log would be changed to be on shared storage as well, with a
> single copy for a shard, not one log per node/replica. All replicas of
> the
> shard, when they're becoming leaders, would access the same transaction
> log. Some of the logic used for implementing the shared storage for the
> index segments will be reused.
> For serving queries, the non leader replicas (also of type ZERO) update
> themselves from the shared storage directly. They behave mostly like
> PULL
> replicas (except the data doesn't come from the leader but from the
> shared
> storage), but can become leader because the shared storage is the
> "source
> of truth" and by reading all the data present there, any replica can get
> itself up to date with all acknowledged updates to become leader (there
> is
> protection so two replicas that think they're leader at the same time do
> not overwrite each other, I can describe this on another occasion).
> Currently the (basic) approach is that a replica checks if it is up to
> date as long as it's getting queries. If it is already up to date, the
> cost
> of that check is a ZooKeeper node read. If it is not up to date, it then
> fetches the updated content from the shared storage. Other strategies
> (check less often, check while not getting queries etc. are easy to
> implement). The updates are currently done asynchronously so do not
> delay
> the queries (that serve the "previous" content, like normal SolrCloud
> replication does).
>
> I'd be happy to discuss and explain this in more detail. Quickly tomorrow
> then more once I've shared the source code branch and the design doc.
> I'd of course be happy to also follow up here with any questions, so don't
> hesitate to ask!
>
> Ilan
>
>
>
> On Wed, Jan 17, 2024 at 8:30 PM Jason Gerlowski <ge...@gmail.com>
> wrote:
>
> > Hey Ilan,
> >
> > Thanks for putting together this writeup. I think I understand the goal
> > conceptually, and it sounds like a good one for Solr! But I'm still
> having
> > trouble understanding how this all would actually work. So a few
> > questions, inline:
> >
> > > A fourth replica type called ZERO is introduced
> >
> > Why the name "Zero"? Is it conveying something about the design that I'm
> > not picking up on?
> >
> > > At Collection creation time, it is possible to specify that the
> > collection exclusively uses replicas of type ZERO rather than being a
> > “normal” collection that uses NRT/TLOG/PULL.
> >
> > Am I correct in understanding this to mean that if "zero" is used, it
> must
> > be used for every replica in the collection? If so, it almost sounds
> like
> > this isn't a new type of replica but a new "collection type" altogether?
> >
> > > This allows scaling compute (more queries, more indexing) independently
> > of storage
> >
> > I think the biggest question I have is: how does the "compute" side of
> this
> > actually work?
> >
> > On the indexing side: what all happens in Solr before giving a response
> > back to users? What happens on a commit? Are updates indexed only on
> the
> > leader (like TLOG/PULL) or on all replicas (like NRT), or some other
> > arrangement altogether?
> >
> > On the querying side: what situations cause index data to be pulled from
> > the remote store?
> >
> > (These last questions might be a bit lengthy to get into via email, but
> > they should probably be in the writeup? Not sure what's best there...)
> >
> > Best,
> >
> > Jason
> >
> > On Sat, Jan 13, 2024 at 9:15 PM Ishan Chattopadhyaya <
> > ichattopadhyaya@gmail.com> wrote:
> >
> > > +1, thanks for the contribution Ilan! Looking forward to seeing this
> > coming
> > > to fruition.
> > >
> > > On Sun, 14 Jan 2024 at 03:40, Ilan Ginzburg <il...@gmail.com>
> wrote:
> > >
> > > > I have created SIP-20
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/SOLR/SIP-20%3A+Separation+of+Compute+and+Storage+in+SolrCloud
> > > >
> > > > In the next few days I will create a Jira + a branch that implements
> > > > the SIP proposal and that includes documentation on how to approach
> > > > that branch and what's in it.
> > > >
> > > > This proposed contribution is based on work done at Salesforce these
> > > > last few years and currently running at scale in multiple regions.
> > > >
> > > > Thanks,
> > > > Ilan
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud
Posted by Ilan Ginzburg <il...@gmail.com>.
Hi Jason,
Good questions!
Note there is a related question on the users list, see thread Node roles
vs SIP-20 Separation of Compute and Storage
<https://lists.apache.org/thread/ox7xbl1hd2j87ccvlyjho4kqqv2jnfmc>
When I share the code (later this week in all likelihood) I will share a
detailed write up. I'm open to discuss/present/write more as needed (and
plan to attend Thursday 18/1 Meetup
<https://cwiki.apache.org/confluence/display/SOLR/2024-01-18+Meeting+notes> to
present the SIP).
To your questions:
- The name "Zero" is because "Shared" was too broad (and already used)
and "Blob" was already used as well in Solr code, and we didn't find any
better name (but the name can be changed, there are a couple thousand
occurrences of it in different forms in the code, I know because "Zero" is
not the name we use internally at Salesforce for that replica type, I've
renamed everything for sharing here). The reason for "Zero" is that there
is only one instance (zero copies) in persistent storage for each shard,
and also this evoques the (longer term) option to have zero (i.e. no)
replicas for a shard on the cluster, pick a node, load and materialize a
replica when needed. But that's longer term.
- Your understanding is correct, currently it's a kind of new type of
collection given all replicas are ZERO (or none are) for a collection. ZERO
could have been a new type of shard, as one could imagine different shards
having different types (but shards do not have types), or why not also
allow a given shard to have ZERO replicas as well as other types of
replicas (this is not supported in the current implementation for
simplicity and we didn't really felt the need). If collectively we think
that PULL + ZERO do make sense, why not. PULL would then be fetching its
content from ZERO replicas rather than directly from the shared storage. I
don't see ZERO coexisting with NRT or TLOG in a given shard though.
Currently as the implementation forces all replicas of all shards of a
"Zero" collection to be ZERO, there is a flag on that collection saying it
is a Zero collection.
- How the compute behaves:
With the implementation in its current state, an indexing batch (as sent
by the client) is processed by the leader replica, committed to local disk
then written into the remote storage (S3) before SolrCloud acknowledges the
batch. Transaction log is not used and the commit is forced. This slows
down indexing, but a design guideline from the start was that the SolrCloud
nodes are stateless (can restart with empty disk) to simplify managing
elasticity (and in a public cloud setting, Availability Zone failures).
The next evolution step of this code that we plan to start upstream once
the branch has been shared is to enable the transaction log for ZERO
replicas, and have indexing behave more like a normal SolrCloud: persist
the transaction log before replying to a client, do not force a commit. The
transaction log would be changed to be on shared storage as well, with a
single copy for a shard, not one log per node/replica. All replicas of the
shard, when they're becoming leaders, would access the same transaction
log. Some of the logic used for implementing the shared storage for the
index segments will be reused.
For serving queries, the non leader replicas (also of type ZERO) update
themselves from the shared storage directly. They behave mostly like PULL
replicas (except the data doesn't come from the leader but from the shared
storage), but can become leader because the shared storage is the "source
of truth" and by reading all the data present there, any replica can get
itself up to date with all acknowledged updates to become leader (there is
protection so two replicas that think they're leader at the same time do
not overwrite each other, I can describe this on another occasion).
Currently the (basic) approach is that a replica checks if it is up to
date as long as it's getting queries. If it is already up to date, the cost
of that check is a ZooKeeper node read. If it is not up to date, it then
fetches the updated content from the shared storage. Other strategies
(check less often, check while not getting queries etc. are easy to
implement). The updates are currently done asynchronously so do not delay
the queries (that serve the "previous" content, like normal SolrCloud
replication does).
I'd be happy to discuss and explain this in more detail. Quickly tomorrow
then more once I've shared the source code branch and the design doc.
I'd of course be happy to also follow up here with any questions, so don't
hesitate to ask!
Ilan
On Wed, Jan 17, 2024 at 8:30 PM Jason Gerlowski <ge...@gmail.com>
wrote:
> Hey Ilan,
>
> Thanks for putting together this writeup. I think I understand the goal
> conceptually, and it sounds like a good one for Solr! But I'm still having
> trouble understanding how this all would actually work. So a few
> questions, inline:
>
> > A fourth replica type called ZERO is introduced
>
> Why the name "Zero"? Is it conveying something about the design that I'm
> not picking up on?
>
> > At Collection creation time, it is possible to specify that the
> collection exclusively uses replicas of type ZERO rather than being a
> “normal” collection that uses NRT/TLOG/PULL.
>
> Am I correct in understanding this to mean that if "zero" is used, it must
> be used for every replica in the collection? If so, it almost sounds like
> this isn't a new type of replica but a new "collection type" altogether?
>
> > This allows scaling compute (more queries, more indexing) independently
> of storage
>
> I think the biggest question I have is: how does the "compute" side of this
> actually work?
>
> On the indexing side: what all happens in Solr before giving a response
> back to users? What happens on a commit? Are updates indexed only on the
> leader (like TLOG/PULL) or on all replicas (like NRT), or some other
> arrangement altogether?
>
> On the querying side: what situations cause index data to be pulled from
> the remote store?
>
> (These last questions might be a bit lengthy to get into via email, but
> they should probably be in the writeup? Not sure what's best there...)
>
> Best,
>
> Jason
>
> On Sat, Jan 13, 2024 at 9:15 PM Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
> > +1, thanks for the contribution Ilan! Looking forward to seeing this
> coming
> > to fruition.
> >
> > On Sun, 14 Jan 2024 at 03:40, Ilan Ginzburg <il...@gmail.com> wrote:
> >
> > > I have created SIP-20
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/SOLR/SIP-20%3A+Separation+of+Compute+and+Storage+in+SolrCloud
> > >
> > > In the next few days I will create a Jira + a branch that implements
> > > the SIP proposal and that includes documentation on how to approach
> > > that branch and what's in it.
> > >
> > > This proposed contribution is based on work done at Salesforce these
> > > last few years and currently running at scale in multiple regions.
> > >
> > > Thanks,
> > > Ilan
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
>
Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud
Posted by Jason Gerlowski <ge...@gmail.com>.
Hey Ilan,
Thanks for putting together this writeup. I think I understand the goal
conceptually, and it sounds like a good one for Solr! But I'm still having
trouble understanding how this all would actually work. So a few
questions, inline:
> A fourth replica type called ZERO is introduced
Why the name "Zero"? Is it conveying something about the design that I'm
not picking up on?
> At Collection creation time, it is possible to specify that the
collection exclusively uses replicas of type ZERO rather than being a
“normal” collection that uses NRT/TLOG/PULL.
Am I correct in understanding this to mean that if "zero" is used, it must
be used for every replica in the collection? If so, it almost sounds like
this isn't a new type of replica but a new "collection type" altogether?
> This allows scaling compute (more queries, more indexing) independently
of storage
I think the biggest question I have is: how does the "compute" side of this
actually work?
On the indexing side: what all happens in Solr before giving a response
back to users? What happens on a commit? Are updates indexed only on the
leader (like TLOG/PULL) or on all replicas (like NRT), or some other
arrangement altogether?
On the querying side: what situations cause index data to be pulled from
the remote store?
(These last questions might be a bit lengthy to get into via email, but
they should probably be in the writeup? Not sure what's best there...)
Best,
Jason
On Sat, Jan 13, 2024 at 9:15 PM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:
> +1, thanks for the contribution Ilan! Looking forward to seeing this coming
> to fruition.
>
> On Sun, 14 Jan 2024 at 03:40, Ilan Ginzburg <il...@gmail.com> wrote:
>
> > I have created SIP-20
> >
> >
> https://cwiki.apache.org/confluence/display/SOLR/SIP-20%3A+Separation+of+Compute+and+Storage+in+SolrCloud
> >
> > In the next few days I will create a Jira + a branch that implements
> > the SIP proposal and that includes documentation on how to approach
> > that branch and what's in it.
> >
> > This proposed contribution is based on work done at Salesforce these
> > last few years and currently running at scale in multiple regions.
> >
> > Thanks,
> > Ilan
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
+1, thanks for the contribution Ilan! Looking forward to seeing this coming
to fruition.
On Sun, 14 Jan 2024 at 03:40, Ilan Ginzburg <il...@gmail.com> wrote:
> I have created SIP-20
>
> https://cwiki.apache.org/confluence/display/SOLR/SIP-20%3A+Separation+of+Compute+and+Storage+in+SolrCloud
>
> In the next few days I will create a Jira + a branch that implements
> the SIP proposal and that includes documentation on how to approach
> that branch and what's in it.
>
> This proposed contribution is based on work done at Salesforce these
> last few years and currently running at scale in multiple regions.
>
> Thanks,
> Ilan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>