You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Nadav Har'El <ny...@scylladb.com> on 2018/02/21 14:26:39 UTC
Issues with Materialized-Views updates during a cluster change?
Hi, I was trying to understand how view tables are updated during a period
of range movements, namely bootstrapping of a new node or decommissioning
one of the nodes. In particular, during the period of data streaming, we
can have a new replica on a "pending node" to which we also need to send
the view update.
I looked at the mutateMV() code, and think I spotted two issues with it,
and I wonder if I'm missing something or these are real problems:
1. It seems that for example when RF=3, each one of the three base replicas
will send a view update to the fourth "pending node". While this is not
wrong, it's also inefficient - why send three copies of the same update?
Wouldn't it be more efficient that just one of the base replicas - the one
which eventually will be paired with the pending node - should send the
updates to it? Is there a problem with such a scheme?
2. There's an optimization that when we're lucky enough that the paired
view replica is the same as this base replica, mutateMV doesn't use the
normal view-mutation-sending code (wrapViewBatchResponseHandler) and just
writes the mutation locally. In particular, in this case we do NOT write to
the pending node (unless I'm missing something). But, sometimes all
replicas will be paired with themselves - this can happen for example when
number of nodes is equal to RF, or when the base and view table have the
same partition keys (but different clustering keys). In this case, it seems
the pending node will not be written at all... Isn't this a bug?
The strange thing about issue 2 is that this code used to be correct (at
least according to my understanding...) - it used to avoid this
optimization if pendingNodes was not empty. But then this was changed in
commit 12103653f31. Why?
https://issues.apache.org/jira/browse/CASSANDRA-13069 contains an
explanation to that change:
"I also removed the pendingEndpoints.isEmpty() condition to skip the
batchlog for local mutations, since this was a pre-CASSANDRA-10674 leftover
when ViewUtils.getViewNaturalEndpoint returned the local address to force
non-paired replicas to be written to the batchlog." (Paulo Motta, 21/Dec/16)
But I don't understand this explanation... Being paired with yourself is
not only a "trick", but also something which really happens (by chance or
in some cases as I showed above, always), and needs to be handled
correctly, even if the cluster grows. If none of the base replicas will
send the view update to the pending node, it will end up missing this
update...
Thanks,
Nadav.
--
Nadav Har'El
nyh@scylladb.com
Re: Issues with Materialized-Views updates during a cluster change?
Posted by Nadav Har'El <ny...@scylladb.com>.
On Thu, Feb 22, 2018 at 12:54 AM, Paulo Motta <pa...@gmail.com>
wrote:
> > 1. It seems that for example when RF=3, each one of the three base
> replicas will send a view update to the fourth "pending node". While this
> is not wrong, it's also inefficient - why send three copies of the same
> update? Wouldn't it be more efficient that just one of the base replicas -
> the one which eventually will be paired with the pending node - should send
> the updates to it? Is there a problem with such a scheme?
>
> This optimization can be done when there's a single pending range per
> view replica set, but when there are multiple pending ranges and there
> are failures, it's possible that the paired view replica changes what
> can lead to missing updates. For instance, see the following scenario:
> - There are 2 pending ranges A' and B'.
>
Is this a realistic case when Cassandra (unless I'm missing something) is
limited to adding or removing a single node at a time?
How/when would we have two pending nodes for a single view partition?
I'm sure this can happen under some sort of generic range movement of some
sort (how does one initiate such movement, and why), but will it happen
under "normal" conditions of node bootstrap or decomission of a single node?
> - Base replica A sends update to pending-paired view replica A'.
> - Base replica B is down, so pending-paired view replica B' does not get
> update.
> - Range movement A' fails and B' succeeds.
> - B' becomes A new paired view replica.
> - A will be out of sync with B'
>
> Furthermore we would need to cache the ring state after the range
> movement is completed to be able to compute the pending-paired view
> replica but we don't have this info easily available currently, so it
> seems that it would not be a trivial change but perhaps worth pursuing
> in the single pending range case.
>
Yes, it seems it will not be trivial. But if this is the common case in
common
operations such as node addition or removal, it may significantly reduce
(from
RF*2 to RF+1) the number of view updates being sent around, and avoid
MV update performance degredation during the streaming process.
>
> > 2. There's an optimization that when we're lucky enough that the paired
> view replica is the same as this base replica, mutateMV doesn't use the
> normal view-mutation-sending code (wrapViewBatchResponseHandler) and just
> writes the mutation locally. In particular, in this case we do NOT write to
> the pending node (unless I'm missing something). But, sometimes all
> replicas will be paired with themselves - this can happen for example when
> number of nodes is equal to RF, or when the base and view table have the
> same partition keys (but different clustering keys). In this case, it seems
> the pending node will not be written at all... Isn't this a bug?
>
> Good catch! This indeed seems to be a regression caused by
> CASSANDRA-13069, so I created CASSANDRA-14251 to restore the correct
> behavior.
>
Good! Thanks.
Nadav.
RE: Mentor
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
That one is JIRA 14264:
https://issues.apache.org/jira/browse/CASSANDRA-14264
I won't be writing that one anytime soon but if it existed I would be
reading it. If the guru's would throw some text in the JIRA as they have it
handy, myself or someone else will take it from there.
Kenneth Brotman
-----Original Message-----
From: Darshan Sreenivasamurthy [mailto:darshants.darshan1@gmail.com]
Sent: Monday, February 26, 2018 11:33 AM
To: dev@cassandra.apache.org
Subject: Re: Mentor
+ 1
> On Feb 26, 2018, at 11:24 AM, Kenneth Brotman
<ke...@yahoo.com.INVALID> wrote:
>
> how about a nice Quick Tour Document for dev's that want to get oriented
on the code efficiently.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Mentor
Posted by Akash Gangil <ak...@gmail.com>.
There isn't really a code walkthrough if that's what you are looking for.
However, a good start is to to check out the code. Here are the steps:
https://github.com/apache/cassandra/tree/trunk/doc/source/development
Next, would be to try and see if any of the low hanging fruits interest
you.
https://issues.apache.org/jira/browse/CASSANDRA-14228?jql=labels%20%3D%20lhf
On Mon, Feb 26, 2018 at 11:32 AM, Darshan Sreenivasamurthy <
darshants.darshan1@gmail.com> wrote:
> + 1
>
>
>
> > On Feb 26, 2018, at 11:24 AM, Kenneth Brotman
> <ke...@yahoo.com.INVALID> wrote:
> >
> > how about a nice Quick Tour Document for dev's that want to get oriented
> on the code efficiently.
>
>
--
Akash
Re: Mentor
Posted by Darshan Sreenivasamurthy <da...@gmail.com>.
+ 1
> On Feb 26, 2018, at 11:24 AM, Kenneth Brotman <ke...@yahoo.com.INVALID> wrote:
>
> how about a nice Quick Tour Document for dev's that want to get oriented on the code efficiently.
RE: Mentor
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
Yeah, how about a nice Quick Tour Document for dev's that want to get oriented on the code efficiently. I would like that too.
Kenneth Brotman
-----Original Message-----
From: Sumant Sahney [mailto:sahneysumant@gmail.com]
Sent: Monday, February 26, 2018 11:12 AM
To: dev@cassandra.apache.org
Subject: Mentor
Hi,
I am new to Open Source.
I am really confused on how to get started with open source. I would really appreciate if someone can help me through the process and help me get started.
Can someone please share the code documentation with me as well. Which I can refer to get an idea about all the modules.
Thanks,
Sumant
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Mentor
Posted by Sumant Sahney <sa...@gmail.com>.
Hi,
I am new to Open Source.
I am really confused on how to get started with open source. I would really appreciate if someone can help me through the process and help me get started.
Can someone please share the code documentation with me as well. Which I can refer to get an idea about all the modules.
Thanks,
Sumant
Re: Issues with Materialized-Views updates during a cluster change?
Posted by Nadav Har'El <ny...@scylladb.com>.
On Mon, Feb 26, 2018 at 2:54 PM, Nadav Har'El <ny...@scylladb.com> wrote:
> On Fri, Feb 23, 2018 at 12:49 AM, Paulo Motta <pa...@gmail.com>
> wrote:
>
>> > Yes, it seems it will not be trivial. But if this is the common case in
>> common operations such as node addition or removal, it may significantly
>> reduce
>> (from RF*2 to RF+1) the number of view updates being sent around, and
>> avoid
>> MV update performance degredation during the streaming process.
>>
>> Agreed, we should definitely look into making this optimization, but
>> just was never done before due to other priorities, please open a
>> ticket for it.
>
>
> Ok, I will, though I'm not sure I understood all the caveats you mentioned
> so you may need to edit the ticket later to add them.
>
I created https://issues.apache.org/jira/browse/CASSANDRA-14262.
Also put there a link for this thread, instead of trying to paraphrase your
arguments
of when it's not enough that just one base replica sends an update to the
pending
view replica.
Nadav.
Re: Issues with Materialized-Views updates during a cluster change?
Posted by Nadav Har'El <ny...@scylladb.com>.
On Mon, Feb 26, 2018 at 2:54 PM, Nadav Har'El <ny...@scylladb.com> wrote:
> When there are permanent inconsistencies though (when the base is
>
>> consistent and the view has extraneous rows), it doesn't really matter
>> if the inconsistency is present on a subset or all view replicas,
>> since the inconsistency is already visible to clients. The only way to
>> fix permanent inconsistencies currently is to drop and re-create the
>> view. CASSANDRA-10346 was created to address this.
>>
>
> So this is why I asked why the view repair, which you suggested in the
> release notes, will help in this case.
>
> I'm also worried that the fact *each* replica sends to the pending node
> (and not the paired replica) will mean that we are more likely to create
> these inconsistencies - if two base replicas have different values, *both*
> values will be sent to the pending view replica, and create an
> inconsistency
> there that cannot be fixed.
>
I think I missed one crucial point, which allows the "each replica sends to
the
same pending view replica" to actually work. The point is that the view
replica keeps not just the view rows, but also (shadowable...) tombstones
for rows that have previously been deleted. So as long as one of the base
replicas had an old value changed and managed to send the tombstone to the
view replica, this old row will be deleted from the view - even if one of
the
base table still has it.
Nadav.
Re: Issues with Materialized-Views updates during a cluster change?
Posted by Nadav Har'El <ny...@scylladb.com>.
On Fri, Feb 23, 2018 at 12:49 AM, Paulo Motta <pa...@gmail.com>
wrote:
> > Is this a realistic case when Cassandra (unless I'm missing something) is
> limited to adding or removing a single node at a time? I'm sure this
> can happen under some sort of generic range movement of some
> sort (how does one initiate such movement, and why), but will it happen
> under "normal" conditions of node bootstrap or decomission of a single
> node?
>
> It's possible to make simultaneous range movements when either
> {{-Dcassandra.consistent.range.movement=false}}(CASSANDRA-7069) or
> {{-Dcassandra.consistent.simultaneousmoves.allow=true}}
> (CASSANDRA-11005) are specified.
>
> In any case, I'm not saying it's not possible, just that we cannot
> apply this optimization when there are simultaneous range movements in
> the same rack.
>
> > How/when would we have two pending nodes for a single view partition?
>
> Actually I meant if there are multiple range movements going on in the
> same rack, not exactly in the same partition.
>
But the code we're discussing now, in mutateMV, isn't it about sending just
one mutation, of a single partition in the view table? So don't we only care
which node (just one? can it be more than one?) this partition will move to?
>
> > Yes, it seems it will not be trivial. But if this is the common case in
> common operations such as node addition or removal, it may significantly
> reduce
> (from RF*2 to RF+1) the number of view updates being sent around, and avoid
> MV update performance degredation during the streaming process.
>
> Agreed, we should definitely look into making this optimization, but
> just was never done before due to other priorities, please open a
> ticket for it.
Ok, I will, though I'm not sure I understood all the caveats you mentioned
so you may need to edit the ticket later to add them.
> There's a similar optimization that can be done for
> view batchlog replays - right now the view update is sent to all
> replicas during batchlog replay, but we could simplify it and also
> send only to the paired view replicas.
>
> > Is it actually possible to repair *only* a view, not its base table? If
> you repair a view table which has an inconsistency, namely one view row in
> one replica and a different view row in another replica, won't the repair
> just cause both versions to be kept, which is wrong?
>
> ...
>
> When there are permanent inconsistencies though (when the base is
> consistent and the view has extraneous rows), it doesn't really matter
> if the inconsistency is present on a subset or all view replicas,
> since the inconsistency is already visible to clients. The only way to
> fix permanent inconsistencies currently is to drop and re-create the
> view. CASSANDRA-10346 was created to address this.
>
So this is why I asked why the view repair, which you suggested in the
release notes, will help in this case.
I'm also worried that the fact *each* replica sends to the pending node
(and not the paired replica) will mean that we are more likely to create
these inconsistencies - if two base replicas have different values, *both*
values will be sent to the pending view replica, and create an inconsistency
there that cannot be fixed.
>
> If you have more comments about CASSANDRA-14251 would you mind adding
> them to the ticket itself so the discussion is registered on the
> relevant JIRA?
>
I think most of the issues I raised later are not really part of 14251 but
different issues, I'll see what I can express clearly enough to become new
JIRA issues and submit them.
Re: Issues with Materialized-Views updates during a cluster change?
Posted by Paulo Motta <pa...@gmail.com>.
> Is this a realistic case when Cassandra (unless I'm missing something) is
limited to adding or removing a single node at a time? I'm sure this
can happen under some sort of generic range movement of some
sort (how does one initiate such movement, and why), but will it happen
under "normal" conditions of node bootstrap or decomission of a single node?
It's possible to make simultaneous range movements when either
{{-Dcassandra.consistent.range.movement=false}}(CASSANDRA-7069) or
{{-Dcassandra.consistent.simultaneousmoves.allow=true}}
(CASSANDRA-11005) are specified.
In any case, I'm not saying it's not possible, just that we cannot
apply this optimization when there are simultaneous range movements in
the same rack.
> How/when would we have two pending nodes for a single view partition?
Actually I meant if there are multiple range movements going on in the
same rack, not exactly in the same partition.
> Yes, it seems it will not be trivial. But if this is the common case in
common operations such as node addition or removal, it may significantly reduce
(from RF*2 to RF+1) the number of view updates being sent around, and avoid
MV update performance degredation during the streaming process.
Agreed, we should definitely look into making this optimization, but
just was never done before due to other priorities, please open a
ticket for it. There's a similar optimization that can be done for
view batchlog replays - right now the view update is sent to all
replicas during batchlog replay, but we could simplify it and also
send only to the paired view replicas.
> Is it actually possible to repair *only* a view, not its base table? If you repair a view table which has an inconsistency, namely one view row in one replica and a different view row in another replica, won't the repair just cause both versions to be kept, which is wrong?
It's possible to repair either the base table or the views. Normally
you will want to repair only the base tables, but sometimes you will
want to repair the views too, for instance, after a node is replaced -
just like you do it with ordinary tables, since the node may have
streamed from an inconsistent view replica.
In this particular case, repairing the base table alone won't help
because the base table can already be in sync, so it's necessary to
repair the view to ensure missed updates during range movements are
propagated to all replicas.
In fact repairing only the views without repairing the base table
beforehand may propagate temporary inconsistencies if stale views were
already garbage collected on a subset of the replicas, so I will
update the notice to state that repair must be run on the base table
(to fix temporary inconsistencies) and then on the views.
When there are permanent inconsistencies though (when the base is
consistent and the view has extraneous rows), it doesn't really matter
if the inconsistency is present on a subset or all view replicas,
since the inconsistency is already visible to clients. The only way to
fix permanent inconsistencies currently is to drop and re-create the
view. CASSANDRA-10346 was created to address this.
If you have more comments about CASSANDRA-14251 would you mind adding
them to the ticket itself so the discussion is registered on the
relevant JIRA?
2018-02-22 7:53 GMT-03:00 Nadav Har'El <ny...@scylladb.com>:
> On Thu, Feb 22, 2018 at 12:54 AM, Paulo Motta <pa...@gmail.com>
> wrote:
>
>>
>> Good catch! This indeed seems to be a regression caused by
>> CASSANDRA-13069, so I created CASSANDRA-14251 to restore the correct
>> behavior.
>>
>
> I have a question about your patch
> https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-14251
> In the NEWS.txt you say that users "should run repair on the views". Is it
> actually possible to repair *only* a view,
> not its base table? If you repair a view table which has an inconsistency,
> namely one view row in one replica and a
> different view row in another replica, won't the repair just cause both
> versions to be kept, which is wrong?
>
> Nadav.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: Issues with Materialized-Views updates during a cluster change?
Posted by Nadav Har'El <ny...@scylladb.com>.
On Thu, Feb 22, 2018 at 12:54 AM, Paulo Motta <pa...@gmail.com>
wrote:
>
> Good catch! This indeed seems to be a regression caused by
> CASSANDRA-13069, so I created CASSANDRA-14251 to restore the correct
> behavior.
>
I have a question about your patch
https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-14251
In the NEWS.txt you say that users "should run repair on the views". Is it
actually possible to repair *only* a view,
not its base table? If you repair a view table which has an inconsistency,
namely one view row in one replica and a
different view row in another replica, won't the repair just cause both
versions to be kept, which is wrong?
Nadav.
Re: Issues with Materialized-Views updates during a cluster change?
Posted by Paulo Motta <pa...@gmail.com>.
> 1. It seems that for example when RF=3, each one of the three base replicas will send a view update to the fourth "pending node". While this is not wrong, it's also inefficient - why send three copies of the same update? Wouldn't it be more efficient that just one of the base replicas - the one which eventually will be paired with the pending node - should send the updates to it? Is there a problem with such a scheme?
This optimization can be done when there's a single pending range per
view replica set, but when there are multiple pending ranges and there
are failures, it's possible that the paired view replica changes what
can lead to missing updates. For instance, see the following scenario:
- There are 2 pending ranges A' and B'.
- Base replica A sends update to pending-paired view replica A'.
- Base replica B is down, so pending-paired view replica B' does not get update.
- Range movement A' fails and B' succeeds.
- B' becomes A new paired view replica.
- A will be out of sync with B'
Furthermore we would need to cache the ring state after the range
movement is completed to be able to compute the pending-paired view
replica but we don't have this info easily available currently, so it
seems that it would not be a trivial change but perhaps worth pursuing
in the single pending range case.
> 2. There's an optimization that when we're lucky enough that the paired view replica is the same as this base replica, mutateMV doesn't use the normal view-mutation-sending code (wrapViewBatchResponseHandler) and just writes the mutation locally. In particular, in this case we do NOT write to the pending node (unless I'm missing something). But, sometimes all replicas will be paired with themselves - this can happen for example when number of nodes is equal to RF, or when the base and view table have the same partition keys (but different clustering keys). In this case, it seems the pending node will not be written at all... Isn't this a bug?
Good catch! This indeed seems to be a regression caused by
CASSANDRA-13069, so I created CASSANDRA-14251 to restore the correct
behavior.
bq. Being paired with yourself is not only a "trick", but also
something which really happens (by chance or in some cases as I showed
above, always), and needs to be handled correctly, even if the cluster
grows. If none of the base replicas will send the view update to the
pending node, it will end up missing this update...
Exactly, I only considered the case where the local address was used
as a marker to indicate there was no paired endpoint, and
brainfarted/missed the more important case when the local node is the
paired endpoint and there is a pending endpoint. Unfortunately this
overlook was not catch by any tests, so I will add one on
CASSANDRA-14251.
2018-02-21 11:26 GMT-03:00 Nadav Har'El <ny...@scylladb.com>:
> Hi, I was trying to understand how view tables are updated during a period
> of range movements, namely bootstrapping of a new node or decommissioning
> one of the nodes. In particular, during the period of data streaming, we can
> have a new replica on a "pending node" to which we also need to send the
> view update.
>
> I looked at the mutateMV() code, and think I spotted two issues with it, and
> I wonder if I'm missing something or these are real problems:
>
> 1. It seems that for example when RF=3, each one of the three base replicas
> will send a view update to the fourth "pending node". While this is not
> wrong, it's also inefficient - why send three copies of the same update?
> Wouldn't it be more efficient that just one of the base replicas - the one
> which eventually will be paired with the pending node - should send the
> updates to it? Is there a problem with such a scheme?
>
> 2. There's an optimization that when we're lucky enough that the paired view
> replica is the same as this base replica, mutateMV doesn't use the normal
> view-mutation-sending code (wrapViewBatchResponseHandler) and just writes
> the mutation locally. In particular, in this case we do NOT write to the
> pending node (unless I'm missing something). But, sometimes all replicas
> will be paired with themselves - this can happen for example when number of
> nodes is equal to RF, or when the base and view table have the same
> partition keys (but different clustering keys). In this case, it seems the
> pending node will not be written at all... Isn't this a bug?
>
> The strange thing about issue 2 is that this code used to be correct (at
> least according to my understanding...) - it used to avoid this optimization
> if pendingNodes was not empty. But then this was changed in commit
> 12103653f31. Why?
> https://issues.apache.org/jira/browse/CASSANDRA-13069 contains an
> explanation to that change:
> "I also removed the pendingEndpoints.isEmpty() condition to skip the
> batchlog for local mutations, since this was a pre-CASSANDRA-10674 leftover
> when ViewUtils.getViewNaturalEndpoint returned the local address to force
> non-paired replicas to be written to the batchlog." (Paulo Motta, 21/Dec/16)
>
> But I don't understand this explanation... Being paired with yourself is not
> only a "trick", but also something which really happens (by chance or in
> some cases as I showed above, always), and needs to be handled correctly,
> even if the cluster grows. If none of the base replicas will send the view
> update to the pending node, it will end up missing this update...
>
> Thanks,
> Nadav.
>
>
> --
> Nadav Har'El
> nyh@scylladb.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org