You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Saijal Chauhan <sa...@goevive.com> on 2020/07/27 16:01:23 UTC
Recreating materialized views in cassandra
Hi,
We are using Cassandra 3.0.13
We have the following datacenters:
- DC1 with 7 Cassandra nodes with RF:3
- DC2 with 2 Cassandra nodes with RF:2
- DC3 with 2 Cassandra nodes with RF:2
We are facing data inconsistency issues between base tables and
materialized views.
The only solution to this problem seems to be the creation of new
materialized views and dropping the old views.
We are planning to recreate 4 materialized views, 2 belonging to the same
base table.
The size of each base table ranges up to 4 to 5GB.
What are all the possible scenarios that we should be watching out for in a
production environment?
Could there be any downtime in the Cassandra cluster while creating or
deleting these materialized views?
Thank you.
Re: Recreating materialized views in cassandra
Posted by Jasonstack Zhao Yang <ja...@gmail.com>.
> Also, do you think putting the '-Dmv_enable_coordinator_batchlog=true'
parameter in cassandra.yaml will solve or reduce the issue to some extent?
It should improve the eventual consistency of MV. Only enable it if there
is enough CPU/IO capacity in the cluster.
Do you mind creating a JIRA describing: what's the workload/queries and how
does it end up in an inconsistent state if you can reproduce it?
On Wed, 29 Jul 2020 at 20:49, Jasonstack Zhao Yang <
jasonstack.zhao@gmail.com> wrote:
> > The cluster started to crash when some partitions in MV crossed 1 GB
> size at few nodes, whereas in other nodes it is less than 50 MB.
> > Should we be worried about this?
>
> Depend on your MV partition key design.
>
> The memory pressure of wide partitions is improved in 3.x (CASSANDRA-11206
> <https://issues.apache.org/jira/browse/CASSANDRA-11206>).
>
> On Tue, 28 Jul 2020 at 14:24, Saijal Chauhan <sa...@goevive.com>
> wrote:
>
>> >do you run "nodetool repair" on both base and view regularly?
>>
>> Yes, we run a full repair on our entire cluster every weekend which
>> includes the keyspaces with the base table and materialized views
>> But still, there are a ton of discrepancies in our base table and
>> materialized view.
>>
>> Also, do you think putting the '-Dmv_enable_coordinator_batchlog=true'
>> parameter in cassandra.yaml will solve or reduce the issue to some extent?
>>
>> Came across a Jira issue
>> <https://issues.apache.org/jira/browse/CASSANDRA-15918> and this
>> <https://medium.com/engineered-publicis-sapient/making-the-right-choices-in-cassandra-with-critical-configurations-and-data-size-speed-d358989d3437>
>> blog which mentions cluster instability while creating and deleting mv's
>>
>> The cluster started to crash when some partitions in MV crossed 1 GB size
>>> at few nodes, whereas in other nodes it is less than 50 MB.
>>
>>
>> Should we be worried about this?
>>
>> On Mon, Jul 27, 2020 at 10:18 PM Jasonstack Zhao Yang <
>> jasonstack.zhao@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> > We are facing data inconsistency issues between base tables and
>>> materialized views.
>>>
>>> do you run "nodetool repair" on both base and view regularly?
>>>
>>> > What are all the possible scenarios that we should be watching out for
>>> in a production environment?
>>>
>>> more cpu/io/gc for populating views.
>>>
>>> > Could there be any downtime in the Cassandra cluster while creating or
>>> deleting these materialized views?
>>>
>>> no, but be careful about the latency/throughput impact on the regular
>>> workload.
>>>
>>> On Tue, 28 Jul 2020 at 00:02, Saijal Chauhan <sa...@goevive.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are using Cassandra 3.0.13
>>>> We have the following datacenters:
>>>>
>>>> - DC1 with 7 Cassandra nodes with RF:3
>>>> - DC2 with 2 Cassandra nodes with RF:2
>>>> - DC3 with 2 Cassandra nodes with RF:2
>>>>
>>>> We are facing data inconsistency issues between base tables and
>>>> materialized views.
>>>> The only solution to this problem seems to be the creation of new
>>>> materialized views and dropping the old views.
>>>>
>>>> We are planning to recreate 4 materialized views, 2 belonging to the
>>>> same base table.
>>>> The size of each base table ranges up to 4 to 5GB.
>>>>
>>>> What are all the possible scenarios that we should be watching out for
>>>> in a production environment?
>>>> Could there be any downtime in the Cassandra cluster while creating or
>>>> deleting these materialized views?
>>>>
>>>> Thank you.
>>>>
>>>
>>
>>
Re: Recreating materialized views in cassandra
Posted by Jasonstack Zhao Yang <ja...@gmail.com>.
> The cluster started to crash when some partitions in MV crossed 1 GB size
at few nodes, whereas in other nodes it is less than 50 MB.
> Should we be worried about this?
Depend on your MV partition key design.
The memory pressure of wide partitions is improved in 3.x (CASSANDRA-11206
<https://issues.apache.org/jira/browse/CASSANDRA-11206>).
On Tue, 28 Jul 2020 at 14:24, Saijal Chauhan <sa...@goevive.com>
wrote:
> >do you run "nodetool repair" on both base and view regularly?
>
> Yes, we run a full repair on our entire cluster every weekend which
> includes the keyspaces with the base table and materialized views
> But still, there are a ton of discrepancies in our base table and
> materialized view.
>
> Also, do you think putting the '-Dmv_enable_coordinator_batchlog=true'
> parameter in cassandra.yaml will solve or reduce the issue to some extent?
>
> Came across a Jira issue
> <https://issues.apache.org/jira/browse/CASSANDRA-15918> and this
> <https://medium.com/engineered-publicis-sapient/making-the-right-choices-in-cassandra-with-critical-configurations-and-data-size-speed-d358989d3437>
> blog which mentions cluster instability while creating and deleting mv's
>
> The cluster started to crash when some partitions in MV crossed 1 GB size
>> at few nodes, whereas in other nodes it is less than 50 MB.
>
>
> Should we be worried about this?
>
> On Mon, Jul 27, 2020 at 10:18 PM Jasonstack Zhao Yang <
> jasonstack.zhao@gmail.com> wrote:
>
>> Hi,
>>
>> > We are facing data inconsistency issues between base tables and
>> materialized views.
>>
>> do you run "nodetool repair" on both base and view regularly?
>>
>> > What are all the possible scenarios that we should be watching out for
>> in a production environment?
>>
>> more cpu/io/gc for populating views.
>>
>> > Could there be any downtime in the Cassandra cluster while creating or
>> deleting these materialized views?
>>
>> no, but be careful about the latency/throughput impact on the regular
>> workload.
>>
>> On Tue, 28 Jul 2020 at 00:02, Saijal Chauhan <sa...@goevive.com>
>> wrote:
>>
>>> Hi,
>>>
>>> We are using Cassandra 3.0.13
>>> We have the following datacenters:
>>>
>>> - DC1 with 7 Cassandra nodes with RF:3
>>> - DC2 with 2 Cassandra nodes with RF:2
>>> - DC3 with 2 Cassandra nodes with RF:2
>>>
>>> We are facing data inconsistency issues between base tables and
>>> materialized views.
>>> The only solution to this problem seems to be the creation of new
>>> materialized views and dropping the old views.
>>>
>>> We are planning to recreate 4 materialized views, 2 belonging to the
>>> same base table.
>>> The size of each base table ranges up to 4 to 5GB.
>>>
>>> What are all the possible scenarios that we should be watching out for
>>> in a production environment?
>>> Could there be any downtime in the Cassandra cluster while creating or
>>> deleting these materialized views?
>>>
>>> Thank you.
>>>
>>
>
>
Re: Recreating materialized views in cassandra
Posted by Saijal Chauhan <sa...@goevive.com>.
>do you run "nodetool repair" on both base and view regularly?
Yes, we run a full repair on our entire cluster every weekend which
includes the keyspaces with the base table and materialized views
But still, there are a ton of discrepancies in our base table and
materialized view.
Also, do you think putting the '-Dmv_enable_coordinator_batchlog=true'
parameter in cassandra.yaml will solve or reduce the issue to some extent?
Came across a Jira issue
<https://issues.apache.org/jira/browse/CASSANDRA-15918> and this
<https://medium.com/engineered-publicis-sapient/making-the-right-choices-in-cassandra-with-critical-configurations-and-data-size-speed-d358989d3437>
blog which mentions cluster instability while creating and deleting mv's
The cluster started to crash when some partitions in MV crossed 1 GB size
> at few nodes, whereas in other nodes it is less than 50 MB.
Should we be worried about this?
On Mon, Jul 27, 2020 at 10:18 PM Jasonstack Zhao Yang <
jasonstack.zhao@gmail.com> wrote:
> Hi,
>
> > We are facing data inconsistency issues between base tables and
> materialized views.
>
> do you run "nodetool repair" on both base and view regularly?
>
> > What are all the possible scenarios that we should be watching out for
> in a production environment?
>
> more cpu/io/gc for populating views.
>
> > Could there be any downtime in the Cassandra cluster while creating or
> deleting these materialized views?
>
> no, but be careful about the latency/throughput impact on the regular
> workload.
>
> On Tue, 28 Jul 2020 at 00:02, Saijal Chauhan <sa...@goevive.com>
> wrote:
>
>> Hi,
>>
>> We are using Cassandra 3.0.13
>> We have the following datacenters:
>>
>> - DC1 with 7 Cassandra nodes with RF:3
>> - DC2 with 2 Cassandra nodes with RF:2
>> - DC3 with 2 Cassandra nodes with RF:2
>>
>> We are facing data inconsistency issues between base tables and
>> materialized views.
>> The only solution to this problem seems to be the creation of new
>> materialized views and dropping the old views.
>>
>> We are planning to recreate 4 materialized views, 2 belonging to the same
>> base table.
>> The size of each base table ranges up to 4 to 5GB.
>>
>> What are all the possible scenarios that we should be watching out for in
>> a production environment?
>> Could there be any downtime in the Cassandra cluster while creating or
>> deleting these materialized views?
>>
>> Thank you.
>>
>
Re: Recreating materialized views in cassandra
Posted by Jasonstack Zhao Yang <ja...@gmail.com>.
Hi,
> We are facing data inconsistency issues between base tables and
materialized views.
do you run "nodetool repair" on both base and view regularly?
> What are all the possible scenarios that we should be watching out for in
a production environment?
more cpu/io/gc for populating views.
> Could there be any downtime in the Cassandra cluster while creating or
deleting these materialized views?
no, but be careful about the latency/throughput impact on the regular
workload.
On Tue, 28 Jul 2020 at 00:02, Saijal Chauhan <sa...@goevive.com>
wrote:
> Hi,
>
> We are using Cassandra 3.0.13
> We have the following datacenters:
>
> - DC1 with 7 Cassandra nodes with RF:3
> - DC2 with 2 Cassandra nodes with RF:2
> - DC3 with 2 Cassandra nodes with RF:2
>
> We are facing data inconsistency issues between base tables and
> materialized views.
> The only solution to this problem seems to be the creation of new
> materialized views and dropping the old views.
>
> We are planning to recreate 4 materialized views, 2 belonging to the same
> base table.
> The size of each base table ranges up to 4 to 5GB.
>
> What are all the possible scenarios that we should be watching out for in
> a production environment?
> Could there be any downtime in the Cassandra cluster while creating or
> deleting these materialized views?
>
> Thank you.
>