You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nitan Kainth <ni...@gmail.com> on 2019/05/28 16:13:38 UTC
Can sstable corruption cause schema mismatch?
Hi,
Two questions:
1. Can sstable corruption in application tables cause schema mismatch?
2. Do we need to disable repair while adding storage while Cassandra is down?
Regards,
Nitan
Cell: 510 449 9629
Re: Can sstable corruption cause schema mismatch?
Posted by Nitan Kainth <ni...@gmail.com>.
All ports are open.
We tried rolling restart and full cluster down and start one mode at a time.
Changes down were:
Storage addition
Ddl for column drop and recreate
Schema version is same for few nodes and few shows unavailable.
Network has been verified in detail and no severe packet drops.
Regards,
Nitan
Cell: 510 449 9629
> On May 29, 2019, at 10:04 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>
> Ideas that come mind are:
>
> - Rolling restart of the cluster
> - Use of 'nodetool resetlocalschema' --> function name speaks for itself. Note that this is to be ran on each node you think is having schema issues
> - Are all nodes showing a schema version showing the same one?
> - Port not fully open across all nodes?
> - Anything in the logs?
>
> Do you know what triggered this situation in the first place?
>
> C*heers,
> -----------------------
> Alain Rodriguez - alain@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>> Le mar. 28 mai 2019 à 18:28, Nitan Kainth <ni...@gmail.com> a écrit :
>> Thank you Alain.
>>
>> Nodetool describecluster shows some nodes unreachable, different output from each node.
>> Node1 can see all 4 nodes up.
>> Node 2 says node 4 and node 5 unreachable
>> Node 3 complains about node node 2 and node 1
>>
>> Nodetool status shows all nodes up and read writes are working for most most operations.
>>
>> Network looks good. Any other ideas?
>>
>>
>> Regards,
>> Nitan
>> Cell: 510 449 9629
>>
>>> On May 28, 2019, at 11:21 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>>>
>>> Hello Nitan,
>>>
>>>> 1. Can sstable corruption in application tables cause schema mismatch?
>>>
>>> I would say it should not. I could imagine in the case that the corrupted table hits some 'system' keyspace sstable. If not I don' see how corrupted data can impact the schema on the node.
>>>
>>>> 2. Do we need to disable repair while adding storage while Cassandra is down?
>>>
>>> I think you don't have to, but that it's a good idea.
>>> Repairs would fail as soon/long as you have a node down that should be involved (I think there is an option to change that behaviour now).
>>> Anyway, stopping repair and restarting it when all nodes are probably allows you a better understanding/control of what's going on. Also, it reduces the load in time of troubles or maintenance, when the cluster is somewhat weaker.
>>>
>>> C*heers,
>>> -----------------------
>>> Alain Rodriguez - alain@thelastpickle.com
>>> France / Spain
>>>
>>> The Last Pickle - Apache Cassandra Consulting
>>> http://www.thelastpickle.com
>>>
>>>
>>>
>>>> Le mar. 28 mai 2019 à 17:13, Nitan Kainth <ni...@gmail.com> a écrit :
>>>> Hi,
>>>>
>>>> Two questions:
>>>> 1. Can sstable corruption in application tables cause schema mismatch?
>>>> 2. Do we need to disable repair while adding storage while Cassandra is down?
>>>>
>>>>
>>>> Regards,
>>>> Nitan
>>>> Cell: 510 449 9629
Re: Can sstable corruption cause schema mismatch?
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Ideas that come mind are:
- Rolling restart of the cluster
- Use of 'nodetool resetlocalschema' --> function name speaks for itself.
Note that this is to be ran on each node you think is having schema issues
- Are all nodes showing a schema version showing the same one?
- Port not fully open across all nodes?
- Anything in the logs?
Do you know what triggered this situation in the first place?
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France / Spain
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
Le mar. 28 mai 2019 à 18:28, Nitan Kainth <ni...@gmail.com> a écrit :
> Thank you Alain.
>
> Nodetool describecluster shows some nodes unreachable, different output
> from each node.
> Node1 can see all 4 nodes up.
> Node 2 says node 4 and node 5 unreachable
> Node 3 complains about node node 2 and node 1
>
> Nodetool status shows all nodes up and read writes are working for most
> most operations.
>
> Network looks good. Any other ideas?
>
>
> Regards,
>
> Nitan
>
> Cell: 510 449 9629
>
> On May 28, 2019, at 11:21 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>
> Hello Nitan,
>
> 1. Can sstable corruption in application tables cause schema mismatch?
>>
>
> I would say it should not. I could imagine in the case that the corrupted
> table hits some 'system' keyspace sstable. If not I don' see how corrupted
> data can impact the schema on the node.
>
>
>> 2. Do we need to disable repair while adding storage while Cassandra is
>> down?
>>
>
> I think you don't have to, but that it's a good idea.
> Repairs would fail as soon/long as you have a node down that should be
> involved (I think there is an option to change that behaviour now).
> Anyway, stopping repair and restarting it when all nodes are probably
> allows you a better understanding/control of what's going on. Also, it
> reduces the load in time of troubles or maintenance, when the cluster is
> somewhat weaker.
>
> C*heers,
> -----------------------
> Alain Rodriguez - alain@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
>
> Le mar. 28 mai 2019 à 17:13, Nitan Kainth <ni...@gmail.com> a
> écrit :
>
>> Hi,
>>
>> Two questions:
>> 1. Can sstable corruption in application tables cause schema mismatch?
>> 2. Do we need to disable repair while adding storage while Cassandra is
>> down?
>>
>>
>> Regards,
>>
>> Nitan
>>
>> Cell: 510 449 9629
>>
>
Re: Can sstable corruption cause schema mismatch?
Posted by Nitan Kainth <ni...@gmail.com>.
Thank you Alain.
Nodetool describecluster shows some nodes unreachable, different output from each node.
Node1 can see all 4 nodes up.
Node 2 says node 4 and node 5 unreachable
Node 3 complains about node node 2 and node 1
Nodetool status shows all nodes up and read writes are working for most most operations.
Network looks good. Any other ideas?
Regards,
Nitan
Cell: 510 449 9629
> On May 28, 2019, at 11:21 AM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>
> Hello Nitan,
>
>> 1. Can sstable corruption in application tables cause schema mismatch?
>
> I would say it should not. I could imagine in the case that the corrupted table hits some 'system' keyspace sstable. If not I don' see how corrupted data can impact the schema on the node.
>
>> 2. Do we need to disable repair while adding storage while Cassandra is down?
>
> I think you don't have to, but that it's a good idea.
> Repairs would fail as soon/long as you have a node down that should be involved (I think there is an option to change that behaviour now).
> Anyway, stopping repair and restarting it when all nodes are probably allows you a better understanding/control of what's going on. Also, it reduces the load in time of troubles or maintenance, when the cluster is somewhat weaker.
>
> C*heers,
> -----------------------
> Alain Rodriguez - alain@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
>
>> Le mar. 28 mai 2019 à 17:13, Nitan Kainth <ni...@gmail.com> a écrit :
>> Hi,
>>
>> Two questions:
>> 1. Can sstable corruption in application tables cause schema mismatch?
>> 2. Do we need to disable repair while adding storage while Cassandra is down?
>>
>>
>> Regards,
>> Nitan
>> Cell: 510 449 9629
Re: Can sstable corruption cause schema mismatch?
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hello Nitan,
1. Can sstable corruption in application tables cause schema mismatch?
>
I would say it should not. I could imagine in the case that the corrupted
table hits some 'system' keyspace sstable. If not I don' see how corrupted
data can impact the schema on the node.
> 2. Do we need to disable repair while adding storage while Cassandra is
> down?
>
I think you don't have to, but that it's a good idea.
Repairs would fail as soon/long as you have a node down that should be
involved (I think there is an option to change that behaviour now).
Anyway, stopping repair and restarting it when all nodes are probably
allows you a better understanding/control of what's going on. Also, it
reduces the load in time of troubles or maintenance, when the cluster is
somewhat weaker.
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France / Spain
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
Le mar. 28 mai 2019 à 17:13, Nitan Kainth <ni...@gmail.com> a écrit :
> Hi,
>
> Two questions:
> 1. Can sstable corruption in application tables cause schema mismatch?
> 2. Do we need to disable repair while adding storage while Cassandra is
> down?
>
>
> Regards,
>
> Nitan
>
> Cell: 510 449 9629
>