You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Pradeep Chhetri <pr...@stashaway.com> on 2017/10/12 04:53:26 UTC
Schema Mismatch Issue in Production
Hello everyone,
We had some issues yesterday in our 3 nodes cluster where the application
tried to create the same table twice quickly and cluster became unstable.
Temporarily, we reduced it to single node cluster which gave us some relief.
Now when we are trying to bootstrap a new node and add it to cluster. we're
seeing schema mismatch issue.
# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.42.247.173 3.07 GiB 256 100.0%
dffc39e5-d4ba-4b10-872e-0e3cc10f5e08 rack1
UN 10.42.209.245 2.25 GiB 256 100.0%
9b99d5d8-818e-4741-9533-259d0fc0e16d rack1
root@cassandra-2:~# nodetool describecluster
Cluster Information:
Name: sa-cassandra
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
e2275d0f-a5fc-39d9-8f11-268b5e9dc295: [10.42.209.245]
5f5f66f5-d6aa-3b90-b674-e08811d4d412: [10.42.247.173]
Freshly bootstrapped node - 10.42.247.173
Single node from original cluster - 10.42.209.245
I read https://docs.datastax.com/en/dse-trblshoot/doc/troubleshooting/
schemaDisagree.html and tried restarting the new node but it didnt help.
Please do suggest. We are facing this issue in production.
Thank you.
Re: Schema Mismatch Issue in Production
Posted by Pradeep Chhetri <pr...@stashaway.com>.
Got the cluster to converge on same schema by restarting just the one
having different version.
Thanks.
On Thu, Oct 12, 2017 at 2:23 PM, Pradeep Chhetri <pr...@stashaway.com>
wrote:
> Hi Carlos,
>
> Thank you for the reply.
>
> I am running 3.9 cassandra version.
>
> I am also not sure what affect does this have on the applications talking
> to the cassandra.
>
> So there is no way to converge the cluster schema version without downtime.
>
> Thank you.
>
> On Thu, Oct 12, 2017 at 2:16 PM, Carlos Rolo <ro...@pythian.com> wrote:
>
>> Which version are you running? I got stuck in a similar situation (With a
>> lot more nodes) and the only way to make it good was to stop the whole
>> cluster, start nodes 1 by 1.
>>
>>
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
>> *linkedin.com/in/carlosjuzarterolo
>> <http://linkedin.com/in/carlosjuzarterolo>*
>> Mobile: +351 918 918 100
>> www.pythian.com
>>
>> On Thu, Oct 12, 2017 at 5:53 AM, Pradeep Chhetri <pr...@stashaway.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> We had some issues yesterday in our 3 nodes cluster where the
>>> application tried to create the same table twice quickly and cluster became
>>> unstable.
>>>
>>> Temporarily, we reduced it to single node cluster which gave us some
>>> relief.
>>>
>>> Now when we are trying to bootstrap a new node and add it to cluster.
>>> we're seeing schema mismatch issue.
>>>
>>> # nodetool status
>>> Datacenter: datacenter1
>>> =======================
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> -- Address Load Tokens Owns (effective) Host ID
>>> Rack
>>> UN 10.42.247.173 3.07 GiB 256 100.0%
>>> dffc39e5-d4ba-4b10-872e-0e3cc10f5e08 rack1
>>> UN 10.42.209.245 2.25 GiB 256 100.0%
>>> 9b99d5d8-818e-4741-9533-259d0fc0e16d rack1
>>>
>>> root@cassandra-2:~# nodetool describecluster
>>> Cluster Information:
>>> Name: sa-cassandra
>>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> Schema versions:
>>> e2275d0f-a5fc-39d9-8f11-268b5e9dc295: [10.42.209.245]
>>>
>>> 5f5f66f5-d6aa-3b90-b674-e08811d4d412: [10.42.247.173]
>>>
>>> Freshly bootstrapped node - 10.42.247.173
>>> Single node from original cluster - 10.42.209.245
>>>
>>> I read https://docs.datastax.com/en/dse-trblshoot/doc/troubles
>>> hooting/schemaDisagree.html and tried restarting the new node but it
>>> didnt help.
>>>
>>> Please do suggest. We are facing this issue in production.
>>>
>>> Thank you.
>>>
>>
>>
>> --
>>
>>
>>
>>
>
Re: Schema Mismatch Issue in Production
Posted by Pradeep Chhetri <pr...@stashaway.com>.
Hi Carlos,
Thank you for the reply.
I am running 3.9 cassandra version.
I am also not sure what affect does this have on the applications talking
to the cassandra.
So there is no way to converge the cluster schema version without downtime.
Thank you.
On Thu, Oct 12, 2017 at 2:16 PM, Carlos Rolo <ro...@pythian.com> wrote:
> Which version are you running? I got stuck in a similar situation (With a
> lot more nodes) and the only way to make it good was to stop the whole
> cluster, start nodes 1 by 1.
>
>
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> *linkedin.com/in/carlosjuzarterolo
> <http://linkedin.com/in/carlosjuzarterolo>*
> Mobile: +351 918 918 100
> www.pythian.com
>
> On Thu, Oct 12, 2017 at 5:53 AM, Pradeep Chhetri <pr...@stashaway.com>
> wrote:
>
>> Hello everyone,
>>
>> We had some issues yesterday in our 3 nodes cluster where the application
>> tried to create the same table twice quickly and cluster became unstable.
>>
>> Temporarily, we reduced it to single node cluster which gave us some
>> relief.
>>
>> Now when we are trying to bootstrap a new node and add it to cluster.
>> we're seeing schema mismatch issue.
>>
>> # nodetool status
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> UN 10.42.247.173 3.07 GiB 256 100.0%
>> dffc39e5-d4ba-4b10-872e-0e3cc10f5e08 rack1
>> UN 10.42.209.245 2.25 GiB 256 100.0%
>> 9b99d5d8-818e-4741-9533-259d0fc0e16d rack1
>>
>> root@cassandra-2:~# nodetool describecluster
>> Cluster Information:
>> Name: sa-cassandra
>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Schema versions:
>> e2275d0f-a5fc-39d9-8f11-268b5e9dc295: [10.42.209.245]
>>
>> 5f5f66f5-d6aa-3b90-b674-e08811d4d412: [10.42.247.173]
>>
>> Freshly bootstrapped node - 10.42.247.173
>> Single node from original cluster - 10.42.209.245
>>
>> I read https://docs.datastax.com/en/dse-trblshoot/doc/troubles
>> hooting/schemaDisagree.html and tried restarting the new node but it
>> didnt help.
>>
>> Please do suggest. We are facing this issue in production.
>>
>> Thank you.
>>
>
>
> --
>
>
>
>
Re: Schema Mismatch Issue in Production
Posted by Nitan Kainth <ni...@gmail.com>.
Rolling restart after making DDL changes saves us. We saw this because of race condition in our app servers, but it could happen for various other reasons like node overloaded, network etc
Sent from my iPhone
> On Oct 12, 2017, at 3:46 AM, Carlos Rolo <ro...@pythian.com> wrote:
>
> Which version are you running? I got stuck in a similar situation (With a lot more nodes) and the only way to make it good was to stop the whole cluster, start nodes 1 by 1.
>
>
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: linkedin.com/in/carlosjuzarterolo
> Mobile: +351 918 918 100
> www.pythian.com
>
>> On Thu, Oct 12, 2017 at 5:53 AM, Pradeep Chhetri <pr...@stashaway.com> wrote:
>> Hello everyone,
>>
>> We had some issues yesterday in our 3 nodes cluster where the application tried to create the same table twice quickly and cluster became unstable.
>>
>> Temporarily, we reduced it to single node cluster which gave us some relief.
>>
>> Now when we are trying to bootstrap a new node and add it to cluster. we're seeing schema mismatch issue.
>>
>> # nodetool status
>> Datacenter: datacenter1
>> =======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID Rack
>> UN 10.42.247.173 3.07 GiB 256 100.0% dffc39e5-d4ba-4b10-872e-0e3cc10f5e08 rack1
>> UN 10.42.209.245 2.25 GiB 256 100.0% 9b99d5d8-818e-4741-9533-259d0fc0e16d rack1
>>
>> root@cassandra-2:~# nodetool describecluster
>> Cluster Information:
>> Name: sa-cassandra
>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Schema versions:
>> e2275d0f-a5fc-39d9-8f11-268b5e9dc295: [10.42.209.245]
>>
>> 5f5f66f5-d6aa-3b90-b674-e08811d4d412: [10.42.247.173]
>>
>> Freshly bootstrapped node - 10.42.247.173
>> Single node from original cluster - 10.42.209.245
>>
>> I read https://docs.datastax.com/en/dse-trblshoot/doc/troubleshooting/schemaDisagree.html and tried restarting the new node but it didnt help.
>>
>> Please do suggest. We are facing this issue in production.
>>
>> Thank you.
>
>
> --
>
>
>
>
Re: Schema Mismatch Issue in Production
Posted by Carlos Rolo <ro...@pythian.com>.
Which version are you running? I got stuck in a similar situation (With a
lot more nodes) and the only way to make it good was to stop the whole
cluster, start nodes 1 by 1.
Regards,
Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
Pythian - Love your data
rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +351 918 918 100
www.pythian.com
On Thu, Oct 12, 2017 at 5:53 AM, Pradeep Chhetri <pr...@stashaway.com>
wrote:
> Hello everyone,
>
> We had some issues yesterday in our 3 nodes cluster where the application
> tried to create the same table twice quickly and cluster became unstable.
>
> Temporarily, we reduced it to single node cluster which gave us some
> relief.
>
> Now when we are trying to bootstrap a new node and add it to cluster.
> we're seeing schema mismatch issue.
>
> # nodetool status
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.42.247.173 3.07 GiB 256 100.0%
> dffc39e5-d4ba-4b10-872e-0e3cc10f5e08 rack1
> UN 10.42.209.245 2.25 GiB 256 100.0%
> 9b99d5d8-818e-4741-9533-259d0fc0e16d rack1
>
> root@cassandra-2:~# nodetool describecluster
> Cluster Information:
> Name: sa-cassandra
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> e2275d0f-a5fc-39d9-8f11-268b5e9dc295: [10.42.209.245]
>
> 5f5f66f5-d6aa-3b90-b674-e08811d4d412: [10.42.247.173]
>
> Freshly bootstrapped node - 10.42.247.173
> Single node from original cluster - 10.42.209.245
>
> I read https://docs.datastax.com/en/dse-trblshoot/doc/troubles
> hooting/schemaDisagree.html and tried restarting the new node but it
> didnt help.
>
> Please do suggest. We are facing this issue in production.
>
> Thank you.
>
--
--