You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Gaurav Sehgal <gs...@gmail.com> on 2014/05/12 21:31:00 UTC
Schema disagreement errors
We have recently started seeing a lot of Schema Disagreement errors. We are
using Cassandra 2.0.6 with Oracle Java 1.7. I went through the Cassandra
FAQ and followed the below steps:
- nodetool disablethrift
- nodetool disablegossip
- nodetool drain
-
'kill <pid>'.
As per the documentation; the commit logs should have been flush; but that
did not happen in our case. The commit logs were still there. So, I removed
them manually to make sure there are no commit logs when cassandra start
up( which was fine in our case as this data can always be replayed). I
also deleted the schema* directory from the /data/system folder.
Though when we started cassandra back up the issue started happening again.
Any help would be appreciated
Cheers!
Gaurav
Re: Schema disagreement errors
Posted by Vincent Mallet <vm...@gmail.com>.
Hey Gaurav,
You should consider moving to 2.0.7 which fixes a bunch of these schema
disagreement problems. You could also play around with nodetool
resetlocalschema on the nodes that are behind, but be careful with that
one. I'd go with 2.0.7 first for sure.
Thanks,
Vince.
On Mon, May 12, 2014 at 12:31 PM, Gaurav Sehgal <gs...@gmail.com> wrote:
> We have recently started seeing a lot of Schema Disagreement errors. We
> are using Cassandra 2.0.6 with Oracle Java 1.7. I went through the
> Cassandra FAQ and followed the below steps:
>
>
>
> - nodetool disablethrift
> - nodetool disablegossip
> - nodetool drain
> -
>
> 'kill <pid>'.
>
>
> As per the documentation; the commit logs should have been flush; but that
> did not happen in our case. The commit logs were still there. So, I removed
> them manually to make sure there are no commit logs when cassandra start
> up( which was fine in our case as this data can always be replayed). I
> also deleted the schema* directory from the /data/system folder.
>
> Though when we started cassandra back up the issue started happening again.
>
>
> Any help would be appreciated
>
> Cheers!
> Gaurav
>
>
>
Re: Schema disagreement errors
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, May 13, 2014 at 5:11 PM, Donald Smith <
Donald.Smith@audiencescience.com> wrote:
> I too have noticed that after doing “nodetool flush” (or “nodetool
> drain”), the commit logs are still there. I think they’re NEW (empty)
> commit logs, but I may be wrong. Anyone know?
>
Assuming they are being correctly marked clean after drain (which
historically has been a nontrivial assumption) they are "new", "empty"
commit log segments which have been "recycled".
=Rob
RE: Schema disagreement errors
Posted by Donald Smith <Do...@audiencescience.com>.
I too have noticed that after doing “nodetool flush” (or “nodetool drain”), the commit logs are still there. I think they’re NEW (empty) commit logs, but I may be wrong. Anyone know?
Don
From: Gaurav Sehgal [mailto:gsehgal1@gmail.com]
Sent: Monday, May 12, 2014 12:31 PM
To: user@cassandra.apache.org
Subject: Schema disagreement errors
We have recently started seeing a lot of Schema Disagreement errors. We are using Cassandra 2.0.6 with Oracle Java 1.7. I went through the Cassandra FAQ and followed the below steps:
* nodetool disablethrift
* nodetool disablegossip
* nodetool drain
* 'kill <pid>'.
As per the documentation; the commit logs should have been flush; but that did not happen in our case. The commit logs were still there. So, I removed them manually to make sure there are no commit logs when cassandra start up( which was fine in our case as this data can always be replayed). I also deleted the schema* directory from the /data/system folder.
Though when we started cassandra back up the issue started happening again.
Any help would be appreciated
Cheers!
Gaurav
Re: Schema disagreement errors
Posted by Duncan Sands <du...@gmail.com>.
Hi Gaurav, a schema versioning bug was fixed in 2.0.7.
Best wishes, Duncan.
On 12/05/14 21:31, Gaurav Sehgal wrote:
> We have recently started seeing a lot of Schema Disagreement errors. We are
> using Cassandra 2.0.6 with Oracle Java 1.7. I went through the Cassandra FAQ and
> followed the below steps:
>
>
> * nodetool disablethrift
> * nodetool disablegossip
> * nodetool drain
> *
>
> 'kill <pid>'.
>
>
> As per the documentation; the commit logs should have been flush; but that did
> not happen in our case. The commit logs were still there. So, I removed them
> manually to make sure there are no commit logs when cassandra start up( which
> was fine in our case as this data can always be replayed). I also deleted the
> schema* directory from the /data/system folder.
>
> Though when we started cassandra back up the issue started happening again.
>
>
> Any help would be appreciated
>
> Cheers!
> Gaurav
>
>
Re: Schema disagreement errors
Posted by "Laing, Michael" <mi...@nytimes.com>.
Upgrade to 2.0.7 fixed this for me.
You can also try 'nodetool resetlocalschema' on disagreeing nodes. This
worked temporarily for me in 2.0.6.
ml
On Mon, May 12, 2014 at 3:31 PM, Gaurav Sehgal <gs...@gmail.com> wrote:
> We have recently started seeing a lot of Schema Disagreement errors. We
> are using Cassandra 2.0.6 with Oracle Java 1.7. I went through the
> Cassandra FAQ and followed the below steps:
>
>
>
> - nodetool disablethrift
> - nodetool disablegossip
> - nodetool drain
> -
>
> 'kill <pid>'.
>
>
> As per the documentation; the commit logs should have been flush; but that
> did not happen in our case. The commit logs were still there. So, I removed
> them manually to make sure there are no commit logs when cassandra start
> up( which was fine in our case as this data can always be replayed). I
> also deleted the schema* directory from the /data/system folder.
>
> Though when we started cassandra back up the issue started happening again.
>
>
> Any help would be appreciated
>
> Cheers!
> Gaurav
>
>
>