You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Saurabh Sethi <Sa...@symantec.com> on 2015/09/16 23:19:45 UTC
No schema agreement from live replicas
We have a 3 node Cassandra cluster (version 2.2.0). Two of the nodes are in one subnet and third one in different subnet.
Replication factor is 2 and two of the nodes are seed nodes. Consistency level for everything is QUORUM.
We are getting the following warning:
WARN com.datastax.driver.core.Cluster - No schema agreement from live replicas after 30 s. The schema may not be up to date on some nodes.
I made sure that the clocks are in-sync.
I then ran cleanup, repair and compact but that solves the problem only for couple of days, the warning then comes back.
We were seeing this issue in 2.1.0 but stopped seeing it in 2.1.4 and 2.1.6. But this came back in version 2.2.0.
Anyone has any idea on how to fix this problem?
Just FYI: We do a lot of schema creation and deletion on Cassandra (basically our CI pipeline does that), if that has anything to do with it.
Thanks,
Saurabh
Re: No schema agreement from live replicas
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
With Rf=2, cl=quorum is effectively the same as
ALL. Expect downtime anytime you restart a node.
On Wed, Sep 16, 2015 at 3:39 PM Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:
> check nodetool describecluster to see the schema versions across your
> nodes.
> <http://docs.datastax.com/en/cassandra/2.1/cassandra/troubleshooting/trblshootSchemaDisagree.html>
> A rolling restart will help propagate schema if you have disagreement
> across your nodes.
>
> Just FYI: We do a lot of schema creation and deletion on Cassandra
>> (basically our CI pipeline does that), if that has anything to do with it.
>
>
> You should not do schema creation and deletion automatically in cassandra
> (especially from a distributed app) because you risk schema disagreement
> which can lead to issues like this one:
>
>
> http://stackoverflow.com/questions/31576180/cassandra-2-1-system-schema-missing
>
> If you do a schema modification from a single threaded process, always
> check for schema agreement before moving forward using
> https://datastax.github.io/java-driver/2.0.10/features/metadata/
>
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
>
>
> <http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Wed, Sep 16, 2015 at 5:19 PM, Saurabh Sethi <Saurabh_Sethi@symantec.com
> > wrote:
>
>> We have a 3 node Cassandra cluster (version 2.2.0). Two of the nodes are
>> in one subnet and third one in different subnet.
>>
>> Replication factor is 2 and two of the nodes are seed nodes. Consistency
>> level for everything is QUORUM.
>>
>> We are getting the following warning:
>>
>> *WARN com.datastax.driver.core.Cluster - No schema agreement from live
>> replicas after 30 s. The schema may not be up to date on some nodes.*
>>
>> I made sure that the clocks are in-sync.
>>
>> I then ran cleanup, repair and compact but that solves the problem only
>> for couple of days, the warning then comes back.
>>
>> We were seeing this issue in 2.1.0 but stopped seeing it in 2.1.4 and
>> 2.1.6. But this came back in version 2.2.0.
>>
>> Anyone has any idea on how to fix this problem?
>>
>> Just FYI: We do a lot of schema creation and deletion on Cassandra
>> (basically our CI pipeline does that), if that has anything to do with it.
>>
>> Thanks,
>> Saurabh
>>
>
>
Re: No schema agreement from live replicas
Posted by Sebastian Estevez <se...@datastax.com>.
check nodetool describecluster to see the schema versions across your nodes.
<http://docs.datastax.com/en/cassandra/2.1/cassandra/troubleshooting/trblshootSchemaDisagree.html>
A rolling restart will help propagate schema if you have disagreement
across your nodes.
Just FYI: We do a lot of schema creation and deletion on Cassandra
> (basically our CI pipeline does that), if that has anything to do with it.
You should not do schema creation and deletion automatically in cassandra
(especially from a distributed app) because you risk schema disagreement
which can lead to issues like this one:
http://stackoverflow.com/questions/31576180/cassandra-2-1-system-schema-missing
If you do a schema modification from a single threaded process, always
check for schema agreement before moving forward using
https://datastax.github.io/java-driver/2.0.10/features/metadata/
All the best,
[image: datastax_logo.png] <http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://cassandrasummit-datastax.com/?utm_campaign=summit15&utm_medium=summiticon&utm_source=emailsignature>
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.
On Wed, Sep 16, 2015 at 5:19 PM, Saurabh Sethi <Sa...@symantec.com>
wrote:
> We have a 3 node Cassandra cluster (version 2.2.0). Two of the nodes are
> in one subnet and third one in different subnet.
>
> Replication factor is 2 and two of the nodes are seed nodes. Consistency
> level for everything is QUORUM.
>
> We are getting the following warning:
>
> *WARN com.datastax.driver.core.Cluster - No schema agreement from live
> replicas after 30 s. The schema may not be up to date on some nodes.*
>
> I made sure that the clocks are in-sync.
>
> I then ran cleanup, repair and compact but that solves the problem only
> for couple of days, the warning then comes back.
>
> We were seeing this issue in 2.1.0 but stopped seeing it in 2.1.4 and
> 2.1.6. But this came back in version 2.2.0.
>
> Anyone has any idea on how to fix this problem?
>
> Just FYI: We do a lot of schema creation and deletion on Cassandra
> (basically our CI pipeline does that), if that has anything to do with it.
>
> Thanks,
> Saurabh
>