You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ben Ashton <be...@bossastudios.com> on 2011/10/05 20:47:06 UTC
Could not reach schema agreement
Hi Guys,
How would I go about fixing this? (running 0.8.4)
[default@unknown] connect 10.58.135.19/9160;
Connected to: "Test Cluster" on 10.58.135.19/9160
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
66bd76c0-ee97-11e0-0000-242d50cf1fbf: [10.234.119.110]
777ae000-cfd5-11e0-0000-242d50cf1fbf: [10.58.135.19,
10.48.234.31, 10.224.55.162]
ERROR [HintedHandoff:2] 2011-10-05 18:39:36,896
AbstractCassandraDaemon.java (line 134) Fatal exception in thread
Thread[HintedHandoff:2,1,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not
reach schema agreement with /10.234.119.110 in 60000ms
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: Could not reach schema
agreement with /10.234.119.110 in 60000ms
at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:293)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:304)
at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:397)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 3 more
Re: Could not reach schema agreement
Posted by aaron morton <aa...@thelastpickle.com>.
Check the data directories, including the snapshot one. Data is not deleted.
If you create a CF the server will look for existing files and load them.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 6/10/2011, at 8:35 AM, Ben Ashton wrote:
> oh no spoke to soon..
>
> All me data are being gone :(
>
> /opt/apache-cassandra-0.8.4/bin/nodetool -h 10.224.55.162 repair
> Exception in thread "main" java.lang.AssertionError: Repairing no
> column families seems pointless, doesn't it
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:625)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:617)
> at org.apache.cassandra.service.AntiEntropyService.getRepairSession(AntiEntropyService.java:129)
> at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1620)
> at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1579)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
>
>
> On 5 October 2011 20:03, Ben Ashton <be...@bossastudios.com> wrote:
>> Ah thats great!
>>
>> I was rubbing my head for a while as google only showed mailing lists
>> posts with the same error.
>>
>> All working now, thanks
>>
>> On 5 October 2011 19:49, Jonathan Ellis <jb...@gmail.com> wrote:
>>> Did you try wiki.apache.org/cassandra/FAQ#schema_disagreement ?
>>>
>>> On Wed, Oct 5, 2011 at 1:47 PM, Ben Ashton <be...@bossastudios.com> wrote:
>>>> Hi Guys,
>>>>
>>>> How would I go about fixing this? (running 0.8.4)
>>>>
>>>> [default@unknown] connect 10.58.135.19/9160;
>>>> Connected to: "Test Cluster" on 10.58.135.19/9160
>>>> [default@unknown] describe cluster;
>>>> Cluster Information:
>>>> Snitch: org.apache.cassandra.locator.SimpleSnitch
>>>> Partitioner: org.apache.cassandra.dht.RandomPartitioner
>>>> Schema versions:
>>>> 66bd76c0-ee97-11e0-0000-242d50cf1fbf: [10.234.119.110]
>>>> 777ae000-cfd5-11e0-0000-242d50cf1fbf: [10.58.135.19,
>>>> 10.48.234.31, 10.224.55.162]
>>>>
>>>> ERROR [HintedHandoff:2] 2011-10-05 18:39:36,896
>>>> AbstractCassandraDaemon.java (line 134) Fatal exception in thread
>>>> Thread[HintedHandoff:2,1,main]
>>>> java.lang.RuntimeException: java.lang.RuntimeException: Could not
>>>> reach schema agreement with /10.234.119.110 in 60000ms
>>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>> Caused by: java.lang.RuntimeException: Could not reach schema
>>>> agreement with /10.234.119.110 in 60000ms
>>>> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:293)
>>>> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:304)
>>>> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
>>>> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:397)
>>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>> ... 3 more
>>>>
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of DataStax, the source for professional Cassandra support
>>> http://www.datastax.com
>>>
>>
Re: Could not reach schema agreement
Posted by Ben Ashton <be...@bossastudios.com>.
oh no spoke to soon..
All me data are being gone :(
/opt/apache-cassandra-0.8.4/bin/nodetool -h 10.224.55.162 repair
Exception in thread "main" java.lang.AssertionError: Repairing no
column families seems pointless, doesn't it
at org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:625)
at org.apache.cassandra.service.AntiEntropyService$RepairSession.<init>(AntiEntropyService.java:617)
at org.apache.cassandra.service.AntiEntropyService.getRepairSession(AntiEntropyService.java:129)
at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1620)
at org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1579)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
On 5 October 2011 20:03, Ben Ashton <be...@bossastudios.com> wrote:
> Ah thats great!
>
> I was rubbing my head for a while as google only showed mailing lists
> posts with the same error.
>
> All working now, thanks
>
> On 5 October 2011 19:49, Jonathan Ellis <jb...@gmail.com> wrote:
>> Did you try wiki.apache.org/cassandra/FAQ#schema_disagreement ?
>>
>> On Wed, Oct 5, 2011 at 1:47 PM, Ben Ashton <be...@bossastudios.com> wrote:
>>> Hi Guys,
>>>
>>> How would I go about fixing this? (running 0.8.4)
>>>
>>> [default@unknown] connect 10.58.135.19/9160;
>>> Connected to: "Test Cluster" on 10.58.135.19/9160
>>> [default@unknown] describe cluster;
>>> Cluster Information:
>>> Snitch: org.apache.cassandra.locator.SimpleSnitch
>>> Partitioner: org.apache.cassandra.dht.RandomPartitioner
>>> Schema versions:
>>> 66bd76c0-ee97-11e0-0000-242d50cf1fbf: [10.234.119.110]
>>> 777ae000-cfd5-11e0-0000-242d50cf1fbf: [10.58.135.19,
>>> 10.48.234.31, 10.224.55.162]
>>>
>>> ERROR [HintedHandoff:2] 2011-10-05 18:39:36,896
>>> AbstractCassandraDaemon.java (line 134) Fatal exception in thread
>>> Thread[HintedHandoff:2,1,main]
>>> java.lang.RuntimeException: java.lang.RuntimeException: Could not
>>> reach schema agreement with /10.234.119.110 in 60000ms
>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> at java.lang.Thread.run(Thread.java:636)
>>> Caused by: java.lang.RuntimeException: Could not reach schema
>>> agreement with /10.234.119.110 in 60000ms
>>> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:293)
>>> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:304)
>>> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
>>> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:397)
>>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>> ... 3 more
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
Re: Could not reach schema agreement
Posted by Ben Ashton <be...@bossastudios.com>.
Ah thats great!
I was rubbing my head for a while as google only showed mailing lists
posts with the same error.
All working now, thanks
On 5 October 2011 19:49, Jonathan Ellis <jb...@gmail.com> wrote:
> Did you try wiki.apache.org/cassandra/FAQ#schema_disagreement ?
>
> On Wed, Oct 5, 2011 at 1:47 PM, Ben Ashton <be...@bossastudios.com> wrote:
>> Hi Guys,
>>
>> How would I go about fixing this? (running 0.8.4)
>>
>> [default@unknown] connect 10.58.135.19/9160;
>> Connected to: "Test Cluster" on 10.58.135.19/9160
>> [default@unknown] describe cluster;
>> Cluster Information:
>> Snitch: org.apache.cassandra.locator.SimpleSnitch
>> Partitioner: org.apache.cassandra.dht.RandomPartitioner
>> Schema versions:
>> 66bd76c0-ee97-11e0-0000-242d50cf1fbf: [10.234.119.110]
>> 777ae000-cfd5-11e0-0000-242d50cf1fbf: [10.58.135.19,
>> 10.48.234.31, 10.224.55.162]
>>
>> ERROR [HintedHandoff:2] 2011-10-05 18:39:36,896
>> AbstractCassandraDaemon.java (line 134) Fatal exception in thread
>> Thread[HintedHandoff:2,1,main]
>> java.lang.RuntimeException: java.lang.RuntimeException: Could not
>> reach schema agreement with /10.234.119.110 in 60000ms
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:636)
>> Caused by: java.lang.RuntimeException: Could not reach schema
>> agreement with /10.234.119.110 in 60000ms
>> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:293)
>> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:304)
>> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
>> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:397)
>> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> ... 3 more
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: Could not reach schema agreement
Posted by Jonathan Ellis <jb...@gmail.com>.
Did you try wiki.apache.org/cassandra/FAQ#schema_disagreement ?
On Wed, Oct 5, 2011 at 1:47 PM, Ben Ashton <be...@bossastudios.com> wrote:
> Hi Guys,
>
> How would I go about fixing this? (running 0.8.4)
>
> [default@unknown] connect 10.58.135.19/9160;
> Connected to: "Test Cluster" on 10.58.135.19/9160
> [default@unknown] describe cluster;
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 66bd76c0-ee97-11e0-0000-242d50cf1fbf: [10.234.119.110]
> 777ae000-cfd5-11e0-0000-242d50cf1fbf: [10.58.135.19,
> 10.48.234.31, 10.224.55.162]
>
> ERROR [HintedHandoff:2] 2011-10-05 18:39:36,896
> AbstractCassandraDaemon.java (line 134) Fatal exception in thread
> Thread[HintedHandoff:2,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not
> reach schema agreement with /10.234.119.110 in 60000ms
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.lang.RuntimeException: Could not reach schema
> agreement with /10.234.119.110 in 60000ms
> at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:293)
> at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:304)
> at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
> at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:397)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com