You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Edmond Lau <ed...@ooyala.com> on 2009/10/29 20:18:26 UTC
can't write with consistency level of one after some nodes fail
I have a freshly started 3-node cluster with a replication factor of
2. If I take down two nodes, I can no longer do any writes, even with
a consistency level of one. I tried on a variety of keys to ensure
that I'd get at least one where the live node was responsible for one
of the replicas. I have not yet tried on trunk. On cassandra 0.4.1,
I get an UnavailableException:
DEBUG [pool-1-thread-1] 2009-10-29 18:53:24,371 CassandraServer.java
(line 408) insert
WARN [pool-1-thread-1] 2009-10-29 18:53:24,388
AbstractReplicationStrategy.java (line 135) Unable to find a live
Endpoint we might be out of live nodes , This is dangerous !!!!
ERROR [pool-1-thread-1] 2009-10-29 18:53:24,390 StorageProxy.java
(line 179) error writing key 1
UnavailableException()
at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:156)
at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:468)
at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:421)
at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:824)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Moreover, if I bring another of the nodes back up, do some writes,
take it back down again, and try to do writes with a consistency level
of one, I get an ApplicationException with the error text "unknown
result". There's nothing in the debug logs about this new error:
DEBUG [pool-1-thread-7] 2009-10-29 19:08:26,411 CassandraServer.java
(line 258) get
DEBUG [pool-1-thread-7] 2009-10-29 19:08:26,411 CassandraServer.java
(line 307) multiget
DEBUG [pool-1-thread-7] 2009-10-29 19:08:26,413 StorageProxy.java
(line 239) weakreadlocal reading
SliceByNamesReadCommand(table='Keyspace1', key='3',
columnParent='QueryPath(columnFamilyName='Standard1',
superColumnName='null', columnName='null')', columns=[[118, 97, 108,
117, 101],])
I would've instead expected the node to accept the write and then have
the key repaired on subsequent reads when the other nodes get back up.
Along the same lines, how does Cassandra handle network partitioning
where 2 writes for the same keys hit 2 different partitions, neither
of which are able to form a quorum? Dynamo maintained version vectors
and put the burden on the client to resolve conflicts, but there's no
similar interface in the thrift api.
Edmond
Re: can't write with consistency level of one after some nodes fail
Posted by Jonathan Ellis <jb...@gmail.com>.
Attached a fix in jira that should fix it.
On Thu, Oct 29, 2009 at 4:00 PM, Edmond Lau <ed...@ooyala.com> wrote:
> CASSANDRA-524
>
> On Thu, Oct 29, 2009 at 2:56 PM, Edmond Lau <ed...@ooyala.com> wrote:
>> Will do.
>>
>> On Thu, Oct 29, 2009 at 2:39 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>> can you create a ticket for this?
>>>
>>> On Thu, Oct 29, 2009 at 3:39 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>> I've seen that bug too. I think we introduced a regression with CASSANDRA-492.
>>>>
>>>> On Thu, Oct 29, 2009 at 3:37 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>>> I've updated to trunk, and I'm still hitting the same issue but it's
>>>>> manifesting itself differently. Again, I'm running with a freshly
>>>>> started 3-node cluster with a replication factor of 2. I then take
>>>>> down two nodes.
>>>>>
>>>>> If I write with a consistency level of ONE on any key, I get an
>>>>> InvalidRequestException:
>>>>>
>>>>> ERROR [pool-1-thread-45] 2009-10-29 21:27:10,120 StorageProxy.java
>>>>> (line 183) error writing key 1
>>>>> InvalidRequestException(why:Cannot block for less than one replica)
>>>>> at org.apache.cassandra.service.QuorumResponseHandler.<init>(QuorumResponseHandler.java:52)
>>>>> at org.apache.cassandra.locator.AbstractReplicationStrategy.getResponseHandler(AbstractReplicationStrategy.java:64)
>>>>> at org.apache.cassandra.service.StorageService.getResponseHandler(StorageService.java:869)
>>>>> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:162)
>>>>> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:473)
>>>>> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:424)
>>>>> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:819)
>>>>> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
>>>>> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>> Oddly, a write with a consistency level of QUORUM succeeds for certain
>>>>> keys (but fails with others) even though I only have one live node.
>>>>>
>>>>> Edmond
>>>>>
>>>>> On Thu, Oct 29, 2009 at 1:38 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>>>>
>>>>>> On Thu, Oct 29, 2009 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>>>>>
>>>>>>> On Thu, Oct 29, 2009 at 1:18 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>>>>> > I have a freshly started 3-node cluster with a replication factor of
>>>>>>> > 2. If I take down two nodes, I can no longer do any writes, even with
>>>>>>> > a consistency level of one. I tried on a variety of keys to ensure
>>>>>>> > that I'd get at least one where the live node was responsible for one
>>>>>>> > of the replicas. I have not yet tried on trunk. On cassandra 0.4.1,
>>>>>>> > I get an UnavailableException.
>>>>>>>
>>>>>>> This sounds like the bug we fixed in CASSANDRA-496 on trunk.
>>>>>>
>>>>>> Excellent - thanks. Time to start using trunk.
>>>>>>
>>>>>>>
>>>>>>> > Along the same lines, how does Cassandra handle network partitioning
>>>>>>> > where 2 writes for the same keys hit 2 different partitions, neither
>>>>>>> > of which are able to form a quorum? Dynamo maintained version vectors
>>>>>>> > and put the burden on the client to resolve conflicts, but there's no
>>>>>>> > similar interface in the thrift api.
>>>>>>>
>>>>>>> If you use QUORUM or ALL consistency, neither write will succeed. If
>>>>>>> you use ONE, both will, and the one with the higher timestamp will
>>>>>>> "win" when the partition heals.
>>>>>>
>>>>>> Got it.
>>>>>>
>>>>>>>
>>>>>>> -Jonathan
>>>>>>
>>>>>> Edmond
>>>>>
>>>>
>>>
>>
>
Re: can't write with consistency level of one after some nodes fail
Posted by Edmond Lau <ed...@ooyala.com>.
CASSANDRA-524
On Thu, Oct 29, 2009 at 2:56 PM, Edmond Lau <ed...@ooyala.com> wrote:
> Will do.
>
> On Thu, Oct 29, 2009 at 2:39 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> can you create a ticket for this?
>>
>> On Thu, Oct 29, 2009 at 3:39 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>> I've seen that bug too. I think we introduced a regression with CASSANDRA-492.
>>>
>>> On Thu, Oct 29, 2009 at 3:37 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>> I've updated to trunk, and I'm still hitting the same issue but it's
>>>> manifesting itself differently. Again, I'm running with a freshly
>>>> started 3-node cluster with a replication factor of 2. I then take
>>>> down two nodes.
>>>>
>>>> If I write with a consistency level of ONE on any key, I get an
>>>> InvalidRequestException:
>>>>
>>>> ERROR [pool-1-thread-45] 2009-10-29 21:27:10,120 StorageProxy.java
>>>> (line 183) error writing key 1
>>>> InvalidRequestException(why:Cannot block for less than one replica)
>>>> at org.apache.cassandra.service.QuorumResponseHandler.<init>(QuorumResponseHandler.java:52)
>>>> at org.apache.cassandra.locator.AbstractReplicationStrategy.getResponseHandler(AbstractReplicationStrategy.java:64)
>>>> at org.apache.cassandra.service.StorageService.getResponseHandler(StorageService.java:869)
>>>> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:162)
>>>> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:473)
>>>> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:424)
>>>> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:819)
>>>> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
>>>> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> Oddly, a write with a consistency level of QUORUM succeeds for certain
>>>> keys (but fails with others) even though I only have one live node.
>>>>
>>>> Edmond
>>>>
>>>> On Thu, Oct 29, 2009 at 1:38 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>>>
>>>>> On Thu, Oct 29, 2009 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>>>>
>>>>>> On Thu, Oct 29, 2009 at 1:18 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>>>> > I have a freshly started 3-node cluster with a replication factor of
>>>>>> > 2. If I take down two nodes, I can no longer do any writes, even with
>>>>>> > a consistency level of one. I tried on a variety of keys to ensure
>>>>>> > that I'd get at least one where the live node was responsible for one
>>>>>> > of the replicas. I have not yet tried on trunk. On cassandra 0.4.1,
>>>>>> > I get an UnavailableException.
>>>>>>
>>>>>> This sounds like the bug we fixed in CASSANDRA-496 on trunk.
>>>>>
>>>>> Excellent - thanks. Time to start using trunk.
>>>>>
>>>>>>
>>>>>> > Along the same lines, how does Cassandra handle network partitioning
>>>>>> > where 2 writes for the same keys hit 2 different partitions, neither
>>>>>> > of which are able to form a quorum? Dynamo maintained version vectors
>>>>>> > and put the burden on the client to resolve conflicts, but there's no
>>>>>> > similar interface in the thrift api.
>>>>>>
>>>>>> If you use QUORUM or ALL consistency, neither write will succeed. If
>>>>>> you use ONE, both will, and the one with the higher timestamp will
>>>>>> "win" when the partition heals.
>>>>>
>>>>> Got it.
>>>>>
>>>>>>
>>>>>> -Jonathan
>>>>>
>>>>> Edmond
>>>>
>>>
>>
>
Re: can't write with consistency level of one after some nodes fail
Posted by Edmond Lau <ed...@ooyala.com>.
Will do.
On Thu, Oct 29, 2009 at 2:39 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> can you create a ticket for this?
>
> On Thu, Oct 29, 2009 at 3:39 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> I've seen that bug too. I think we introduced a regression with CASSANDRA-492.
>>
>> On Thu, Oct 29, 2009 at 3:37 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>> I've updated to trunk, and I'm still hitting the same issue but it's
>>> manifesting itself differently. Again, I'm running with a freshly
>>> started 3-node cluster with a replication factor of 2. I then take
>>> down two nodes.
>>>
>>> If I write with a consistency level of ONE on any key, I get an
>>> InvalidRequestException:
>>>
>>> ERROR [pool-1-thread-45] 2009-10-29 21:27:10,120 StorageProxy.java
>>> (line 183) error writing key 1
>>> InvalidRequestException(why:Cannot block for less than one replica)
>>> at org.apache.cassandra.service.QuorumResponseHandler.<init>(QuorumResponseHandler.java:52)
>>> at org.apache.cassandra.locator.AbstractReplicationStrategy.getResponseHandler(AbstractReplicationStrategy.java:64)
>>> at org.apache.cassandra.service.StorageService.getResponseHandler(StorageService.java:869)
>>> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:162)
>>> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:473)
>>> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:424)
>>> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:819)
>>> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
>>> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>>
>>> Oddly, a write with a consistency level of QUORUM succeeds for certain
>>> keys (but fails with others) even though I only have one live node.
>>>
>>> Edmond
>>>
>>> On Thu, Oct 29, 2009 at 1:38 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>>
>>>> On Thu, Oct 29, 2009 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>>>
>>>>> On Thu, Oct 29, 2009 at 1:18 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>>> > I have a freshly started 3-node cluster with a replication factor of
>>>>> > 2. If I take down two nodes, I can no longer do any writes, even with
>>>>> > a consistency level of one. I tried on a variety of keys to ensure
>>>>> > that I'd get at least one where the live node was responsible for one
>>>>> > of the replicas. I have not yet tried on trunk. On cassandra 0.4.1,
>>>>> > I get an UnavailableException.
>>>>>
>>>>> This sounds like the bug we fixed in CASSANDRA-496 on trunk.
>>>>
>>>> Excellent - thanks. Time to start using trunk.
>>>>
>>>>>
>>>>> > Along the same lines, how does Cassandra handle network partitioning
>>>>> > where 2 writes for the same keys hit 2 different partitions, neither
>>>>> > of which are able to form a quorum? Dynamo maintained version vectors
>>>>> > and put the burden on the client to resolve conflicts, but there's no
>>>>> > similar interface in the thrift api.
>>>>>
>>>>> If you use QUORUM or ALL consistency, neither write will succeed. If
>>>>> you use ONE, both will, and the one with the higher timestamp will
>>>>> "win" when the partition heals.
>>>>
>>>> Got it.
>>>>
>>>>>
>>>>> -Jonathan
>>>>
>>>> Edmond
>>>
>>
>
Re: can't write with consistency level of one after some nodes fail
Posted by Jonathan Ellis <jb...@gmail.com>.
can you create a ticket for this?
On Thu, Oct 29, 2009 at 3:39 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> I've seen that bug too. I think we introduced a regression with CASSANDRA-492.
>
> On Thu, Oct 29, 2009 at 3:37 PM, Edmond Lau <ed...@ooyala.com> wrote:
>> I've updated to trunk, and I'm still hitting the same issue but it's
>> manifesting itself differently. Again, I'm running with a freshly
>> started 3-node cluster with a replication factor of 2. I then take
>> down two nodes.
>>
>> If I write with a consistency level of ONE on any key, I get an
>> InvalidRequestException:
>>
>> ERROR [pool-1-thread-45] 2009-10-29 21:27:10,120 StorageProxy.java
>> (line 183) error writing key 1
>> InvalidRequestException(why:Cannot block for less than one replica)
>> at org.apache.cassandra.service.QuorumResponseHandler.<init>(QuorumResponseHandler.java:52)
>> at org.apache.cassandra.locator.AbstractReplicationStrategy.getResponseHandler(AbstractReplicationStrategy.java:64)
>> at org.apache.cassandra.service.StorageService.getResponseHandler(StorageService.java:869)
>> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:162)
>> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:473)
>> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:424)
>> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:819)
>> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
>> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> Oddly, a write with a consistency level of QUORUM succeeds for certain
>> keys (but fails with others) even though I only have one live node.
>>
>> Edmond
>>
>> On Thu, Oct 29, 2009 at 1:38 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>
>>> On Thu, Oct 29, 2009 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>>
>>>> On Thu, Oct 29, 2009 at 1:18 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>>> > I have a freshly started 3-node cluster with a replication factor of
>>>> > 2. If I take down two nodes, I can no longer do any writes, even with
>>>> > a consistency level of one. I tried on a variety of keys to ensure
>>>> > that I'd get at least one where the live node was responsible for one
>>>> > of the replicas. I have not yet tried on trunk. On cassandra 0.4.1,
>>>> > I get an UnavailableException.
>>>>
>>>> This sounds like the bug we fixed in CASSANDRA-496 on trunk.
>>>
>>> Excellent - thanks. Time to start using trunk.
>>>
>>>>
>>>> > Along the same lines, how does Cassandra handle network partitioning
>>>> > where 2 writes for the same keys hit 2 different partitions, neither
>>>> > of which are able to form a quorum? Dynamo maintained version vectors
>>>> > and put the burden on the client to resolve conflicts, but there's no
>>>> > similar interface in the thrift api.
>>>>
>>>> If you use QUORUM or ALL consistency, neither write will succeed. If
>>>> you use ONE, both will, and the one with the higher timestamp will
>>>> "win" when the partition heals.
>>>
>>> Got it.
>>>
>>>>
>>>> -Jonathan
>>>
>>> Edmond
>>
>
Re: can't write with consistency level of one after some nodes fail
Posted by Jonathan Ellis <jb...@gmail.com>.
I've seen that bug too. I think we introduced a regression with CASSANDRA-492.
On Thu, Oct 29, 2009 at 3:37 PM, Edmond Lau <ed...@ooyala.com> wrote:
> I've updated to trunk, and I'm still hitting the same issue but it's
> manifesting itself differently. Again, I'm running with a freshly
> started 3-node cluster with a replication factor of 2. I then take
> down two nodes.
>
> If I write with a consistency level of ONE on any key, I get an
> InvalidRequestException:
>
> ERROR [pool-1-thread-45] 2009-10-29 21:27:10,120 StorageProxy.java
> (line 183) error writing key 1
> InvalidRequestException(why:Cannot block for less than one replica)
> at org.apache.cassandra.service.QuorumResponseHandler.<init>(QuorumResponseHandler.java:52)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getResponseHandler(AbstractReplicationStrategy.java:64)
> at org.apache.cassandra.service.StorageService.getResponseHandler(StorageService.java:869)
> at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:162)
> at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:473)
> at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:424)
> at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:819)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
> Oddly, a write with a consistency level of QUORUM succeeds for certain
> keys (but fails with others) even though I only have one live node.
>
> Edmond
>
> On Thu, Oct 29, 2009 at 1:38 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>
>> On Thu, Oct 29, 2009 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>
>>> On Thu, Oct 29, 2009 at 1:18 PM, Edmond Lau <ed...@ooyala.com> wrote:
>>> > I have a freshly started 3-node cluster with a replication factor of
>>> > 2. If I take down two nodes, I can no longer do any writes, even with
>>> > a consistency level of one. I tried on a variety of keys to ensure
>>> > that I'd get at least one where the live node was responsible for one
>>> > of the replicas. I have not yet tried on trunk. On cassandra 0.4.1,
>>> > I get an UnavailableException.
>>>
>>> This sounds like the bug we fixed in CASSANDRA-496 on trunk.
>>
>> Excellent - thanks. Time to start using trunk.
>>
>>>
>>> > Along the same lines, how does Cassandra handle network partitioning
>>> > where 2 writes for the same keys hit 2 different partitions, neither
>>> > of which are able to form a quorum? Dynamo maintained version vectors
>>> > and put the burden on the client to resolve conflicts, but there's no
>>> > similar interface in the thrift api.
>>>
>>> If you use QUORUM or ALL consistency, neither write will succeed. If
>>> you use ONE, both will, and the one with the higher timestamp will
>>> "win" when the partition heals.
>>
>> Got it.
>>
>>>
>>> -Jonathan
>>
>> Edmond
>
Re: can't write with consistency level of one after some nodes fail
Posted by Edmond Lau <ed...@ooyala.com>.
I've updated to trunk, and I'm still hitting the same issue but it's
manifesting itself differently. Again, I'm running with a freshly
started 3-node cluster with a replication factor of 2. I then take
down two nodes.
If I write with a consistency level of ONE on any key, I get an
InvalidRequestException:
ERROR [pool-1-thread-45] 2009-10-29 21:27:10,120 StorageProxy.java
(line 183) error writing key 1
InvalidRequestException(why:Cannot block for less than one replica)
at org.apache.cassandra.service.QuorumResponseHandler.<init>(QuorumResponseHandler.java:52)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getResponseHandler(AbstractReplicationStrategy.java:64)
at org.apache.cassandra.service.StorageService.getResponseHandler(StorageService.java:869)
at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:162)
at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:473)
at org.apache.cassandra.service.CassandraServer.insert(CassandraServer.java:424)
at org.apache.cassandra.service.Cassandra$Processor$insert.process(Cassandra.java:819)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Oddly, a write with a consistency level of QUORUM succeeds for certain
keys (but fails with others) even though I only have one live node.
Edmond
On Thu, Oct 29, 2009 at 1:38 PM, Edmond Lau <ed...@ooyala.com> wrote:
>
> On Thu, Oct 29, 2009 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> On Thu, Oct 29, 2009 at 1:18 PM, Edmond Lau <ed...@ooyala.com> wrote:
>> > I have a freshly started 3-node cluster with a replication factor of
>> > 2. If I take down two nodes, I can no longer do any writes, even with
>> > a consistency level of one. I tried on a variety of keys to ensure
>> > that I'd get at least one where the live node was responsible for one
>> > of the replicas. I have not yet tried on trunk. On cassandra 0.4.1,
>> > I get an UnavailableException.
>>
>> This sounds like the bug we fixed in CASSANDRA-496 on trunk.
>
> Excellent - thanks. Time to start using trunk.
>
>>
>> > Along the same lines, how does Cassandra handle network partitioning
>> > where 2 writes for the same keys hit 2 different partitions, neither
>> > of which are able to form a quorum? Dynamo maintained version vectors
>> > and put the burden on the client to resolve conflicts, but there's no
>> > similar interface in the thrift api.
>>
>> If you use QUORUM or ALL consistency, neither write will succeed. If
>> you use ONE, both will, and the one with the higher timestamp will
>> "win" when the partition heals.
>
> Got it.
>
>>
>> -Jonathan
>
> Edmond
Re: can't write with consistency level of one after some nodes fail
Posted by Edmond Lau <ed...@ooyala.com>.
On Thu, Oct 29, 2009 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Thu, Oct 29, 2009 at 1:18 PM, Edmond Lau <ed...@ooyala.com> wrote:
> > I have a freshly started 3-node cluster with a replication factor of
> > 2. If I take down two nodes, I can no longer do any writes, even with
> > a consistency level of one. I tried on a variety of keys to ensure
> > that I'd get at least one where the live node was responsible for one
> > of the replicas. I have not yet tried on trunk. On cassandra 0.4.1,
> > I get an UnavailableException.
>
> This sounds like the bug we fixed in CASSANDRA-496 on trunk.
>
Excellent - thanks. Time to start using trunk.
>
> > Along the same lines, how does Cassandra handle network partitioning
> > where 2 writes for the same keys hit 2 different partitions, neither
> > of which are able to form a quorum? Dynamo maintained version vectors
> > and put the burden on the client to resolve conflicts, but there's no
> > similar interface in the thrift api.
>
> If you use QUORUM or ALL consistency, neither write will succeed. If
> you use ONE, both will, and the one with the higher timestamp will
> "win" when the partition heals.
>
Got it.
>
> -Jonathan
>
Edmond
Re: can't write with consistency level of one after some nodes fail
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Oct 29, 2009 at 1:18 PM, Edmond Lau <ed...@ooyala.com> wrote:
> I have a freshly started 3-node cluster with a replication factor of
> 2. If I take down two nodes, I can no longer do any writes, even with
> a consistency level of one. I tried on a variety of keys to ensure
> that I'd get at least one where the live node was responsible for one
> of the replicas. I have not yet tried on trunk. On cassandra 0.4.1,
> I get an UnavailableException.
This sounds like the bug we fixed in CASSANDRA-496 on trunk.
> Along the same lines, how does Cassandra handle network partitioning
> where 2 writes for the same keys hit 2 different partitions, neither
> of which are able to form a quorum? Dynamo maintained version vectors
> and put the burden on the client to resolve conflicts, but there's no
> similar interface in the thrift api.
If you use QUORUM or ALL consistency, neither write will succeed. If
you use ONE, both will, and the one with the higher timestamp will
"win" when the partition heals.
-Jonathan