You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by William Oberman <ob...@civicscience.com> on 2011/04/20 23:59:56 UTC
system_* consistency level?
Hi,
My unit tests started failing once I upgraded from a single node cassandra
cluster to a full "N" node cluster (I'm starting with 4). I had a few
various bugs, mostly due to forgetting to read/write at a quorum level in
places I needed stronger consistency guarantees. But, I kept getting
random, intermittent failure (the worst kind). I'm 99% sure I see why,
after some painful debugging, but I don't know what to do about it. The
basic flaw in my understanding of cassandra seems to boil down to: I thought
system mutations of keyspaces/column families where of a stronger
consistency than ONE, but that appears to not be true. Any way for me to
update a cluster at something more like QUORUM?
The basic idea is in my unit test.setup() I clone my real keyspace as
keyspace_UUID (with all of the exact same CFs) to get a fresh space to play
in. In a single node environment, no issues. But, in a cluster, it seems
that it takes a while for the system_add_keyspace call to propagate. No
worries I think, I just modify my setup() to do
describe_keyspace(keyspace_UUID) in a while loop until the cluster is
ready. My random failures drop considerably, but every once and awhile I
see a similar kind of failure. Then I find out that schema updates seem to
propagate on a per node basis. At least, that's what I have to assume as
I'm using phpcassa which uses a connection pool, and I see in my logging
that my setup() succeeds because one connection in the pool sees the new
keyspace, but when my tests run I grab a connection from the pool that is
missing it!
Do I have a solution other than changing my setup yet again to loop over all
cassandra servers doing a describe_keyspace()?
--
Will Oberman
Civic Science, Inc.
3030 Penn Avenue., First Floor
Pittsburgh, PA 15201
(M) 412-480-7835
(E) oberman@civicscience.com
Re: system_* consistency level?
Posted by William Oberman <ob...@civicscience.com>.
That was the trick. Thanks!
On Apr 20, 2011, at 6:05 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> See the comments for describe_schema_versions.
>
> On Wed, Apr 20, 2011 at 4:59 PM, William Oberman
> <ob...@civicscience.com> wrote:
>> Hi,
>>
>> My unit tests started failing once I upgraded from a single node cassandra
>> cluster to a full "N" node cluster (I'm starting with 4). I had a few
>> various bugs, mostly due to forgetting to read/write at a quorum level in
>> places I needed stronger consistency guarantees. But, I kept getting
>> random, intermittent failure (the worst kind). I'm 99% sure I see why,
>> after some painful debugging, but I don't know what to do about it. The
>> basic flaw in my understanding of cassandra seems to boil down to: I thought
>> system mutations of keyspaces/column families where of a stronger
>> consistency than ONE, but that appears to not be true. Any way for me to
>> update a cluster at something more like QUORUM?
>>
>> The basic idea is in my unit test.setup() I clone my real keyspace as
>> keyspace_UUID (with all of the exact same CFs) to get a fresh space to play
>> in. In a single node environment, no issues. But, in a cluster, it seems
>> that it takes a while for the system_add_keyspace call to propagate. No
>> worries I think, I just modify my setup() to do
>> describe_keyspace(keyspace_UUID) in a while loop until the cluster is
>> ready. My random failures drop considerably, but every once and awhile I
>> see a similar kind of failure. Then I find out that schema updates seem to
>> propagate on a per node basis. At least, that's what I have to assume as
>> I'm using phpcassa which uses a connection pool, and I see in my logging
>> that my setup() succeeds because one connection in the pool sees the new
>> keyspace, but when my tests run I grab a connection from the pool that is
>> missing it!
>>
>> Do I have a solution other than changing my setup yet again to loop over all
>> cassandra servers doing a describe_keyspace()?
>>
>> --
>> Will Oberman
>> Civic Science, Inc.
>> 3030 Penn Avenue., First Floor
>> Pittsburgh, PA 15201
>> (M) 412-480-7835
>> (E) oberman@civicscience.com
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: system_* consistency level?
Posted by Jonathan Ellis <jb...@gmail.com>.
See the comments for describe_schema_versions.
On Wed, Apr 20, 2011 at 4:59 PM, William Oberman
<ob...@civicscience.com> wrote:
> Hi,
>
> My unit tests started failing once I upgraded from a single node cassandra
> cluster to a full "N" node cluster (I'm starting with 4). I had a few
> various bugs, mostly due to forgetting to read/write at a quorum level in
> places I needed stronger consistency guarantees. But, I kept getting
> random, intermittent failure (the worst kind). I'm 99% sure I see why,
> after some painful debugging, but I don't know what to do about it. The
> basic flaw in my understanding of cassandra seems to boil down to: I thought
> system mutations of keyspaces/column families where of a stronger
> consistency than ONE, but that appears to not be true. Any way for me to
> update a cluster at something more like QUORUM?
>
> The basic idea is in my unit test.setup() I clone my real keyspace as
> keyspace_UUID (with all of the exact same CFs) to get a fresh space to play
> in. In a single node environment, no issues. But, in a cluster, it seems
> that it takes a while for the system_add_keyspace call to propagate. No
> worries I think, I just modify my setup() to do
> describe_keyspace(keyspace_UUID) in a while loop until the cluster is
> ready. My random failures drop considerably, but every once and awhile I
> see a similar kind of failure. Then I find out that schema updates seem to
> propagate on a per node basis. At least, that's what I have to assume as
> I'm using phpcassa which uses a connection pool, and I see in my logging
> that my setup() succeeds because one connection in the pool sees the new
> keyspace, but when my tests run I grab a connection from the pool that is
> missing it!
>
> Do I have a solution other than changing my setup yet again to loop over all
> cassandra servers doing a describe_keyspace()?
>
> --
> Will Oberman
> Civic Science, Inc.
> 3030 Penn Avenue., First Floor
> Pittsburgh, PA 15201
> (M) 412-480-7835
> (E) oberman@civicscience.com
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com