You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kevin Burton <bu...@spinn3r.com> on 2015/01/18 16:02:52 UTC
"Not enough replica available” when consistency is ONE?
I’m really confused here.
I”m calling:
acquireInsert.setConsistencyLevel( ConsistencyLevel.ONE );
but I”m still getting the exception:
com.datastax.driver.core.exceptions.UnavailableException: Not enough
replica available for query at consistency SERIAL (2 required but only 1
alive)
Does it matter that I’m using:
ifNotExists();
and that maybe cassandra needs two because it’s using a coordinator ?
If so then an exception should probably be thrown when I try to set a wrong
consistency level.
which would be weird because I *do* have at least two replicas online. I
have 4 nodes in my cluster right now...
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: "Not enough replica available” when consistency is ONE?
Posted by Panagiotis Garefalakis <pa...@gmail.com>.
Hello,
My feeling is you got the hole CAS operations concept wrong. CAS operations
a.k.a lightweight transactions are not meant to used everywhere, only in
specific parts of your application where serialisable consistency is
necessary. For any other case there is a variety of consistency levels you
could use.
Check this article which helped me a lot:
http://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0
You didn't mention the replication factor of your CF but my feeling is you
are using RF=2 which is also not a good choice because you cannot afford a
single node failure!
http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html
Cheers,
Panagiotis
On Mon, Jan 19, 2015 at 1:29 AM, Kevin Burton <bu...@spinn3r.com> wrote:
> OK.. so if I’m running with 2 replicas, then BOTH of them need to be
> online for this to work. Correct? Because with two replicas I need 2 to
> form a quorum.
>
> This is somewhat confusing them. Because if you have two replicas, and
> you’re depending on these types of transactions, then this is a VERY
> dangerous state. Because if ANY of your Cassandra nodes goes offline, then
> your entire application crashes. So the more nodes you have, the HIGHER
> the probability that your application will crash.
>
> Which is just what happened to me. And in retrospect, this makes total
> sense, but of course I just missed this in the application design.
>
> So ConsistencyLevel.ONE and if not exists are essentially mutually
> incompatible and shouldn’t the driver throw an exception if the user
> requests this configuration?
>
> Its dangerous enough that it probably shouldn’t be supported.
>
>
>
> On Sun, Jan 18, 2015 at 7:43 AM, Eric Stevens <mi...@gmail.com> wrote:
>
>> Check out
>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_tunable_consistency_c.html
>>
>> > Cassandra 2.0 uses the Paxos consensus protocol, which resembles
>> 2-phase commit, to support linearizable consistency. All operations are
>> quorum-based ...
>>
>> This kicks in whenever you do CAS operations (eg, IF NOT EXISTS).
>> Otherwise a cluster which became network partitioned would end up being
>> able to have two separate CAS statements which both succeeded, but which
>> disagreed with each other.
>>
>> On Sun, Jan 18, 2015 at 8:02 AM, Kevin Burton <bu...@spinn3r.com> wrote:
>>
>>> I’m really confused here.
>>>
>>> I”m calling:
>>>
>>> acquireInsert.setConsistencyLevel( ConsistencyLevel.ONE );
>>>
>>> but I”m still getting the exception:
>>>
>>> com.datastax.driver.core.exceptions.UnavailableException: Not enough
>>> replica available for query at consistency SERIAL (2 required but only 1
>>> alive)
>>>
>>> Does it matter that I’m using:
>>>
>>> ifNotExists();
>>>
>>> and that maybe cassandra needs two because it’s using a coordinator ?
>>>
>>> If so then an exception should probably be thrown when I try to set a
>>> wrong consistency level.
>>>
>>> which would be weird because I *do* have at least two replicas online. I
>>> have 4 nodes in my cluster right now...
>>>
>>> --
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> <https://plus.google.com/102718274791889610666/posts>
>>> <http://spinn3r.com>
>>>
>>>
>>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
Re: "Not enough replica available” when consistency is ONE?
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Mon, Jan 19, 2015 at 2:29 AM, Kevin Burton <bu...@spinn3r.com> wrote:
> So ConsistencyLevel.ONE and if not exists are essentially mutually
> incompatible and shouldn’t the driver throw an exception if the user
> requests this configuration?
>
The subtlety is that this consistency level (CL.ONE in your case) is
actually
used by conditional operations (aka CAS operations, i.e. any insert/update
with
a 'IF'). For those operations there is in fact 2 consistency level that is
taken into account: the serial consistency level, that your driver should
allow
you to set and for which there is really only a choice between SERIAL and
LOCAL_SERIAL, and the usual consistency level, the one you've set to ONE.
There
is 2 phases (I'm simplifying) to CAS operations and each CL apply to one of
them: the first phase is the serial phase and the so-called serial
consistency
applies to it. For that phase you basically need a quorum of nodes (or a
local quorum if you use LOCAL_SERIAL) and it's because this phase couldn't
be
achieved that you got your exception. After this phase, the write has
been decided (nodes have agreed on committing it if you will) but it may
not be
visible to non-serial reads just yet because the actual write may not have
been
committed to all nodes. It's to this 2nd "committing" phase that the
"normal"
consistency level applies. Concretely, it means that if you do a CAS write
with
a normal CL of ONE, and you then do a read with CL.ONE, you're not
guaranteed
to see your write right away. But if you use QUORUM, then a QORUM read
guarantees you to see that write.
Anyway, my point being that it wouldn't make sense for the driver to throw
an
exception since there is nothing wrong in practice. But it is true that
users
needs to understand how conditional updates differ from normal updates to
avoid
surprises.
--
Sylvain
Re: "Not enough replica available” when consistency is ONE?
Posted by Kevin Burton <bu...@spinn3r.com>.
OK.. so if I’m running with 2 replicas, then BOTH of them need to be online
for this to work. Correct? Because with two replicas I need 2 to form a
quorum.
This is somewhat confusing them. Because if you have two replicas, and
you’re depending on these types of transactions, then this is a VERY
dangerous state. Because if ANY of your Cassandra nodes goes offline, then
your entire application crashes. So the more nodes you have, the HIGHER
the probability that your application will crash.
Which is just what happened to me. And in retrospect, this makes total
sense, but of course I just missed this in the application design.
So ConsistencyLevel.ONE and if not exists are essentially mutually
incompatible and shouldn’t the driver throw an exception if the user
requests this configuration?
Its dangerous enough that it probably shouldn’t be supported.
On Sun, Jan 18, 2015 at 7:43 AM, Eric Stevens <mi...@gmail.com> wrote:
> Check out
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_tunable_consistency_c.html
>
> > Cassandra 2.0 uses the Paxos consensus protocol, which resembles
> 2-phase commit, to support linearizable consistency. All operations are
> quorum-based ...
>
> This kicks in whenever you do CAS operations (eg, IF NOT EXISTS).
> Otherwise a cluster which became network partitioned would end up being
> able to have two separate CAS statements which both succeeded, but which
> disagreed with each other.
>
> On Sun, Jan 18, 2015 at 8:02 AM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> I’m really confused here.
>>
>> I”m calling:
>>
>> acquireInsert.setConsistencyLevel( ConsistencyLevel.ONE );
>>
>> but I”m still getting the exception:
>>
>> com.datastax.driver.core.exceptions.UnavailableException: Not enough
>> replica available for query at consistency SERIAL (2 required but only 1
>> alive)
>>
>> Does it matter that I’m using:
>>
>> ifNotExists();
>>
>> and that maybe cassandra needs two because it’s using a coordinator ?
>>
>> If so then an exception should probably be thrown when I try to set a
>> wrong consistency level.
>>
>> which would be weird because I *do* have at least two replicas online. I
>> have 4 nodes in my cluster right now...
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: "Not enough replica available” when consistency is ONE?
Posted by Eric Stevens <mi...@gmail.com>.
Check out
http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_tunable_consistency_c.html
> Cassandra 2.0 uses the Paxos consensus protocol, which resembles 2-phase
commit, to support linearizable consistency. All operations are quorum-based
...
This kicks in whenever you do CAS operations (eg, IF NOT EXISTS).
Otherwise a cluster which became network partitioned would end up being
able to have two separate CAS statements which both succeeded, but which
disagreed with each other.
On Sun, Jan 18, 2015 at 8:02 AM, Kevin Burton <bu...@spinn3r.com> wrote:
> I’m really confused here.
>
> I”m calling:
>
> acquireInsert.setConsistencyLevel( ConsistencyLevel.ONE );
>
> but I”m still getting the exception:
>
> com.datastax.driver.core.exceptions.UnavailableException: Not enough
> replica available for query at consistency SERIAL (2 required but only 1
> alive)
>
> Does it matter that I’m using:
>
> ifNotExists();
>
> and that maybe cassandra needs two because it’s using a coordinator ?
>
> If so then an exception should probably be thrown when I try to set a
> wrong consistency level.
>
> which would be weird because I *do* have at least two replicas online. I
> have 4 nodes in my cluster right now...
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>