You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by André Cruz <an...@co.sapo.pt> on 2013/03/10 17:00:06 UTC
Quorum read after quorum write guarantee
Hello.
In my application it sometimes happens that I execute a multiget (I use pycassa) to fetch data that I have just inserted. I use quorum writes and reads, and my RF is 3.
I've noticed that sometimes (1 in 1000 perhaps) an insert followed (300ms after) by a multiget will not find the just inserted data. Is this normal? Or is something wrong? Can there be some delay to obtain the inserted data even with quorum?
Best regards,
André
Re: Quorum read after quorum write guarantee
Posted by André Cruz <an...@co.sapo.pt>.
Yes, same thread. Cassandra 1.1.5 btw.
Sent from my iPhone
On 10/03/2013, at 16:47, Dave Brosius <db...@mebigfatguy.com> wrote:
> is the read and write happening on the same thread?
>
> On 03/10/2013 12:00 PM, André Cruz wrote:
>> Hello.
>>
>> In my application it sometimes happens that I execute a multiget (I use pycassa) to fetch data that I have just inserted. I use quorum writes and reads, and my RF is 3.
>>
>> I've noticed that sometimes (1 in 1000 perhaps) an insert followed (300ms after) by a multiget will not find the just inserted data. Is this normal? Or is something wrong? Can there be some delay to obtain the inserted data even with quorum?
>>
>> Best regards,
>> André
>
Re: Quorum read after quorum write guarantee
Posted by Dave Brosius <db...@mebigfatguy.com>.
is the read and write happening on the same thread?
On 03/10/2013 12:00 PM, André Cruz wrote:
> Hello.
>
> In my application it sometimes happens that I execute a multiget (I use pycassa) to fetch data that I have just inserted. I use quorum writes and reads, and my RF is 3.
>
> I've noticed that sometimes (1 in 1000 perhaps) an insert followed (300ms after) by a multiget will not find the just inserted data. Is this normal? Or is something wrong? Can there be some delay to obtain the inserted data even with quorum?
>
> Best regards,
> André
Re: Quorum read after quorum write guarantee
Posted by aaron morton <aa...@thelastpickle.com>.
>> If you are doing reads and writes using QUOURM double check that your code is correct. If it is provide some more info on what you are seeing.
>
> It seems correct. What more info could be of value?
Can you reproduce the fault in a small script/program?
We have a pretty solid history of Quourm / Quourm operations exhibiting strong consistency, so if there is a problem we'll need a way to reproduce it.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 12/03/2013, at 2:31 AM, André Cruz <an...@co.sapo.pt> wrote:
> On Mar 12, 2013, at 6:04 AM, aaron morton <aa...@thelastpickle.com> wrote:
>
>>> by a multiget will not find the just inserted data.
>> Can you explain how the data is not found.
>> Does it not find new columns or does it return stale columns ?
>
> It does not find new columns, I don't overwrite data.
>
>> If the read is run again does it return the expected value?
>
> Yes, when I go and check manually after this happens, the data is already there. However, I think I saw an instance when the multiget query was executed twice in quick succession, the first time it returned all the results, the second time it didn't. This only happened once, so I filed it under X-Files.
>
>> if you are getting stale data double check the the nodes / clients have their clocks synchronised.
>
> That's a good tip, I'll check.
>
>> If you are doing reads and writes using QUOURM double check that your code is correct. If it is provide some more info on what you are seeing.
>
> It seems correct. What more info could be of value?
>
> Thanks,
> André
>
Re: Quorum read after quorum write guarantee
Posted by André Cruz <an...@co.sapo.pt>.
On Mar 12, 2013, at 6:04 AM, aaron morton <aa...@thelastpickle.com> wrote:
>> by a multiget will not find the just inserted data.
> Can you explain how the data is not found.
> Does it not find new columns or does it return stale columns ?
It does not find new columns, I don't overwrite data.
> If the read is run again does it return the expected value?
Yes, when I go and check manually after this happens, the data is already there. However, I think I saw an instance when the multiget query was executed twice in quick succession, the first time it returned all the results, the second time it didn't. This only happened once, so I filed it under X-Files.
> if you are getting stale data double check the the nodes / clients have their clocks synchronised.
That's a good tip, I'll check.
> If you are doing reads and writes using QUOURM double check that your code is correct. If it is provide some more info on what you are seeing.
It seems correct. What more info could be of value?
Thanks,
André
Re: Quorum read after quorum write guarantee
Posted by aaron morton <aa...@thelastpickle.com>.
> by a multiget will not find the just inserted data.
Can you explain how the data is not found.
Does it not find new columns or does it return stale columns ?
If the read is run again does it return the expected value?
if you are getting stale data double check the the nodes / clients have their clocks synchronised.
If you are doing reads and writes using QUOURM double check that your code is correct. If it is provide some more info on what you are seeing.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 11/03/2013, at 12:50 PM, André Cruz <an...@co.sapo.pt> wrote:
> On Mar 11, 2013, at 5:02 PM, Tyler Hobbs <ty...@datastax.com> wrote:
>
>> What kind of inserts and multiget queries are you running?
>
> I use the ColumnFamily objects. The pool is initialised with "write_consistency_level=ConsistencyLevel.QUORUM".
>
> The insert is a regular insert, so the QUORUM is used. When fetching I use:
>
> CF.multiget(blocks, columns=['size'], read_consistency_level=ConsistencyLevel.QUORUM)
>
> I have 6 nodes and a RF 3.
>
> André
>
>
Re: Quorum read after quorum write guarantee
Posted by André Cruz <an...@co.sapo.pt>.
On Mar 11, 2013, at 5:02 PM, Tyler Hobbs <ty...@datastax.com> wrote:
> What kind of inserts and multiget queries are you running?
I use the ColumnFamily objects. The pool is initialised with "write_consistency_level=ConsistencyLevel.QUORUM".
The insert is a regular insert, so the QUORUM is used. When fetching I use:
CF.multiget(blocks, columns=['size'], read_consistency_level=ConsistencyLevel.QUORUM)
I have 6 nodes and a RF 3.
André
Re: Quorum read after quorum write guarantee
Posted by Tyler Hobbs <ty...@datastax.com>.
What kind of inserts and multiget queries are you running?
On Sun, Mar 10, 2013 at 1:16 PM, André Cruz <an...@co.sapo.pt> wrote:
> On 10/03/2013, at 16:49, Chuan-Heng Hsiao <hs...@gmail.com>
> wrote:
> > However, my guess is that cassandra only guarantee that
> > if you successfully write and you successfully read, then quorum will
> > give you the latest data.
>
> That's what I thought, but that's not what I'm seeing all the time. I have
> no errors reading or writing.
>
> André
--
Tyler Hobbs
DataStax <http://datastax.com/>
Re: Quorum read after quorum write guarantee
Posted by André Cruz <an...@co.sapo.pt>.
On 10/03/2013, at 16:49, Chuan-Heng Hsiao <hs...@gmail.com> wrote:
> However, my guess is that cassandra only guarantee that
> if you successfully write and you successfully read, then quorum will
> give you the latest data.
That's what I thought, but that's not what I'm seeing all the time. I have no errors reading or writing.
André
Re: Quorum read after quorum write guarantee
Posted by Chuan-Heng Hsiao <hs...@gmail.com>.
Hi André,
I am just a user of cassandra and did not look into the code deeply.
However, my guess is that cassandra only guarantee that
if you successfully write and you successfully read, then quorum will
give you the latest data.
not finding the just inserted data may be due to the failure of
successfully write or the failure of successfully read.
if it is the failure of successfully write, then pycassa should give
you some error message.
if it is the failure of successfully read, then pycassa may give you
timeout error or not-found error.
To test it fairly, it is better to setup the scenario as continuously
write/read same key/column name,
but with different column value. Then you should be able to see that
when you read,
either you read the latest data (successful write/read), or you read
the old data (fail write/success read),
or you read nothing (fail read)
Hopefully my guess is on the right track, and hopefully you understand
what I mean.
Sincerely,
Hsiao
On Mon, Mar 11, 2013 at 12:00 AM, André Cruz <an...@co.sapo.pt> wrote:
> Hello.
>
> In my application it sometimes happens that I execute a multiget (I use pycassa) to fetch data that I have just inserted. I use quorum writes and reads, and my RF is 3.
>
> I've noticed that sometimes (1 in 1000 perhaps) an insert followed (300ms after) by a multiget will not find the just inserted data. Is this normal? Or is something wrong? Can there be some delay to obtain the inserted data even with quorum?
>
> Best regards,
> André