You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Rick Whitesel (rwhitese)" <rw...@cisco.com> on 2011/09/26 20:55:28 UTC
Update of column sometimes takes 10 seconds
Hi All:
We have a simple junit test that inserts a column, immediately updates
that column and then validates that the data updated. Cassandra is run
embedded in the unit test. Sometimes the test will pass, i.e. the
updated data is correct, and sometimes the test will fail. The
configuration is set to:
<CommitLogSync>periodic</CommitLogSync> and
<CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
We are running version 0.6.9. We plan to update to the latest version
but cannot until after the release we are wrapping up. We are using the
client batch mutate to create and update the data. From what I
understand, the commit log write will return immediately and the data
will be store in memory. If that is the case, then why would our test
sometimes fail?
-Rick Whitesel
Simplify, Standardize and Conserve
Rick Whitesel
Technical Lead
Customer Contact Business Unit
rwhitese@cisco.com <ma...@cisco.com>
Phone :978-936-0479
500 Beaver Brook Road
Boxboro, MA 01719
Mailing address:
1414 Massachusetts Avenue
Boxboro, MA 01719
United States
www.cisco.com <http://www.cisco.com/>
Re: Update of column sometimes takes 10 seconds
Posted by aaron morton <aa...@thelastpickle.com>.
Try turning up the logging to DEBUG and watch the requests come through.
Check that the two inserts do indeed have different time stamps.
In cases of lost updates, timestamps are most often the cause of the kerfuffle.
btw, in this situation the commit log is a red hearing riding a scapegoat. Data is dumped into the commit log for durability at the start of the write request, but it is never read from there. Data is read from memtables and sstables.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 27/09/2011, at 9:27 AM, Rick Whitesel (rwhitese) wrote:
> Thank you to all for the quick response. The test that fails is doing a insert, another insert (to update data) and then a get to validate. If I make multiple copies of the same test and execute them in succession, different copies will fail on successive runs. Each test only has a single get, so one would assume the get in testN must be causing testN+1 to fail. It is going to take me a little bit to dig down deep enough into our code to see if that is the issue.
>
> -Rick
>
> From: Jonathan Ellis [mailto:jbellis@gmail.com]
> Sent: Monday, September 26, 2011 3:34 PM
> To: user@cassandra.apache.org
> Subject: Re: Update of column sometimes takes 10 seconds
>
> Sounds a lot like this to me: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Updates-lost-td6739612.html
>
> On Mon, Sep 26, 2011 at 1:55 PM, Rick Whitesel (rwhitese) <rw...@cisco.com> wrote:
> Hi All:
>
> We have a simple junit test that inserts a column, immediately updates that column and then validates that the data updated. Cassandra is run embedded in the unit test. Sometimes the test will pass, i.e. the updated data is correct, and sometimes the test will fail. The configuration is set to:
>
> <CommitLogSync>periodic</CommitLogSync> and <CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
>
> We are running version 0.6.9. We plan to update to the latest version but cannot until after the release we are wrapping up. We are using the client batch mutate to create and update the data. From what I understand, the commit log write will return immediately and the data will be store in memory. If that is the case, then why would our test sometimes fail?
>
> -Rick Whitesel
>
>
> Simplify, Standardize and Conserve
>
>
> <image001.gif>
>
> Rick Whitesel
> Technical Lead
> Customer Contact Business Unit
>
> rwhitese@cisco.com
> Phone :978-936-0479
>
> 500 Beaver Brook Road
> Boxboro, MA 01719
> Mailing address:
> 1414 Massachusetts Avenue
> Boxboro, MA 01719
> United States
> www.cisco.com
>
>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
RE: Update of column sometimes takes 10 seconds
Posted by "Rick Whitesel (rwhitese)" <rw...@cisco.com>.
Thank you to all for the quick response. The test that fails is doing a
insert, another insert (to update data) and then a get to validate. If I
make multiple copies of the same test and execute them in succession,
different copies will fail on successive runs. Each test only has a
single get, so one would assume the get in testN must be causing testN+1
to fail. It is going to take me a little bit to dig down deep enough
into our code to see if that is the issue.
-Rick
From: Jonathan Ellis [mailto:jbellis@gmail.com]
Sent: Monday, September 26, 2011 3:34 PM
To: user@cassandra.apache.org
Subject: Re: Update of column sometimes takes 10 seconds
Sounds a lot like this to me:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Updates
-lost-td6739612.html
On Mon, Sep 26, 2011 at 1:55 PM, Rick Whitesel (rwhitese) <
rwhitese@cisco.com> wrote:
Hi All:
We have a simple junit test that inserts a column, immediately updates
that column and then validates that the data updated. Cassandra is run
embedded in the unit test. Sometimes the test will pass, i.e. the
updated data is correct, and sometimes the test will fail. The
configuration is set to:
<CommitLogSync>periodic</CommitLogSync> and
<CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
We are running version 0.6.9. We plan to update to the latest version
but cannot until after the release we are wrapping up. We are using the
client batch mutate to create and update the data. From what I
understand, the commit log write will return immediately and the data
will be store in memory. If that is the case, then why would our test
sometimes fail?
-Rick Whitesel
Simplify, Standardize and Conserve
Rick Whitesel
Technical Lead
Customer Contact Business Unit
rwhitese@cisco.com <ma...@cisco.com>
Phone :978-936-0479
500 Beaver Brook Road
Boxboro, MA 01719
Mailing address:
1414 Massachusetts Avenue
Boxboro, MA 01719
United States
www.cisco.com <http://www.cisco.com/>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Update of column sometimes takes 10 seconds
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds a lot like this to me:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Updates-lost-td6739612.html
On Mon, Sep 26, 2011 at 1:55 PM, Rick Whitesel (rwhitese) <
rwhitese@cisco.com> wrote:
> Hi All:****
>
> ** **
>
> We have a simple junit test that inserts a column, immediately updates that
> column and then validates that the data updated. Cassandra is run embedded
> in the unit test. Sometimes the test will pass, i.e. the updated data is
> correct, and sometimes the test will fail. The configuration is set to:***
> *
>
> ** **
>
> <CommitLogSync>periodic</CommitLogSync> and
> <CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>****
>
> ** **
>
> We are running version 0.6.9. We plan to update to the latest version but
> cannot until after the release we are wrapping up. We are using the client
> batch mutate to create and update the data. From what I understand, the
> commit log write will return immediately and the data will be store in
> memory. If that is the case, then why would our test sometimes fail?****
>
> ** **
>
> -Rick Whitesel****
>
> ** **
>
> ****
>
> *Simplify, Standardize and Conserve*****
>
> ** **
>
> ** **
>
> [image: cid:01608604-FE45-42D2-AACA-4C66FCE5AB8C@cisco.com]****
>
> ** **
>
> *Rick Whitesel*
> *Technical Lead*
> *Customer Contact Business Unit**
> *
> rwhitese@cisco.com <no...@cisco.com>
> Phone :*978-936-0479*****
>
>
> 500 Beaver Brook Road
> Boxboro, MA 01719
> Mailing address:
> 1414 Massachusetts Avenue
> Boxboro, MA 01719
> United States
> www.cisco.com****
>
> ** **
>
> ** **
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
RE: Update of column sometimes takes 10 seconds
Posted by "Rick Whitesel (rwhitese)" <rw...@cisco.com>.
Waiting 10 seconds between the update and reading the updated data seems
to always work. Not waiting the 10 seconds will cause the test to
randomly pass or fail.
-Rick
From: Jim Ancona [mailto:jim@anconafamily.com]
Sent: Monday, September 26, 2011 3:04 PM
To: user@cassandra.apache.org
Subject: Re: Update of column sometimes takes 10 seconds
Do you actually see the update occur if you wait for 10 seconds (as your
subject implies), or do you just see intermittent failures when running
the unit test? If it's the latter, are you sure that the update has a
greater timestamp than the insert? I've seen similar unit tests fail
because because the timestamp values were the same.
Jim
On Mon, Sep 26, 2011 at 2:55 PM, Rick Whitesel (rwhitese) <
rwhitese@cisco.com> wrote:
Hi All:
We have a simple junit test that inserts a column, immediately updates
that column and then validates that the data updated. Cassandra is run
embedded in the unit test. Sometimes the test will pass, i.e. the
updated data is correct, and sometimes the test will fail. The
configuration is set to:
<CommitLogSync>periodic</CommitLogSync> and
<CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
We are running version 0.6.9. We plan to update to the latest version
but cannot until after the release we are wrapping up. We are using the
client batch mutate to create and update the data. From what I
understand, the commit log write will return immediately and the data
will be store in memory. If that is the case, then why would our test
sometimes fail?
-Rick Whitesel
Simplify, Standardize and Conserve
Rick Whitesel
Technical Lead
Customer Contact Business Unit
rwhitese@cisco.com <ma...@cisco.com>
Phone :978-936-0479
500 Beaver Brook Road
Boxboro, MA 01719
Mailing address:
1414 Massachusetts Avenue
Boxboro, MA 01719
United States
www.cisco.com <http://www.cisco.com/>
Re: Update of column sometimes takes 10 seconds
Posted by Jim Ancona <ji...@anconafamily.com>.
Do you actually see the update occur if you wait for 10 seconds (as your
subject implies), or do you just see intermittent failures when running the
unit test? If it's the latter, are you sure that the update has a greater
timestamp than the insert? I've seen similar unit tests fail because because
the timestamp values were the same.
Jim
On Mon, Sep 26, 2011 at 2:55 PM, Rick Whitesel (rwhitese) <
rwhitese@cisco.com> wrote:
> Hi All:****
>
> ** **
>
> We have a simple junit test that inserts a column, immediately updates that
> column and then validates that the data updated. Cassandra is run embedded
> in the unit test. Sometimes the test will pass, i.e. the updated data is
> correct, and sometimes the test will fail. The configuration is set to:***
> *
>
> ** **
>
> <CommitLogSync>periodic</CommitLogSync> and
> <CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>****
>
> ** **
>
> We are running version 0.6.9. We plan to update to the latest version but
> cannot until after the release we are wrapping up. We are using the client
> batch mutate to create and update the data. From what I understand, the
> commit log write will return immediately and the data will be store in
> memory. If that is the case, then why would our test sometimes fail?****
>
> ** **
>
> -Rick Whitesel****
>
> ** **
>
> ****
>
> *Simplify, Standardize and Conserve*****
>
> ** **
>
> ** **
>
> [image: cid:01608604-FE45-42D2-AACA-4C66FCE5AB8C@cisco.com]****
>
> ** **
>
> *Rick Whitesel*
> *Technical Lead*
> *Customer Contact Business Unit**
> *
> rwhitese@cisco.com <no...@cisco.com>
> Phone :*978-936-0479*****
>
>
> 500 Beaver Brook Road
> Boxboro, MA 01719
> Mailing address:
> 1414 Massachusetts Avenue
> Boxboro, MA 01719
> United States
> www.cisco.com****
>
> ** **
>
> ** **
>