You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Chris Eineke (JIRA)" <ji...@apache.org> on 2013/08/22 19:49:52 UTC
[jira] [Updated] (CASSANDRA-5922) Delete doesn't delete data.
[ https://issues.apache.org/jira/browse/CASSANDRA-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Eineke updated CASSANDRA-5922:
------------------------------------
Description:
In a nutshell, I'm running several test cases against my Cassandra JPA implementation (astyanax-jpa, see https://github.com/ceineke/astyanax-jpa) and sometimes (!) batched deletes seem not to delete all rows specified in the batch.
Here's the sequence of prepared CQL3 statements that is causing the issue to appear:
TRUNCATE compositeentity;
(delete all records so we have a clean slate)
INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree, astring, auuid) VALUES (?, ?, ?, ?, ?);
INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree, astring, auuid) VALUES (?, ?, ?, ?, ?);
(insert two unique rows into the table)
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
(load both rows from the table to validate their existence)
SELECT COUNT(1) FROM compositeentity;
(counts rows to validate the number of records in the table)
BEGIN BATCH DELETE FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?; DELETE FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?; APPLY BATCH;
(uses a logged batch to delete the two rows from the table)
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
(tries loads the rows from the table to check that they don't exist anymore)
After the delete, Cassandra has deleted only the first row so that the second SELECT here actually returns data. So far, this behaviour occurs randomly.
This happens even if there's a long sleep (1s, 10s) between the batch delete and the selects. It is always the second row that isn't deleted, never the first.
Thinking it might be timing issue (based on http://ria101.wordpress.com/2011/02/08/cassandra-the-importance-of-system-clocks-avoiding-oom-and-how-to-escape-oom-meltdown/), I've set up NTP to keep the clocks synchronized across all nodes (one node acts as the master which syncs to time.nrc.ca and {0,1,2,3}.ca.pool.ntp.org, whereas the remaining ones sync against the master. This hasn't reduced the number of times this behaviour crops up.
(I am executing all statements with QUORUM level consistency.)
I'm open to suggestions as to why this occurs and how I can fix it, if this can be fixed.
was:
In a nutcase, I'm running several test cases against my Cassandra JPA implementation (astyanax-jpa, see https://github.com/ceineke/astyanax-jpa) and sometimes(!) batched deletes seem not to delete all data.
Here's the sequence of prepared CQL3 statements that is causing the issue to appear:
TRUNCATE compositeentity;
(delete all records so we have a clean slate)
INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree, astring, auuid) VALUES (?, ?, ?, ?, ?);
INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree, astring, auuid) VALUES (?, ?, ?, ?, ?);
(insert two unique rows into the table)
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
(load both rows from the table to validate their existence)
SELECT COUNT(1) FROM compositeentity;
(counts rows to validate the number of records in the table)
BEGIN BATCH DELETE FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?; DELETE FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?; APPLY BATCH;
(uses a logged batch to delete the two rows from the table)
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
(tries loads the rows from the table to check that they don't exist anymore)
After the delete, Cassandra has deleted only the first row so that the second SELECT here actually returns data. So far, this behaviour occurs randomly.
This happens even if there's a long sleep (1s, 10s) between the batch delete and the selects. It is always the second row that isn't deleted, never the first.
Thinking it might be timing issue (based on http://ria101.wordpress.com/2011/02/08/cassandra-the-importance-of-system-clocks-avoiding-oom-and-how-to-escape-oom-meltdown/), I've set up NTP to keep the clocks synchronized across all nodes (one node acts as the master which syncs to time.nrc.ca and {0,1,2,3}.ca.pool.ntp.org, whereas the remaining ones sync against the master. This hasn't reduced the number of times this behaviour crops up.
(I am executing all statements with QUORUM level consistency.)
I'm open to suggestions as to why this occurs and how I can fix it, if this can be fixed.
> Delete doesn't delete data.
> ---------------------------
>
> Key: CASSANDRA-5922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5922
> Project: Cassandra
> Issue Type: Bug
> Environment: 4-node cluster w/ Cassandra v1.2.8
> Oracle JDK 1.6.0_45
> Netflix Astyanax 1.56.42
> Quorum read and write consistency level
> Reporter: Chris Eineke
>
> In a nutshell, I'm running several test cases against my Cassandra JPA implementation (astyanax-jpa, see https://github.com/ceineke/astyanax-jpa) and sometimes (!) batched deletes seem not to delete all rows specified in the batch.
> Here's the sequence of prepared CQL3 statements that is causing the issue to appear:
> TRUNCATE compositeentity;
> (delete all records so we have a clean slate)
> INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree, astring, auuid) VALUES (?, ?, ?, ?, ?);
> INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree, astring, auuid) VALUES (?, ?, ?, ?, ?);
> (insert two unique rows into the table)
> SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
> SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
> (load both rows from the table to validate their existence)
> SELECT COUNT(1) FROM compositeentity;
> (counts rows to validate the number of records in the table)
> BEGIN BATCH DELETE FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?; DELETE FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?; APPLY BATCH;
> (uses a logged batch to delete the two rows from the table)
> SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
> SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
> (tries loads the rows from the table to check that they don't exist anymore)
> After the delete, Cassandra has deleted only the first row so that the second SELECT here actually returns data. So far, this behaviour occurs randomly.
> This happens even if there's a long sleep (1s, 10s) between the batch delete and the selects. It is always the second row that isn't deleted, never the first.
> Thinking it might be timing issue (based on http://ria101.wordpress.com/2011/02/08/cassandra-the-importance-of-system-clocks-avoiding-oom-and-how-to-escape-oom-meltdown/), I've set up NTP to keep the clocks synchronized across all nodes (one node acts as the master which syncs to time.nrc.ca and {0,1,2,3}.ca.pool.ntp.org, whereas the remaining ones sync against the master. This hasn't reduced the number of times this behaviour crops up.
> (I am executing all statements with QUORUM level consistency.)
> I'm open to suggestions as to why this occurs and how I can fix it, if this can be fixed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira