You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Robert Wille <rw...@fold3.com> on 2015/10/03 00:58:26 UTC

Duplicate records returned

When I run the query "SELECT image FROM roll WHERE roll = :roll“ against this table

CREATE TABLE roll (
roll INT,
image BIGINT,
data VARCHAR static,
mid VARCHAR,
imp_st VARCHAR,
PRIMARY KEY ((roll), image)
) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };

I often get duplicate records back. Seems like a very simple query to botch. I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1. I don’t see duplicates in cqlsh. Any thoughts?

Thanks

Robert

Re: Duplicate records returned

Posted by Robert Wille <rw...@fold3.com>.
If anyone is following this, I also logged the bug at https://datastax-oss.atlassian.net/browse/JAVA-943. I suspect that its a driver bug, so I anticipate CASSANDRA-10442 being closed, and hopefully the folks at datastax can get this fixed. This bug must affect a whole lot of people.

On Oct 3, 2015, at 2:33 PM, Robert Wille <rw...@fold3.com>> wrote:

It's a paging bug. I ALWAYS get a duplicated record every fetchSize records. Easily duplicated 100% of the time.

I’ve logged a bug: https://issues.apache.org/jira/browse/CASSANDRA-10442

Robert

On Oct 3, 2015, at 10:59 AM, Robert Wille <rw...@fold3.com>> wrote:

Oops, I was trimming out some irrelevant stuff, and trimmed out too much. The second snippet should be this:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;

for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

if (imageId == lastImageId)
{
logger.warn("Cassandra duplicated " + imageId);
continue;
}

total++;
lastImageId = imageId;
}


On Oct 3, 2015, at 10:54 AM, Robert Wille <rw...@fold3.com>> wrote:

I don’t think its an application problem. The following simple snippets produce different totals:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
for (Row row : rs)
{
total++;
}

---------------------

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;
for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

total++;
lastImageId = imageId;
}

This doesn’t happen for all partitions. In fact most don’t have this problem (maybe 20% do this). But the ones that do repeat records, do so deterministically. I see this problem in multiple tables.

I’m only retrieving the clustering key, so it has nothing to do with the data field.

I suspect this is a paging problem, and might be a driver issue.

Robert

On Oct 3, 2015, at 9:29 AM, Eric Stevens <mi...@gmail.com>> wrote:

Can you give us an example of the duplicate records that comes back?  How reliable is it (i.e. is it every record, is it one record per read, etc)?  By any chance is it just the `data` field that duplicates while the other fields change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause it to happen.  Not to say it's impossible, but Cassandra's standard read path involves collapsing duplicate or otherwise overlapping answers from multiple replicas; such a thing would be a pretty substantial deviation.  Especially since you don't see the duplicates in cqlsh, I have a hunch this is an application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille <rw...@fold3.com>> wrote:
When I run the query "SELECT image FROM roll WHERE roll = :roll“ against this table

CREATE TABLE roll (
roll INT,
image BIGINT,
data VARCHAR static,
mid VARCHAR,
imp_st VARCHAR,
PRIMARY KEY ((roll), image)
) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };

I often get duplicate records back. Seems like a very simple query to botch. I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1. I don’t see duplicates in cqlsh. Any thoughts?

Thanks

Robert

Re: Duplicate records returned

Posted by Robert Wille <rw...@fold3.com>.
It's a paging bug. I ALWAYS get a duplicated record every fetchSize records. Easily duplicated 100% of the time.

I’ve logged a bug: https://issues.apache.org/jira/browse/CASSANDRA-10442

Robert

On Oct 3, 2015, at 10:59 AM, Robert Wille <rw...@fold3.com>> wrote:

Oops, I was trimming out some irrelevant stuff, and trimmed out too much. The second snippet should be this:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;

for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

if (imageId == lastImageId)
{
logger.warn("Cassandra duplicated " + imageId);
continue;
}

total++;
lastImageId = imageId;
}


On Oct 3, 2015, at 10:54 AM, Robert Wille <rw...@fold3.com>> wrote:

I don’t think its an application problem. The following simple snippets produce different totals:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
for (Row row : rs)
{
total++;
}

---------------------

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;
for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

total++;
lastImageId = imageId;
}

This doesn’t happen for all partitions. In fact most don’t have this problem (maybe 20% do this). But the ones that do repeat records, do so deterministically. I see this problem in multiple tables.

I’m only retrieving the clustering key, so it has nothing to do with the data field.

I suspect this is a paging problem, and might be a driver issue.

Robert

On Oct 3, 2015, at 9:29 AM, Eric Stevens <mi...@gmail.com>> wrote:

Can you give us an example of the duplicate records that comes back?  How reliable is it (i.e. is it every record, is it one record per read, etc)?  By any chance is it just the `data` field that duplicates while the other fields change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause it to happen.  Not to say it's impossible, but Cassandra's standard read path involves collapsing duplicate or otherwise overlapping answers from multiple replicas; such a thing would be a pretty substantial deviation.  Especially since you don't see the duplicates in cqlsh, I have a hunch this is an application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille <rw...@fold3.com>> wrote:
When I run the query "SELECT image FROM roll WHERE roll = :roll“ against this table

CREATE TABLE roll (
roll INT,
image BIGINT,
data VARCHAR static,
mid VARCHAR,
imp_st VARCHAR,
PRIMARY KEY ((roll), image)
) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };

I often get duplicate records back. Seems like a very simple query to botch. I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1. I don’t see duplicates in cqlsh. Any thoughts?

Thanks

Robert

Re: Duplicate records returned

Posted by Robert Wille <rw...@fold3.com>.
Oops, I was trimming out some irrelevant stuff, and trimmed out too much. The second snippet should be this:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;


for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

if (imageId == lastImageId)
{
logger.warn("Cassandra duplicated " + imageId);
continue;
}

total++;
lastImageId = imageId;
}


On Oct 3, 2015, at 10:54 AM, Robert Wille <rw...@fold3.com>> wrote:

I don’t think its an application problem. The following simple snippets produce different totals:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
for (Row row : rs)
{
total++;
}

---------------------

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;
for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

total++;
lastImageId = imageId;
}

This doesn’t happen for all partitions. In fact most don’t have this problem (maybe 20% do this). But the ones that do repeat records, do so deterministically. I see this problem in multiple tables.

I’m only retrieving the clustering key, so it has nothing to do with the data field.

I suspect this is a paging problem, and might be a driver issue.

Robert

On Oct 3, 2015, at 9:29 AM, Eric Stevens <mi...@gmail.com>> wrote:

Can you give us an example of the duplicate records that comes back?  How reliable is it (i.e. is it every record, is it one record per read, etc)?  By any chance is it just the `data` field that duplicates while the other fields change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause it to happen.  Not to say it's impossible, but Cassandra's standard read path involves collapsing duplicate or otherwise overlapping answers from multiple replicas; such a thing would be a pretty substantial deviation.  Especially since you don't see the duplicates in cqlsh, I have a hunch this is an application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille <rw...@fold3.com>> wrote:
When I run the query "SELECT image FROM roll WHERE roll = :roll“ against this table

CREATE TABLE roll (
roll INT,
image BIGINT,
data VARCHAR static,
mid VARCHAR,
imp_st VARCHAR,
PRIMARY KEY ((roll), image)
) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };

I often get duplicate records back. Seems like a very simple query to botch. I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1. I don’t see duplicates in cqlsh. Any thoughts?

Thanks

Robert

Re: Duplicate records returned

Posted by Robert Wille <rw...@fold3.com>.
I don’t think its an application problem. The following simple snippets produce different totals:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
for (Row row : rs)
{
total++;
}

---------------------

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;
for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

total++;
lastImageId = imageId;
}

This doesn’t happen for all partitions. In fact most don’t have this problem (maybe 20% do this). But the ones that do repeat records, do so deterministically. I see this problem in multiple tables.

I’m only retrieving the clustering key, so it has nothing to do with the data field.

I suspect this is a paging problem, and might be a driver issue.

Robert

On Oct 3, 2015, at 9:29 AM, Eric Stevens <mi...@gmail.com>> wrote:

Can you give us an example of the duplicate records that comes back?  How reliable is it (i.e. is it every record, is it one record per read, etc)?  By any chance is it just the `data` field that duplicates while the other fields change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause it to happen.  Not to say it's impossible, but Cassandra's standard read path involves collapsing duplicate or otherwise overlapping answers from multiple replicas; such a thing would be a pretty substantial deviation.  Especially since you don't see the duplicates in cqlsh, I have a hunch this is an application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille <rw...@fold3.com>> wrote:
When I run the query "SELECT image FROM roll WHERE roll = :roll“ against this table

CREATE TABLE roll (
roll INT,
image BIGINT,
data VARCHAR static,
mid VARCHAR,
imp_st VARCHAR,
PRIMARY KEY ((roll), image)
) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };

I often get duplicate records back. Seems like a very simple query to botch. I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1. I don’t see duplicates in cqlsh. Any thoughts?

Thanks

Robert

Re: Duplicate records returned

Posted by Eric Stevens <mi...@gmail.com>.
Can you give us an example of the duplicate records that comes back?  How
reliable is it (i.e. is it every record, is it one record per read, etc)?
By any chance is it just the `data` field that duplicates while the other
fields change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause
it to happen.  Not to say it's impossible, but Cassandra's standard read
path involves collapsing duplicate or otherwise overlapping answers from
multiple replicas; such a thing would be a pretty substantial deviation.
Especially since you don't see the duplicates in cqlsh, I have a hunch this
is an application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille <rw...@fold3.com> wrote:

> When I run the query "SELECT image FROM roll WHERE roll = :roll“ against
> this table
>
> CREATE TABLE roll (
> roll INT,
> image BIGINT,
> data VARCHAR static,
> mid VARCHAR,
> imp_st VARCHAR,
> PRIMARY KEY ((roll), image)
> ) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' :
> 'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };
>
> I often get duplicate records back. Seems like a very simple query to
> botch. I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1.
> I don’t see duplicates in cqlsh. Any thoughts?
>
> Thanks
>
> Robert
>
>