You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Benyi Wang <be...@gmail.com> on 2015/03/30 20:41:17 UTC
Why select returns tombstoned results?
Create table tomb_test (
guid text,
content text,
range text,
rank int,
id text,
cnt int
primary key (guid, content, range, rank)
)
Sometime I delete the rows using cassandra java driver using this query
DELETE FROM tomb_test WHERE guid=? and content=? and range=?
in Batch statement with UNLOGGED. CONSISTENCE_LEVEL is local_one.
But if I run
SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
range='week'
or
SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
range='week' and rank = 1
The result shows the deleted rows.
If I run this select, the deleted rows are not shown
SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1'
If I run delete statement in cqlsh, the deleted rows won't show up.
How can I fix this?
Re: Why select returns tombstoned results?
Posted by Benyi Wang <be...@gmail.com>.
All servers are running ntpd. I guess the time should be synced across all
servers.
My dataset is too large to use sstable2json. It would take long time.
I will try to repair to see if the issue is gone.
On Tue, Mar 31, 2015 at 7:49 AM, Ken Hancock <ke...@schange.com>
wrote:
> Have you checked time sync across all servers? The fact that you've
> changed consistency levels and you're getting different results may
> indicate something inherently wrong with the cluster such as writes being
> dropped or time differences between the nodes.
>
> A brute-force approach to better understand what's going on (especially if
> you have an example of the wrong data being returned) is to do a
> sstable2json on all your tables and simply grep for an example key.
>
> On Mon, Mar 30, 2015 at 4:39 PM, Benyi Wang <be...@gmail.com> wrote:
>
>> Thanks for replying.
>>
>> In cqlsh, if I change to Quorum (Consistency quorum), sometime the select
>> return the deleted row, sometime not.
>>
>> I have two virtual data centers: service (3 nodes) and analytics(4 nodes
>> collocate with Hadoop data nodes).The table has 3 replicas in service and 2
>> in analytics. When I wrote, I wrote into analytics using local_one. So I
>> guest the data may not replicated to all nodes yet.
>>
>> I will try to use strong consistency for write.
>>
>>
>>
>> On Mon, Mar 30, 2015 at 11:59 AM, Prem Yadav <ip...@gmail.com>
>> wrote:
>>
>>> Increase the read CL to quorum and you should get correct results.
>>> How many nodes do you have in the cluster and what is the replication
>>> factor for the keyspace?
>>>
>>> On Mon, Mar 30, 2015 at 7:41 PM, Benyi Wang <be...@gmail.com>
>>> wrote:
>>>
>>>> Create table tomb_test (
>>>> guid text,
>>>> content text,
>>>> range text,
>>>> rank int,
>>>> id text,
>>>> cnt int
>>>> primary key (guid, content, range, rank)
>>>> )
>>>>
>>>> Sometime I delete the rows using cassandra java driver using this query
>>>>
>>>> DELETE FROM tomb_test WHERE guid=? and content=? and range=?
>>>>
>>>> in Batch statement with UNLOGGED. CONSISTENCE_LEVEL is local_one.
>>>>
>>>> But if I run
>>>>
>>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>>>> range='week'
>>>> or
>>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>>>> range='week' and rank = 1
>>>>
>>>> The result shows the deleted rows.
>>>>
>>>> If I run this select, the deleted rows are not shown
>>>>
>>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1'
>>>>
>>>> If I run delete statement in cqlsh, the deleted rows won't show up.
>>>>
>>>> How can I fix this?
>>>>
>>>>
>>>
>>
>
>
> --
> *Ken Hancock *| System Architect, Advanced Advertising
> SeaChange International
> 50 Nagog Park
> Acton, Massachusetts 01720
> ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
> <http://www.schange.com/en-US/Company/InvestorRelations.aspx>
> Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hancock@schange.com
> | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks[image: LinkedIn]
> <http://www.linkedin.com/in/kenhancock>
>
> [image: SeaChange International]
> <http://www.schange.com/>This e-mail and any attachments may contain
> information which is SeaChange International confidential. The information
> enclosed is intended only for the addressees herein and may not be copied
> or forwarded without permission from SeaChange International.
>
Re: Why select returns tombstoned results?
Posted by Ken Hancock <ke...@schange.com>.
Have you checked time sync across all servers? The fact that you've
changed consistency levels and you're getting different results may
indicate something inherently wrong with the cluster such as writes being
dropped or time differences between the nodes.
A brute-force approach to better understand what's going on (especially if
you have an example of the wrong data being returned) is to do a
sstable2json on all your tables and simply grep for an example key.
On Mon, Mar 30, 2015 at 4:39 PM, Benyi Wang <be...@gmail.com> wrote:
> Thanks for replying.
>
> In cqlsh, if I change to Quorum (Consistency quorum), sometime the select
> return the deleted row, sometime not.
>
> I have two virtual data centers: service (3 nodes) and analytics(4 nodes
> collocate with Hadoop data nodes).The table has 3 replicas in service and 2
> in analytics. When I wrote, I wrote into analytics using local_one. So I
> guest the data may not replicated to all nodes yet.
>
> I will try to use strong consistency for write.
>
>
>
> On Mon, Mar 30, 2015 at 11:59 AM, Prem Yadav <ip...@gmail.com> wrote:
>
>> Increase the read CL to quorum and you should get correct results.
>> How many nodes do you have in the cluster and what is the replication
>> factor for the keyspace?
>>
>> On Mon, Mar 30, 2015 at 7:41 PM, Benyi Wang <be...@gmail.com>
>> wrote:
>>
>>> Create table tomb_test (
>>> guid text,
>>> content text,
>>> range text,
>>> rank int,
>>> id text,
>>> cnt int
>>> primary key (guid, content, range, rank)
>>> )
>>>
>>> Sometime I delete the rows using cassandra java driver using this query
>>>
>>> DELETE FROM tomb_test WHERE guid=? and content=? and range=?
>>>
>>> in Batch statement with UNLOGGED. CONSISTENCE_LEVEL is local_one.
>>>
>>> But if I run
>>>
>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>>> range='week'
>>> or
>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>>> range='week' and rank = 1
>>>
>>> The result shows the deleted rows.
>>>
>>> If I run this select, the deleted rows are not shown
>>>
>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1'
>>>
>>> If I run delete statement in cqlsh, the deleted rows won't show up.
>>>
>>> How can I fix this?
>>>
>>>
>>
>
--
*Ken Hancock *| System Architect, Advanced Advertising
SeaChange International
50 Nagog Park
Acton, Massachusetts 01720
ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
<http://www.schange.com/en-US/Company/InvestorRelations.aspx>
Office: +1 (978) 889-3329 | [image: Google Talk:]
ken.hancock@schange.com | [image:
Skype:]hancockks | [image: Yahoo IM:]hancockks[image: LinkedIn]
<http://www.linkedin.com/in/kenhancock>
[image: SeaChange International]
<http://www.schange.com/>This e-mail and any attachments may contain
information which is SeaChange International confidential. The information
enclosed is intended only for the addressees herein and may not be copied
or forwarded without permission from SeaChange International.
Re: Why select returns tombstoned results?
Posted by Benyi Wang <be...@gmail.com>.
Unfortunately I'm using 2.1.2. Is it possible that I downgrade to 2.0.13
without wiping out the data? I'm worrying about if there is a bug in 2.1.2.
On Tue, Mar 31, 2015 at 4:37 AM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:
> What version of Cassandra are you running? Are you by any chance running
> repairs on your data?
>
> On Mon, Mar 30, 2015 at 5:39 PM, Benyi Wang <be...@gmail.com> wrote:
>
>> Thanks for replying.
>>
>> In cqlsh, if I change to Quorum (Consistency quorum), sometime the select
>> return the deleted row, sometime not.
>>
>> I have two virtual data centers: service (3 nodes) and analytics(4 nodes
>> collocate with Hadoop data nodes).The table has 3 replicas in service and 2
>> in analytics. When I wrote, I wrote into analytics using local_one. So I
>> guest the data may not replicated to all nodes yet.
>>
>> I will try to use strong consistency for write.
>>
>>
>>
>> On Mon, Mar 30, 2015 at 11:59 AM, Prem Yadav <ip...@gmail.com>
>> wrote:
>>
>>> Increase the read CL to quorum and you should get correct results.
>>> How many nodes do you have in the cluster and what is the replication
>>> factor for the keyspace?
>>>
>>> On Mon, Mar 30, 2015 at 7:41 PM, Benyi Wang <be...@gmail.com>
>>> wrote:
>>>
>>>> Create table tomb_test (
>>>> guid text,
>>>> content text,
>>>> range text,
>>>> rank int,
>>>> id text,
>>>> cnt int
>>>> primary key (guid, content, range, rank)
>>>> )
>>>>
>>>> Sometime I delete the rows using cassandra java driver using this query
>>>>
>>>> DELETE FROM tomb_test WHERE guid=? and content=? and range=?
>>>>
>>>> in Batch statement with UNLOGGED. CONSISTENCE_LEVEL is local_one.
>>>>
>>>> But if I run
>>>>
>>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>>>> range='week'
>>>> or
>>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>>>> range='week' and rank = 1
>>>>
>>>> The result shows the deleted rows.
>>>>
>>>> If I run this select, the deleted rows are not shown
>>>>
>>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1'
>>>>
>>>> If I run delete statement in cqlsh, the deleted rows won't show up.
>>>>
>>>> How can I fix this?
>>>>
>>>>
>>>
>>
>
>
> --
> *Paulo Motta*
>
> Chaordic | *Platform*
> *www.chaordic.com.br <http://www.chaordic.com.br/>*
> +55 48 3232.3200
>
Re: Why select returns tombstoned results?
Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
What version of Cassandra are you running? Are you by any chance running
repairs on your data?
On Mon, Mar 30, 2015 at 5:39 PM, Benyi Wang <be...@gmail.com> wrote:
> Thanks for replying.
>
> In cqlsh, if I change to Quorum (Consistency quorum), sometime the select
> return the deleted row, sometime not.
>
> I have two virtual data centers: service (3 nodes) and analytics(4 nodes
> collocate with Hadoop data nodes).The table has 3 replicas in service and 2
> in analytics. When I wrote, I wrote into analytics using local_one. So I
> guest the data may not replicated to all nodes yet.
>
> I will try to use strong consistency for write.
>
>
>
> On Mon, Mar 30, 2015 at 11:59 AM, Prem Yadav <ip...@gmail.com> wrote:
>
>> Increase the read CL to quorum and you should get correct results.
>> How many nodes do you have in the cluster and what is the replication
>> factor for the keyspace?
>>
>> On Mon, Mar 30, 2015 at 7:41 PM, Benyi Wang <be...@gmail.com>
>> wrote:
>>
>>> Create table tomb_test (
>>> guid text,
>>> content text,
>>> range text,
>>> rank int,
>>> id text,
>>> cnt int
>>> primary key (guid, content, range, rank)
>>> )
>>>
>>> Sometime I delete the rows using cassandra java driver using this query
>>>
>>> DELETE FROM tomb_test WHERE guid=? and content=? and range=?
>>>
>>> in Batch statement with UNLOGGED. CONSISTENCE_LEVEL is local_one.
>>>
>>> But if I run
>>>
>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>>> range='week'
>>> or
>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>>> range='week' and rank = 1
>>>
>>> The result shows the deleted rows.
>>>
>>> If I run this select, the deleted rows are not shown
>>>
>>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1'
>>>
>>> If I run delete statement in cqlsh, the deleted rows won't show up.
>>>
>>> How can I fix this?
>>>
>>>
>>
>
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
Re: Why select returns tombstoned results?
Posted by Benyi Wang <be...@gmail.com>.
Thanks for replying.
In cqlsh, if I change to Quorum (Consistency quorum), sometime the select
return the deleted row, sometime not.
I have two virtual data centers: service (3 nodes) and analytics(4 nodes
collocate with Hadoop data nodes).The table has 3 replicas in service and 2
in analytics. When I wrote, I wrote into analytics using local_one. So I
guest the data may not replicated to all nodes yet.
I will try to use strong consistency for write.
On Mon, Mar 30, 2015 at 11:59 AM, Prem Yadav <ip...@gmail.com> wrote:
> Increase the read CL to quorum and you should get correct results.
> How many nodes do you have in the cluster and what is the replication
> factor for the keyspace?
>
> On Mon, Mar 30, 2015 at 7:41 PM, Benyi Wang <be...@gmail.com> wrote:
>
>> Create table tomb_test (
>> guid text,
>> content text,
>> range text,
>> rank int,
>> id text,
>> cnt int
>> primary key (guid, content, range, rank)
>> )
>>
>> Sometime I delete the rows using cassandra java driver using this query
>>
>> DELETE FROM tomb_test WHERE guid=? and content=? and range=?
>>
>> in Batch statement with UNLOGGED. CONSISTENCE_LEVEL is local_one.
>>
>> But if I run
>>
>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>> range='week'
>> or
>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
>> range='week' and rank = 1
>>
>> The result shows the deleted rows.
>>
>> If I run this select, the deleted rows are not shown
>>
>> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1'
>>
>> If I run delete statement in cqlsh, the deleted rows won't show up.
>>
>> How can I fix this?
>>
>>
>
Re: Why select returns tombstoned results?
Posted by Prem Yadav <ip...@gmail.com>.
Increase the read CL to quorum and you should get correct results.
How many nodes do you have in the cluster and what is the replication
factor for the keyspace?
On Mon, Mar 30, 2015 at 7:41 PM, Benyi Wang <be...@gmail.com> wrote:
> Create table tomb_test (
> guid text,
> content text,
> range text,
> rank int,
> id text,
> cnt int
> primary key (guid, content, range, rank)
> )
>
> Sometime I delete the rows using cassandra java driver using this query
>
> DELETE FROM tomb_test WHERE guid=? and content=? and range=?
>
> in Batch statement with UNLOGGED. CONSISTENCE_LEVEL is local_one.
>
> But if I run
>
> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
> range='week'
> or
> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1' and
> range='week' and rank = 1
>
> The result shows the deleted rows.
>
> If I run this select, the deleted rows are not shown
>
> SELECT * FROM tomb_test WHERE guid='guid-1' and content='content-1'
>
> If I run delete statement in cqlsh, the deleted rows won't show up.
>
> How can I fix this?
>
>