You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Joe Obernberger <jo...@gmail.com> on 2021/10/14 15:44:08 UTC
Tombstones? 4.0.1
Hi all - I have a table where I've needed to delete a number of rows.
I've run repair, but I still can't select from the table.
select * from doc.indexorganize limit 10;
OperationTimedOut: errors={'172.16.100.37:9042': 'Client request
timeout. See Session.execute[_async](timeout)'},
last_host=172.16.100.37:9042
Info on the table:
nodetool tablestats doc.indexorganize
Total number of tables: 97
----------------
Keyspace : doc
Read Count: 170275408
Read Latency: 1.6486837044783356 ms
Write Count: 6821769404
Write Latency: 0.08147347268570909 ms
Pending Flushes: 0
Table: indexorganize
SSTable count: 21
Old SSTable count: 0
Space used (live): 1536557040
Space used (total): 1536557040
Space used by snapshots (total): 1728378992
Off heap memory used (total): 46251932
SSTable Compression Ratio: 0.5218383898575761
Number of partitions (estimate): 17365415
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 12
Local read count: 17346304
Local read latency: NaN ms
Local write count: 31340451
Local write latency: NaN ms
Pending flushes: 0
Percent repaired: 100.0
Bytes repaired: 1.084GiB
Bytes unrepaired: 0.000KiB
Bytes pending repair: 0.000KiB
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 38030728
Bloom filter off heap memory used: 38030560
Index summary off heap memory used: 7653060
Compression metadata off heap memory used: 568312
Compacted partition minimum bytes: 51
Compacted partition maximum bytes: 86
Compacted partition mean bytes: 67
Average live cells per slice (last five minutes):
73.53164556962025
Maximum live cells per slice (last five minutes): 5722
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
Dropped Mutations: 0
nodetool tablehistograms doc.indexorganize
doc/indexorganize histograms
Percentile Read Latency Write Latency SSTables Partition
Size Cell Count
(micros) (micros) (bytes)
50% 0.00 0.00 0.00
60 1
75% 0.00 0.00 0.00
86 2
95% 0.00 0.00 0.00
86 2
98% 0.00 0.00 0.00
86 2
99% 0.00 0.00 0.00
86 2
Min 0.00 0.00 0.00
51 0
Max 0.00 0.00 0.00
86 2
Any ideas on what I can do? Thank you!
-Joe
Re: Tombstones? 4.0.1
Posted by Jeff Jirsa <jj...@gmail.com>.
This is not the right data model for Cassandra. Strong encouragement to
watch one of Patrick McFadin's data modeling videos on youtube.
You very much want to always query where a WHERE clause, which usually
means knowing a partition key (or set of partition keys) likely to contain
your data, and using sorting within those keys to make data easy to access.
This may mean a partition key that's a date (so all writes for a given date
land on one replica set, and you only query within that set), or
(date,bucket) tuple (where bucket is an int from 0-1000, for example, which
avoids hostspotting), then you can read (date, 7) and (date, 996) and
everything else in concurrent async queries, or something else that gives
you deterministic partitioning so you're not walking past all of those dead
tombstones.
Reading with a naive SELECT with no WHERE is going to be perhaps the least
performant way to do this in cassandra, and you are probably miserable at
both the response time and the failure rate, because this is not how
Cassandra is designed to work.
On Mon, Oct 25, 2021 at 3:56 PM Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> Hi Jeff - yes, I'm doing a select without where - specifically: select
> uuid from table limit 1000;
> Not inserting nulls, and nothing is TTL'd.
> At this point with zero rows, the above select fails.
>
> Sounds like my application needs a redesign as doing 1 billion inserts,
> and 100 million deletes results in an unusable table. I'm using Cassandra
> to de-duplicate data and that's not a good use case for it.
>
> -Joe
> On 10/25/2021 6:51 PM, Jeff Jirsa wrote:
>
> The tombstone threshold is "how many tombstones are encountered within a
> single read command", and the default is something like 100,000 (
> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1293-L1294
> )
>
> Deletes are not forbidden, but you have to read in such a way that you
> touch less than 100,000 deletes per read.
>
> Are you doing full table scans or SELECT without WHERE?
> Are you inserting nulls in some columns?
> Are you TTL'ing everything ?
>
>
>
> On Mon, Oct 25, 2021 at 3:28 PM Joe Obernberger <
> joseph.obernberger@gmail.com> wrote:
>
>> Update - after 10 days, I'm able to use the table again; prior to that
>> all selects timed out.
>> Are deletes basically forbidden with Cassandra? If you have a table
>> where you want to do lots of inserts and deletes, is there an option that
>> works in Cassandra? Even thought the table now has zero rows, after
>> deleting them, I can no longer do a select from the table as it times out.
>> Thank you!
>>
>> -Joe
>> On 10/14/2021 3:38 PM, Joe Obernberger wrote:
>>
>> I'm not sure if tombstones is the issue; is it? Grace is set to 10 days,
>> that time has not passed yet.
>>
>> -Joe
>> On 10/14/2021 1:37 PM, James Brown wrote:
>>
>> What is gc_grace_seconds set to on the table? Once that passes, you can
>> do `nodetool scrub` to more emphatically remove tombstones...
>>
>> On Thu, Oct 14, 2021 at 8:49 AM Joe Obernberger <
>> joseph.obernberger@gmail.com> wrote:
>>
>>> Hi all - I have a table where I've needed to delete a number of rows.
>>> I've run repair, but I still can't select from the table.
>>>
>>> select * from doc.indexorganize limit 10;
>>> OperationTimedOut: errors={'172.16.100.37:9042': 'Client request
>>> timeout. See Session.execute[_async](timeout)'},
>>> last_host=172.16.100.37:9042
>>>
>>> Info on the table:
>>>
>>> nodetool tablestats doc.indexorganize
>>> Total number of tables: 97
>>> ----------------
>>> Keyspace : doc
>>> Read Count: 170275408
>>> Read Latency: 1.6486837044783356 ms
>>> Write Count: 6821769404
>>> Write Latency: 0.08147347268570909 ms
>>> Pending Flushes: 0
>>> Table: indexorganize
>>> SSTable count: 21
>>> Old SSTable count: 0
>>> Space used (live): 1536557040
>>> Space used (total): 1536557040
>>> Space used by snapshots (total): 1728378992
>>> Off heap memory used (total): 46251932
>>> SSTable Compression Ratio: 0.5218383898575761
>>> Number of partitions (estimate): 17365415
>>> Memtable cell count: 0
>>> Memtable data size: 0
>>> Memtable off heap memory used: 0
>>> Memtable switch count: 12
>>> Local read count: 17346304
>>> Local read latency: NaN ms
>>> Local write count: 31340451
>>> Local write latency: NaN ms
>>> Pending flushes: 0
>>> Percent repaired: 100.0
>>> Bytes repaired: 1.084GiB
>>> Bytes unrepaired: 0.000KiB
>>> Bytes pending repair: 0.000KiB
>>> Bloom filter false positives: 0
>>> Bloom filter false ratio: 0.00000
>>> Bloom filter space used: 38030728
>>> Bloom filter off heap memory used: 38030560
>>> Index summary off heap memory used: 7653060
>>> Compression metadata off heap memory used: 568312
>>> Compacted partition minimum bytes: 51
>>> Compacted partition maximum bytes: 86
>>> Compacted partition mean bytes: 67
>>> Average live cells per slice (last five minutes):
>>> 73.53164556962025
>>> Maximum live cells per slice (last five minutes): 5722
>>> Average tombstones per slice (last five minutes): 1.0
>>> Maximum tombstones per slice (last five minutes): 1
>>> Dropped Mutations: 0
>>>
>>> nodetool tablehistograms doc.indexorganize
>>> doc/indexorganize histograms
>>> Percentile Read Latency Write Latency SSTables Partition
>>> Size Cell Count
>>> (micros) (micros)
>>> (bytes)
>>> 50% 0.00 0.00 0.00
>>> 60 1
>>> 75% 0.00 0.00 0.00
>>> 86 2
>>> 95% 0.00 0.00 0.00
>>> 86 2
>>> 98% 0.00 0.00 0.00
>>> 86 2
>>> 99% 0.00 0.00 0.00
>>> 86 2
>>> Min 0.00 0.00 0.00
>>> 51 0
>>> Max 0.00 0.00 0.00
>>> 86 2
>>>
>>> Any ideas on what I can do? Thank you!
>>>
>>> -Joe
>>>
>>>
>>
>> --
>> James Brown
>> Engineer
>>
>>
>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
>> www.avg.com
>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>> <#m_-3623639998410431590_m_-996371029387335963_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>>
Re: Tombstones? 4.0.1
Posted by Joe Obernberger <jo...@gmail.com>.
Hi Jeff - yes, I'm doing a select without where - specifically: select
uuid from table limit 1000;
Not inserting nulls, and nothing is TTL'd.
At this point with zero rows, the above select fails.
Sounds like my application needs a redesign as doing 1 billion inserts,
and 100 million deletes results in an unusable table. I'm using
Cassandra to de-duplicate data and that's not a good use case for it.
-Joe
On 10/25/2021 6:51 PM, Jeff Jirsa wrote:
> The tombstone threshold is "how many tombstones are encountered within
> a single read command", and the default is something like 100,000 (
> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1293-L1294
> )
>
> Deletes are not forbidden, but you have to read in such a way that you
> touch less than 100,000 deletes per read.
>
> Are you doing full table scans or SELECT without WHERE?
> Are you inserting nulls in some columns?
> Are you TTL'ing everything ?
>
>
>
> On Mon, Oct 25, 2021 at 3:28 PM Joe Obernberger
> <jo...@gmail.com> wrote:
>
> Update - after 10 days, I'm able to use the table again; prior to
> that all selects timed out.
> Are deletes basically forbidden with Cassandra? If you have a
> table where you want to do lots of inserts and deletes, is there
> an option that works in Cassandra? Even thought the table now has
> zero rows, after deleting them, I can no longer do a select from
> the table as it times out.
> Thank you!
>
> -Joe
>
> On 10/14/2021 3:38 PM, Joe Obernberger wrote:
>>
>> I'm not sure if tombstones is the issue; is it? Grace is set to
>> 10 days, that time has not passed yet.
>>
>> -Joe
>>
>> On 10/14/2021 1:37 PM, James Brown wrote:
>>> What is gc_grace_seconds set to on the table? Once that passes,
>>> you can do `nodetool scrub` to more emphatically remove
>>> tombstones...
>>>
>>> On Thu, Oct 14, 2021 at 8:49 AM Joe Obernberger
>>> <jo...@gmail.com> wrote:
>>>
>>> Hi all - I have a table where I've needed to delete a number
>>> of rows.
>>> I've run repair, but I still can't select from the table.
>>>
>>> select * from doc.indexorganize limit 10;
>>> OperationTimedOut: errors={'172.16.100.37:9042
>>> <http://172.16.100.37:9042>': 'Client request
>>> timeout. See Session.execute[_async](timeout)'},
>>> last_host=172.16.100.37:9042 <http://172.16.100.37:9042>
>>>
>>> Info on the table:
>>>
>>> nodetool tablestats doc.indexorganize
>>> Total number of tables: 97
>>> ----------------
>>> Keyspace : doc
>>> Read Count: 170275408
>>> Read Latency: 1.6486837044783356 ms
>>> Write Count: 6821769404
>>> Write Latency: 0.08147347268570909 ms
>>> Pending Flushes: 0
>>> Table: indexorganize
>>> SSTable count: 21
>>> Old SSTable count: 0
>>> Space used (live): 1536557040
>>> Space used (total): 1536557040
>>> Space used by snapshots (total): 1728378992
>>> Off heap memory used (total): 46251932
>>> SSTable Compression Ratio: 0.5218383898575761
>>> Number of partitions (estimate): 17365415
>>> Memtable cell count: 0
>>> Memtable data size: 0
>>> Memtable off heap memory used: 0
>>> Memtable switch count: 12
>>> Local read count: 17346304
>>> Local read latency: NaN ms
>>> Local write count: 31340451
>>> Local write latency: NaN ms
>>> Pending flushes: 0
>>> Percent repaired: 100.0
>>> Bytes repaired: 1.084GiB
>>> Bytes unrepaired: 0.000KiB
>>> Bytes pending repair: 0.000KiB
>>> Bloom filter false positives: 0
>>> Bloom filter false ratio: 0.00000
>>> Bloom filter space used: 38030728
>>> Bloom filter off heap memory used: 38030560
>>> Index summary off heap memory used: 7653060
>>> Compression metadata off heap memory used:
>>> 568312
>>> Compacted partition minimum bytes: 51
>>> Compacted partition maximum bytes: 86
>>> Compacted partition mean bytes: 67
>>> Average live cells per slice (last five
>>> minutes):
>>> 73.53164556962025
>>> Maximum live cells per slice (last five
>>> minutes): 5722
>>> Average tombstones per slice (last five
>>> minutes): 1.0
>>> Maximum tombstones per slice (last five
>>> minutes): 1
>>> Dropped Mutations: 0
>>>
>>> nodetool tablehistograms doc.indexorganize
>>> doc/indexorganize histograms
>>> Percentile Read Latency Write Latency SSTables
>>> Partition
>>> Size Cell Count
>>> (micros)
>>> (micros) (bytes)
>>> 50% 0.00 0.00 0.00
>>> 60 1
>>> 75% 0.00 0.00 0.00
>>> 86 2
>>> 95% 0.00 0.00 0.00
>>> 86 2
>>> 98% 0.00 0.00 0.00
>>> 86 2
>>> 99% 0.00 0.00 0.00
>>> 86 2
>>> Min 0.00 0.00 0.00
>>> 51 0
>>> Max 0.00 0.00 0.00
>>> 86 2
>>>
>>> Any ideas on what I can do? Thank you!
>>>
>>> -Joe
>>>
>>>
>>>
>>> --
>>> James Brown
>>> Engineer
>>>
>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>> Virus-free. www.avg.com
>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>
>>>
>>> <#m_-996371029387335963_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
Re: Tombstones? 4.0.1
Posted by Jeff Jirsa <jj...@gmail.com>.
The tombstone threshold is "how many tombstones are encountered within a
single read command", and the default is something like 100,000 (
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L1293-L1294
)
Deletes are not forbidden, but you have to read in such a way that you
touch less than 100,000 deletes per read.
Are you doing full table scans or SELECT without WHERE?
Are you inserting nulls in some columns?
Are you TTL'ing everything ?
On Mon, Oct 25, 2021 at 3:28 PM Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> Update - after 10 days, I'm able to use the table again; prior to that all
> selects timed out.
> Are deletes basically forbidden with Cassandra? If you have a table where
> you want to do lots of inserts and deletes, is there an option that works
> in Cassandra? Even thought the table now has zero rows, after deleting
> them, I can no longer do a select from the table as it times out.
> Thank you!
>
> -Joe
> On 10/14/2021 3:38 PM, Joe Obernberger wrote:
>
> I'm not sure if tombstones is the issue; is it? Grace is set to 10 days,
> that time has not passed yet.
>
> -Joe
> On 10/14/2021 1:37 PM, James Brown wrote:
>
> What is gc_grace_seconds set to on the table? Once that passes, you can do
> `nodetool scrub` to more emphatically remove tombstones...
>
> On Thu, Oct 14, 2021 at 8:49 AM Joe Obernberger <
> joseph.obernberger@gmail.com> wrote:
>
>> Hi all - I have a table where I've needed to delete a number of rows.
>> I've run repair, but I still can't select from the table.
>>
>> select * from doc.indexorganize limit 10;
>> OperationTimedOut: errors={'172.16.100.37:9042': 'Client request
>> timeout. See Session.execute[_async](timeout)'},
>> last_host=172.16.100.37:9042
>>
>> Info on the table:
>>
>> nodetool tablestats doc.indexorganize
>> Total number of tables: 97
>> ----------------
>> Keyspace : doc
>> Read Count: 170275408
>> Read Latency: 1.6486837044783356 ms
>> Write Count: 6821769404
>> Write Latency: 0.08147347268570909 ms
>> Pending Flushes: 0
>> Table: indexorganize
>> SSTable count: 21
>> Old SSTable count: 0
>> Space used (live): 1536557040
>> Space used (total): 1536557040
>> Space used by snapshots (total): 1728378992
>> Off heap memory used (total): 46251932
>> SSTable Compression Ratio: 0.5218383898575761
>> Number of partitions (estimate): 17365415
>> Memtable cell count: 0
>> Memtable data size: 0
>> Memtable off heap memory used: 0
>> Memtable switch count: 12
>> Local read count: 17346304
>> Local read latency: NaN ms
>> Local write count: 31340451
>> Local write latency: NaN ms
>> Pending flushes: 0
>> Percent repaired: 100.0
>> Bytes repaired: 1.084GiB
>> Bytes unrepaired: 0.000KiB
>> Bytes pending repair: 0.000KiB
>> Bloom filter false positives: 0
>> Bloom filter false ratio: 0.00000
>> Bloom filter space used: 38030728
>> Bloom filter off heap memory used: 38030560
>> Index summary off heap memory used: 7653060
>> Compression metadata off heap memory used: 568312
>> Compacted partition minimum bytes: 51
>> Compacted partition maximum bytes: 86
>> Compacted partition mean bytes: 67
>> Average live cells per slice (last five minutes):
>> 73.53164556962025
>> Maximum live cells per slice (last five minutes): 5722
>> Average tombstones per slice (last five minutes): 1.0
>> Maximum tombstones per slice (last five minutes): 1
>> Dropped Mutations: 0
>>
>> nodetool tablehistograms doc.indexorganize
>> doc/indexorganize histograms
>> Percentile Read Latency Write Latency SSTables Partition
>> Size Cell Count
>> (micros) (micros) (bytes)
>> 50% 0.00 0.00 0.00
>> 60 1
>> 75% 0.00 0.00 0.00
>> 86 2
>> 95% 0.00 0.00 0.00
>> 86 2
>> 98% 0.00 0.00 0.00
>> 86 2
>> 99% 0.00 0.00 0.00
>> 86 2
>> Min 0.00 0.00 0.00
>> 51 0
>> Max 0.00 0.00 0.00
>> 86 2
>>
>> Any ideas on what I can do? Thank you!
>>
>> -Joe
>>
>>
>
> --
> James Brown
> Engineer
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> <#m_-996371029387335963_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
>
Re: Tombstones? 4.0.1
Posted by Dmitry Saprykin <sa...@gmail.com>.
If the table has zero rows you could truncate it
On Mon, Oct 25, 2021 at 6:29 PM Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> Update - after 10 days, I'm able to use the table again; prior to that all
> selects timed out.
> Are deletes basically forbidden with Cassandra? If you have a table where
> you want to do lots of inserts and deletes, is there an option that works
> in Cassandra? Even thought the table now has zero rows, after deleting
> them, I can no longer do a select from the table as it times out.
> Thank you!
>
> -Joe
> On 10/14/2021 3:38 PM, Joe Obernberger wrote:
>
> I'm not sure if tombstones is the issue; is it? Grace is set to 10 days,
> that time has not passed yet.
>
> -Joe
> On 10/14/2021 1:37 PM, James Brown wrote:
>
> What is gc_grace_seconds set to on the table? Once that passes, you can do
> `nodetool scrub` to more emphatically remove tombstones...
>
> On Thu, Oct 14, 2021 at 8:49 AM Joe Obernberger <
> joseph.obernberger@gmail.com> wrote:
>
>> Hi all - I have a table where I've needed to delete a number of rows.
>> I've run repair, but I still can't select from the table.
>>
>> select * from doc.indexorganize limit 10;
>> OperationTimedOut: errors={'172.16.100.37:9042': 'Client request
>> timeout. See Session.execute[_async](timeout)'},
>> last_host=172.16.100.37:9042
>>
>> Info on the table:
>>
>> nodetool tablestats doc.indexorganize
>> Total number of tables: 97
>> ----------------
>> Keyspace : doc
>> Read Count: 170275408
>> Read Latency: 1.6486837044783356 ms
>> Write Count: 6821769404
>> Write Latency: 0.08147347268570909 ms
>> Pending Flushes: 0
>> Table: indexorganize
>> SSTable count: 21
>> Old SSTable count: 0
>> Space used (live): 1536557040
>> Space used (total): 1536557040
>> Space used by snapshots (total): 1728378992
>> Off heap memory used (total): 46251932
>> SSTable Compression Ratio: 0.5218383898575761
>> Number of partitions (estimate): 17365415
>> Memtable cell count: 0
>> Memtable data size: 0
>> Memtable off heap memory used: 0
>> Memtable switch count: 12
>> Local read count: 17346304
>> Local read latency: NaN ms
>> Local write count: 31340451
>> Local write latency: NaN ms
>> Pending flushes: 0
>> Percent repaired: 100.0
>> Bytes repaired: 1.084GiB
>> Bytes unrepaired: 0.000KiB
>> Bytes pending repair: 0.000KiB
>> Bloom filter false positives: 0
>> Bloom filter false ratio: 0.00000
>> Bloom filter space used: 38030728
>> Bloom filter off heap memory used: 38030560
>> Index summary off heap memory used: 7653060
>> Compression metadata off heap memory used: 568312
>> Compacted partition minimum bytes: 51
>> Compacted partition maximum bytes: 86
>> Compacted partition mean bytes: 67
>> Average live cells per slice (last five minutes):
>> 73.53164556962025
>> Maximum live cells per slice (last five minutes): 5722
>> Average tombstones per slice (last five minutes): 1.0
>> Maximum tombstones per slice (last five minutes): 1
>> Dropped Mutations: 0
>>
>> nodetool tablehistograms doc.indexorganize
>> doc/indexorganize histograms
>> Percentile Read Latency Write Latency SSTables Partition
>> Size Cell Count
>> (micros) (micros) (bytes)
>> 50% 0.00 0.00 0.00
>> 60 1
>> 75% 0.00 0.00 0.00
>> 86 2
>> 95% 0.00 0.00 0.00
>> 86 2
>> 98% 0.00 0.00 0.00
>> 86 2
>> 99% 0.00 0.00 0.00
>> 86 2
>> Min 0.00 0.00 0.00
>> 51 0
>> Max 0.00 0.00 0.00
>> 86 2
>>
>> Any ideas on what I can do? Thank you!
>>
>> -Joe
>>
>>
>
> --
> James Brown
> Engineer
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> <#m_-8876101864410815383_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
>
Re: Tombstones? 4.0.1
Posted by Joe Obernberger <jo...@gmail.com>.
Update - after 10 days, I'm able to use the table again; prior to that
all selects timed out.
Are deletes basically forbidden with Cassandra? If you have a table
where you want to do lots of inserts and deletes, is there an option
that works in Cassandra? Even thought the table now has zero rows,
after deleting them, I can no longer do a select from the table as it
times out.
Thank you!
-Joe
On 10/14/2021 3:38 PM, Joe Obernberger wrote:
>
> I'm not sure if tombstones is the issue; is it? Grace is set to 10
> days, that time has not passed yet.
>
> -Joe
>
> On 10/14/2021 1:37 PM, James Brown wrote:
>> What is gc_grace_seconds set to on the table? Once that passes, you
>> can do `nodetool scrub` to more emphatically remove tombstones...
>>
>> On Thu, Oct 14, 2021 at 8:49 AM Joe Obernberger
>> <jo...@gmail.com> wrote:
>>
>> Hi all - I have a table where I've needed to delete a number of
>> rows.
>> I've run repair, but I still can't select from the table.
>>
>> select * from doc.indexorganize limit 10;
>> OperationTimedOut: errors={'172.16.100.37:9042
>> <http://172.16.100.37:9042>': 'Client request
>> timeout. See Session.execute[_async](timeout)'},
>> last_host=172.16.100.37:9042 <http://172.16.100.37:9042>
>>
>> Info on the table:
>>
>> nodetool tablestats doc.indexorganize
>> Total number of tables: 97
>> ----------------
>> Keyspace : doc
>> Read Count: 170275408
>> Read Latency: 1.6486837044783356 ms
>> Write Count: 6821769404
>> Write Latency: 0.08147347268570909 ms
>> Pending Flushes: 0
>> Table: indexorganize
>> SSTable count: 21
>> Old SSTable count: 0
>> Space used (live): 1536557040
>> Space used (total): 1536557040
>> Space used by snapshots (total): 1728378992
>> Off heap memory used (total): 46251932
>> SSTable Compression Ratio: 0.5218383898575761
>> Number of partitions (estimate): 17365415
>> Memtable cell count: 0
>> Memtable data size: 0
>> Memtable off heap memory used: 0
>> Memtable switch count: 12
>> Local read count: 17346304
>> Local read latency: NaN ms
>> Local write count: 31340451
>> Local write latency: NaN ms
>> Pending flushes: 0
>> Percent repaired: 100.0
>> Bytes repaired: 1.084GiB
>> Bytes unrepaired: 0.000KiB
>> Bytes pending repair: 0.000KiB
>> Bloom filter false positives: 0
>> Bloom filter false ratio: 0.00000
>> Bloom filter space used: 38030728
>> Bloom filter off heap memory used: 38030560
>> Index summary off heap memory used: 7653060
>> Compression metadata off heap memory used: 568312
>> Compacted partition minimum bytes: 51
>> Compacted partition maximum bytes: 86
>> Compacted partition mean bytes: 67
>> Average live cells per slice (last five minutes):
>> 73.53164556962025
>> Maximum live cells per slice (last five
>> minutes): 5722
>> Average tombstones per slice (last five
>> minutes): 1.0
>> Maximum tombstones per slice (last five minutes): 1
>> Dropped Mutations: 0
>>
>> nodetool tablehistograms doc.indexorganize
>> doc/indexorganize histograms
>> Percentile Read Latency Write Latency SSTables Partition
>> Size Cell Count
>> (micros)
>> (micros) (bytes)
>> 50% 0.00 0.00 0.00
>> 60 1
>> 75% 0.00 0.00 0.00
>> 86 2
>> 95% 0.00 0.00 0.00
>> 86 2
>> 98% 0.00 0.00 0.00
>> 86 2
>> 99% 0.00 0.00 0.00
>> 86 2
>> Min 0.00 0.00 0.00
>> 51 0
>> Max 0.00 0.00 0.00
>> 86 2
>>
>> Any ideas on what I can do? Thank you!
>>
>> -Joe
>>
>>
>>
>> --
>> James Brown
>> Engineer
>>
>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>> Virus-free. www.avg.com
>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>
>>
>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Tombstones? 4.0.1
Posted by Joe Obernberger <jo...@gmail.com>.
I'm not sure if tombstones is the issue; is it? Grace is set to 10
days, that time has not passed yet.
-Joe
On 10/14/2021 1:37 PM, James Brown wrote:
> What is gc_grace_seconds set to on the table? Once that passes, you
> can do `nodetool scrub` to more emphatically remove tombstones...
>
> On Thu, Oct 14, 2021 at 8:49 AM Joe Obernberger
> <jo...@gmail.com> wrote:
>
> Hi all - I have a table where I've needed to delete a number of rows.
> I've run repair, but I still can't select from the table.
>
> select * from doc.indexorganize limit 10;
> OperationTimedOut: errors={'172.16.100.37:9042
> <http://172.16.100.37:9042>': 'Client request
> timeout. See Session.execute[_async](timeout)'},
> last_host=172.16.100.37:9042 <http://172.16.100.37:9042>
>
> Info on the table:
>
> nodetool tablestats doc.indexorganize
> Total number of tables: 97
> ----------------
> Keyspace : doc
> Read Count: 170275408
> Read Latency: 1.6486837044783356 ms
> Write Count: 6821769404
> Write Latency: 0.08147347268570909 ms
> Pending Flushes: 0
> Table: indexorganize
> SSTable count: 21
> Old SSTable count: 0
> Space used (live): 1536557040
> Space used (total): 1536557040
> Space used by snapshots (total): 1728378992
> Off heap memory used (total): 46251932
> SSTable Compression Ratio: 0.5218383898575761
> Number of partitions (estimate): 17365415
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 12
> Local read count: 17346304
> Local read latency: NaN ms
> Local write count: 31340451
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 100.0
> Bytes repaired: 1.084GiB
> Bytes unrepaired: 0.000KiB
> Bytes pending repair: 0.000KiB
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.00000
> Bloom filter space used: 38030728
> Bloom filter off heap memory used: 38030560
> Index summary off heap memory used: 7653060
> Compression metadata off heap memory used: 568312
> Compacted partition minimum bytes: 51
> Compacted partition maximum bytes: 86
> Compacted partition mean bytes: 67
> Average live cells per slice (last five minutes):
> 73.53164556962025
> Maximum live cells per slice (last five minutes):
> 5722
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
>
> nodetool tablehistograms doc.indexorganize
> doc/indexorganize histograms
> Percentile Read Latency Write Latency SSTables Partition
> Size Cell Count
> (micros) (micros)
> (bytes)
> 50% 0.00 0.00 0.00
> 60 1
> 75% 0.00 0.00 0.00
> 86 2
> 95% 0.00 0.00 0.00
> 86 2
> 98% 0.00 0.00 0.00
> 86 2
> 99% 0.00 0.00 0.00
> 86 2
> Min 0.00 0.00 0.00
> 51 0
> Max 0.00 0.00 0.00
> 86 2
>
> Any ideas on what I can do? Thank you!
>
> -Joe
>
>
>
> --
> James Brown
> Engineer
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> Virus-free. www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Tombstones? 4.0.1
Posted by James Brown <jb...@easypost.com>.
What is gc_grace_seconds set to on the table? Once that passes, you can do
`nodetool scrub` to more emphatically remove tombstones...
On Thu, Oct 14, 2021 at 8:49 AM Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> Hi all - I have a table where I've needed to delete a number of rows.
> I've run repair, but I still can't select from the table.
>
> select * from doc.indexorganize limit 10;
> OperationTimedOut: errors={'172.16.100.37:9042': 'Client request
> timeout. See Session.execute[_async](timeout)'},
> last_host=172.16.100.37:9042
>
> Info on the table:
>
> nodetool tablestats doc.indexorganize
> Total number of tables: 97
> ----------------
> Keyspace : doc
> Read Count: 170275408
> Read Latency: 1.6486837044783356 ms
> Write Count: 6821769404
> Write Latency: 0.08147347268570909 ms
> Pending Flushes: 0
> Table: indexorganize
> SSTable count: 21
> Old SSTable count: 0
> Space used (live): 1536557040
> Space used (total): 1536557040
> Space used by snapshots (total): 1728378992
> Off heap memory used (total): 46251932
> SSTable Compression Ratio: 0.5218383898575761
> Number of partitions (estimate): 17365415
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 12
> Local read count: 17346304
> Local read latency: NaN ms
> Local write count: 31340451
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 100.0
> Bytes repaired: 1.084GiB
> Bytes unrepaired: 0.000KiB
> Bytes pending repair: 0.000KiB
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.00000
> Bloom filter space used: 38030728
> Bloom filter off heap memory used: 38030560
> Index summary off heap memory used: 7653060
> Compression metadata off heap memory used: 568312
> Compacted partition minimum bytes: 51
> Compacted partition maximum bytes: 86
> Compacted partition mean bytes: 67
> Average live cells per slice (last five minutes):
> 73.53164556962025
> Maximum live cells per slice (last five minutes): 5722
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
>
> nodetool tablehistograms doc.indexorganize
> doc/indexorganize histograms
> Percentile Read Latency Write Latency SSTables Partition
> Size Cell Count
> (micros) (micros) (bytes)
> 50% 0.00 0.00 0.00
> 60 1
> 75% 0.00 0.00 0.00
> 86 2
> 95% 0.00 0.00 0.00
> 86 2
> 98% 0.00 0.00 0.00
> 86 2
> 99% 0.00 0.00 0.00
> 86 2
> Min 0.00 0.00 0.00
> 51 0
> Max 0.00 0.00 0.00
> 86 2
>
> Any ideas on what I can do? Thank you!
>
> -Joe
>
>
--
James Brown
Engineer