You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jean Carlo <je...@gmail.com> on 2016/02/10 14:33:07 UTC
3k sstables during a repair incremental !!
Hello guys!
I am testing the repair inc in my custer cassandra. I am doing my test over
these tables
*CREATE TABLE pns_nonreg_bench.cf3* (
s text,
sp int,
d text,
dp int,
m map<text, text>,
t timestamp,
PRIMARY KEY (s, sp, d, dp)
) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
AND compaction = {'class': 'org.apache.cassandra.db.compaction.
*LeveledCompactionStrategy*'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.*SnappyCompressor*'}
*CREATE TABLE pns_nonreg_bench.cf1* (
ise text PRIMARY KEY,
int_col int,
text_col text,
ts_col timestamp,
uuid_col uuid
) WITH bloom_filter_fp_chance = 0.01
AND compaction = {'class': 'org.apache.cassandra.db.compaction.
*LeveledCompactionStrategy*'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.*SnappyCompressor*'}
*table cf1 Space used (live): 665.7 MB*
*table cf2*
* Space used (live): 697.03 MB*
It happens that when I do repair -inc -par on theses tables, *cf2 got a
pick of 3k sstables*. When the repair finish, it takes 30 min or more to
finish all the compactations and return to 6 sstable.
I am a little concern about if this will happen on production. is it normal?
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
Re: 3k sstables during a repair incremental !!
Posted by Jean Carlo <je...@gmail.com>.
Hi !
@Paulo
.
Yes we are using vnodes, 256 per node and we have 6 nodes in the cluster.
RF=3. The data is inserted using jmeter with consistency=LOCAL_ONE.
Because this is a test, we generate our data and we insert them using
jmeter.
After the repair finished, all the nodes seems to be freezed, no compaction
were runing, even if nodetool tpstats says
CompactionExecutor
2 83 1424 0 0
After I restart cassandra, the comactions started to run.
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Thu, Feb 11, 2016 at 8:42 AM, Marcus Eriksson <kr...@gmail.com> wrote:
> The reason for this is probably
> https://issues.apache.org/jira/browse/CASSANDRA-10831 (which only affects
> 2.1)
>
> So, if you had problems with incremental repair and LCS before, upgrade to
> 2.1.13 and try again
>
> /Marcus
>
> On Wed, Feb 10, 2016 at 2:59 PM, horschi <ho...@gmail.com> wrote:
>
>> Hi Jean,
>>
>> we had the same issue, but on SizeTieredCompaction. During repair the
>> number of SSTables and pending compactions were exploding.
>>
>> It not only affected latencies, at some point Cassandra ran out of heap.
>>
>> After the upgrade to 2.2 things got much better.
>>
>> regards,
>> Christian
>>
>>
>> On Wed, Feb 10, 2016 at 2:46 PM, Jean Carlo <je...@gmail.com>
>> wrote:
>> > Hi Horschi !!!
>> >
>> > I have the 2.1.12. But I think it is something related to Level
>> compaction
>> > strategy. It is impressive that we passed from 6 sstables to 3k sstable.
>> > I think this will affect the latency on production because the number of
>> > compactions going on
>> >
>> >
>> >
>> > Best regards
>> >
>> > Jean Carlo
>> >
>> > "The best way to predict the future is to invent it" Alan Kay
>> >
>> > On Wed, Feb 10, 2016 at 2:37 PM, horschi <ho...@gmail.com> wrote:
>> >>
>> >> Hi Jean,
>> >>
>> >> which Cassandra version do you use?
>> >>
>> >> Incremental repair got much better in 2.2 (for us at least).
>> >>
>> >> kind regards,
>> >> Christian
>> >>
>> >> On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <jean.jeancarl48@gmail.com
>> >
>> >> wrote:
>> >> > Hello guys!
>> >> >
>> >> > I am testing the repair inc in my custer cassandra. I am doing my
>> test
>> >> > over
>> >> > these tables
>> >> >
>> >> > CREATE TABLE pns_nonreg_bench.cf3 (
>> >> > s text,
>> >> > sp int,
>> >> > d text,
>> >> > dp int,
>> >> > m map<text, text>,
>> >> > t timestamp,
>> >> > PRIMARY KEY (s, sp, d, dp)
>> >> > ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>> >> >
>> >> > AND compaction = {'class':
>> >> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>> >> > AND compression = {'sstable_compression':
>> >> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
>> >> >
>> >> > CREATE TABLE pns_nonreg_bench.cf1 (
>> >> > ise text PRIMARY KEY,
>> >> > int_col int,
>> >> > text_col text,
>> >> > ts_col timestamp,
>> >> > uuid_col uuid
>> >> > ) WITH bloom_filter_fp_chance = 0.01
>> >> > AND compaction = {'class':
>> >> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>> >> > AND compression = {'sstable_compression':
>> >> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
>> >> >
>> >> > table cf1
>> >> > Space used (live): 665.7 MB
>> >> > table cf2
>> >> > Space used (live): 697.03 MB
>> >> >
>> >> > It happens that when I do repair -inc -par on theses tables, cf2 got
>> a
>> >> > pick
>> >> > of 3k sstables. When the repair finish, it takes 30 min or more to
>> >> > finish
>> >> > all the compactations and return to 6 sstable.
>> >> >
>> >> > I am a little concern about if this will happen on production. is it
>> >> > normal?
>> >> >
>> >> > Saludos
>> >> >
>> >> > Jean Carlo
>> >> >
>> >> > "The best way to predict the future is to invent it" Alan Kay
>> >
>> >
>>
>
>
Re: 3k sstables during a repair incremental !!
Posted by Marcus Eriksson <kr...@gmail.com>.
The reason for this is probably
https://issues.apache.org/jira/browse/CASSANDRA-10831 (which only affects
2.1)
So, if you had problems with incremental repair and LCS before, upgrade to
2.1.13 and try again
/Marcus
On Wed, Feb 10, 2016 at 2:59 PM, horschi <ho...@gmail.com> wrote:
> Hi Jean,
>
> we had the same issue, but on SizeTieredCompaction. During repair the
> number of SSTables and pending compactions were exploding.
>
> It not only affected latencies, at some point Cassandra ran out of heap.
>
> After the upgrade to 2.2 things got much better.
>
> regards,
> Christian
>
>
> On Wed, Feb 10, 2016 at 2:46 PM, Jean Carlo <je...@gmail.com>
> wrote:
> > Hi Horschi !!!
> >
> > I have the 2.1.12. But I think it is something related to Level
> compaction
> > strategy. It is impressive that we passed from 6 sstables to 3k sstable.
> > I think this will affect the latency on production because the number of
> > compactions going on
> >
> >
> >
> > Best regards
> >
> > Jean Carlo
> >
> > "The best way to predict the future is to invent it" Alan Kay
> >
> > On Wed, Feb 10, 2016 at 2:37 PM, horschi <ho...@gmail.com> wrote:
> >>
> >> Hi Jean,
> >>
> >> which Cassandra version do you use?
> >>
> >> Incremental repair got much better in 2.2 (for us at least).
> >>
> >> kind regards,
> >> Christian
> >>
> >> On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <je...@gmail.com>
> >> wrote:
> >> > Hello guys!
> >> >
> >> > I am testing the repair inc in my custer cassandra. I am doing my test
> >> > over
> >> > these tables
> >> >
> >> > CREATE TABLE pns_nonreg_bench.cf3 (
> >> > s text,
> >> > sp int,
> >> > d text,
> >> > dp int,
> >> > m map<text, text>,
> >> > t timestamp,
> >> > PRIMARY KEY (s, sp, d, dp)
> >> > ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
> >> >
> >> > AND compaction = {'class':
> >> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> >> > AND compression = {'sstable_compression':
> >> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >> >
> >> > CREATE TABLE pns_nonreg_bench.cf1 (
> >> > ise text PRIMARY KEY,
> >> > int_col int,
> >> > text_col text,
> >> > ts_col timestamp,
> >> > uuid_col uuid
> >> > ) WITH bloom_filter_fp_chance = 0.01
> >> > AND compaction = {'class':
> >> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> >> > AND compression = {'sstable_compression':
> >> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >> >
> >> > table cf1
> >> > Space used (live): 665.7 MB
> >> > table cf2
> >> > Space used (live): 697.03 MB
> >> >
> >> > It happens that when I do repair -inc -par on theses tables, cf2 got a
> >> > pick
> >> > of 3k sstables. When the repair finish, it takes 30 min or more to
> >> > finish
> >> > all the compactations and return to 6 sstable.
> >> >
> >> > I am a little concern about if this will happen on production. is it
> >> > normal?
> >> >
> >> > Saludos
> >> >
> >> > Jean Carlo
> >> >
> >> > "The best way to predict the future is to invent it" Alan Kay
> >
> >
>
Re: 3k sstables during a repair incremental !!
Posted by Jean Carlo <je...@gmail.com>.
Hello Horschi,
Yes I understand. Thx!!!!
Best regards
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Feb 10, 2016 at 3:00 PM, horschi <ho...@gmail.com> wrote:
> btw: I am not saying incremental Repair in 2.1 is broken, but ... ;-)
>
> On Wed, Feb 10, 2016 at 2:59 PM, horschi <ho...@gmail.com> wrote:
> > Hi Jean,
> >
> > we had the same issue, but on SizeTieredCompaction. During repair the
> > number of SSTables and pending compactions were exploding.
> >
> > It not only affected latencies, at some point Cassandra ran out of heap.
> >
> > After the upgrade to 2.2 things got much better.
> >
> > regards,
> > Christian
> >
> >
> > On Wed, Feb 10, 2016 at 2:46 PM, Jean Carlo <je...@gmail.com>
> wrote:
> >> Hi Horschi !!!
> >>
> >> I have the 2.1.12. But I think it is something related to Level
> compaction
> >> strategy. It is impressive that we passed from 6 sstables to 3k sstable.
> >> I think this will affect the latency on production because the number of
> >> compactions going on
> >>
> >>
> >>
> >> Best regards
> >>
> >> Jean Carlo
> >>
> >> "The best way to predict the future is to invent it" Alan Kay
> >>
> >> On Wed, Feb 10, 2016 at 2:37 PM, horschi <ho...@gmail.com> wrote:
> >>>
> >>> Hi Jean,
> >>>
> >>> which Cassandra version do you use?
> >>>
> >>> Incremental repair got much better in 2.2 (for us at least).
> >>>
> >>> kind regards,
> >>> Christian
> >>>
> >>> On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <jean.jeancarl48@gmail.com
> >
> >>> wrote:
> >>> > Hello guys!
> >>> >
> >>> > I am testing the repair inc in my custer cassandra. I am doing my
> test
> >>> > over
> >>> > these tables
> >>> >
> >>> > CREATE TABLE pns_nonreg_bench.cf3 (
> >>> > s text,
> >>> > sp int,
> >>> > d text,
> >>> > dp int,
> >>> > m map<text, text>,
> >>> > t timestamp,
> >>> > PRIMARY KEY (s, sp, d, dp)
> >>> > ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
> >>> >
> >>> > AND compaction = {'class':
> >>> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> >>> > AND compression = {'sstable_compression':
> >>> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >>> >
> >>> > CREATE TABLE pns_nonreg_bench.cf1 (
> >>> > ise text PRIMARY KEY,
> >>> > int_col int,
> >>> > text_col text,
> >>> > ts_col timestamp,
> >>> > uuid_col uuid
> >>> > ) WITH bloom_filter_fp_chance = 0.01
> >>> > AND compaction = {'class':
> >>> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> >>> > AND compression = {'sstable_compression':
> >>> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >>> >
> >>> > table cf1
> >>> > Space used (live): 665.7 MB
> >>> > table cf2
> >>> > Space used (live): 697.03 MB
> >>> >
> >>> > It happens that when I do repair -inc -par on theses tables, cf2 got
> a
> >>> > pick
> >>> > of 3k sstables. When the repair finish, it takes 30 min or more to
> >>> > finish
> >>> > all the compactations and return to 6 sstable.
> >>> >
> >>> > I am a little concern about if this will happen on production. is it
> >>> > normal?
> >>> >
> >>> > Saludos
> >>> >
> >>> > Jean Carlo
> >>> >
> >>> > "The best way to predict the future is to invent it" Alan Kay
> >>
> >>
>
Re: 3k sstables during a repair incremental !!
Posted by horschi <ho...@gmail.com>.
btw: I am not saying incremental Repair in 2.1 is broken, but ... ;-)
On Wed, Feb 10, 2016 at 2:59 PM, horschi <ho...@gmail.com> wrote:
> Hi Jean,
>
> we had the same issue, but on SizeTieredCompaction. During repair the
> number of SSTables and pending compactions were exploding.
>
> It not only affected latencies, at some point Cassandra ran out of heap.
>
> After the upgrade to 2.2 things got much better.
>
> regards,
> Christian
>
>
> On Wed, Feb 10, 2016 at 2:46 PM, Jean Carlo <je...@gmail.com> wrote:
>> Hi Horschi !!!
>>
>> I have the 2.1.12. But I think it is something related to Level compaction
>> strategy. It is impressive that we passed from 6 sstables to 3k sstable.
>> I think this will affect the latency on production because the number of
>> compactions going on
>>
>>
>>
>> Best regards
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>> On Wed, Feb 10, 2016 at 2:37 PM, horschi <ho...@gmail.com> wrote:
>>>
>>> Hi Jean,
>>>
>>> which Cassandra version do you use?
>>>
>>> Incremental repair got much better in 2.2 (for us at least).
>>>
>>> kind regards,
>>> Christian
>>>
>>> On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <je...@gmail.com>
>>> wrote:
>>> > Hello guys!
>>> >
>>> > I am testing the repair inc in my custer cassandra. I am doing my test
>>> > over
>>> > these tables
>>> >
>>> > CREATE TABLE pns_nonreg_bench.cf3 (
>>> > s text,
>>> > sp int,
>>> > d text,
>>> > dp int,
>>> > m map<text, text>,
>>> > t timestamp,
>>> > PRIMARY KEY (s, sp, d, dp)
>>> > ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>>> >
>>> > AND compaction = {'class':
>>> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>>> > AND compression = {'sstable_compression':
>>> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
>>> >
>>> > CREATE TABLE pns_nonreg_bench.cf1 (
>>> > ise text PRIMARY KEY,
>>> > int_col int,
>>> > text_col text,
>>> > ts_col timestamp,
>>> > uuid_col uuid
>>> > ) WITH bloom_filter_fp_chance = 0.01
>>> > AND compaction = {'class':
>>> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>>> > AND compression = {'sstable_compression':
>>> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
>>> >
>>> > table cf1
>>> > Space used (live): 665.7 MB
>>> > table cf2
>>> > Space used (live): 697.03 MB
>>> >
>>> > It happens that when I do repair -inc -par on theses tables, cf2 got a
>>> > pick
>>> > of 3k sstables. When the repair finish, it takes 30 min or more to
>>> > finish
>>> > all the compactations and return to 6 sstable.
>>> >
>>> > I am a little concern about if this will happen on production. is it
>>> > normal?
>>> >
>>> > Saludos
>>> >
>>> > Jean Carlo
>>> >
>>> > "The best way to predict the future is to invent it" Alan Kay
>>
>>
Re: 3k sstables during a repair incremental !!
Posted by horschi <ho...@gmail.com>.
Hi Jean,
we had the same issue, but on SizeTieredCompaction. During repair the
number of SSTables and pending compactions were exploding.
It not only affected latencies, at some point Cassandra ran out of heap.
After the upgrade to 2.2 things got much better.
regards,
Christian
On Wed, Feb 10, 2016 at 2:46 PM, Jean Carlo <je...@gmail.com> wrote:
> Hi Horschi !!!
>
> I have the 2.1.12. But I think it is something related to Level compaction
> strategy. It is impressive that we passed from 6 sstables to 3k sstable.
> I think this will affect the latency on production because the number of
> compactions going on
>
>
>
> Best regards
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Wed, Feb 10, 2016 at 2:37 PM, horschi <ho...@gmail.com> wrote:
>>
>> Hi Jean,
>>
>> which Cassandra version do you use?
>>
>> Incremental repair got much better in 2.2 (for us at least).
>>
>> kind regards,
>> Christian
>>
>> On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <je...@gmail.com>
>> wrote:
>> > Hello guys!
>> >
>> > I am testing the repair inc in my custer cassandra. I am doing my test
>> > over
>> > these tables
>> >
>> > CREATE TABLE pns_nonreg_bench.cf3 (
>> > s text,
>> > sp int,
>> > d text,
>> > dp int,
>> > m map<text, text>,
>> > t timestamp,
>> > PRIMARY KEY (s, sp, d, dp)
>> > ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>> >
>> > AND compaction = {'class':
>> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>> > AND compression = {'sstable_compression':
>> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
>> >
>> > CREATE TABLE pns_nonreg_bench.cf1 (
>> > ise text PRIMARY KEY,
>> > int_col int,
>> > text_col text,
>> > ts_col timestamp,
>> > uuid_col uuid
>> > ) WITH bloom_filter_fp_chance = 0.01
>> > AND compaction = {'class':
>> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>> > AND compression = {'sstable_compression':
>> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
>> >
>> > table cf1
>> > Space used (live): 665.7 MB
>> > table cf2
>> > Space used (live): 697.03 MB
>> >
>> > It happens that when I do repair -inc -par on theses tables, cf2 got a
>> > pick
>> > of 3k sstables. When the repair finish, it takes 30 min or more to
>> > finish
>> > all the compactations and return to 6 sstable.
>> >
>> > I am a little concern about if this will happen on production. is it
>> > normal?
>> >
>> > Saludos
>> >
>> > Jean Carlo
>> >
>> > "The best way to predict the future is to invent it" Alan Kay
>
>
Re: 3k sstables during a repair incremental !!
Posted by Jean Carlo <je...@gmail.com>.
Hi Horschi !!!
I have the 2.1.12. But I think it is something related to Level compaction
strategy. It is impressive that we passed from 6 sstables to 3k sstable.
I think this will affect the latency on production because the number of
compactions going on
Best regards
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Feb 10, 2016 at 2:37 PM, horschi <ho...@gmail.com> wrote:
> Hi Jean,
>
> which Cassandra version do you use?
>
> Incremental repair got much better in 2.2 (for us at least).
>
> kind regards,
> Christian
>
> On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <je...@gmail.com>
> wrote:
> > Hello guys!
> >
> > I am testing the repair inc in my custer cassandra. I am doing my test
> over
> > these tables
> >
> > CREATE TABLE pns_nonreg_bench.cf3 (
> > s text,
> > sp int,
> > d text,
> > dp int,
> > m map<text, text>,
> > t timestamp,
> > PRIMARY KEY (s, sp, d, dp)
> > ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
> >
> > AND compaction = {'class':
> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> > AND compression = {'sstable_compression':
> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >
> > CREATE TABLE pns_nonreg_bench.cf1 (
> > ise text PRIMARY KEY,
> > int_col int,
> > text_col text,
> > ts_col timestamp,
> > uuid_col uuid
> > ) WITH bloom_filter_fp_chance = 0.01
> > AND compaction = {'class':
> > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> > AND compression = {'sstable_compression':
> > 'org.apache.cassandra.io.compress.SnappyCompressor'}
> >
> > table cf1
> > Space used (live): 665.7 MB
> > table cf2
> > Space used (live): 697.03 MB
> >
> > It happens that when I do repair -inc -par on theses tables, cf2 got a
> pick
> > of 3k sstables. When the repair finish, it takes 30 min or more to finish
> > all the compactations and return to 6 sstable.
> >
> > I am a little concern about if this will happen on production. is it
> normal?
> >
> > Saludos
> >
> > Jean Carlo
> >
> > "The best way to predict the future is to invent it" Alan Kay
>
Re: 3k sstables during a repair incremental !!
Posted by horschi <ho...@gmail.com>.
Hi Jean,
which Cassandra version do you use?
Incremental repair got much better in 2.2 (for us at least).
kind regards,
Christian
On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <je...@gmail.com> wrote:
> Hello guys!
>
> I am testing the repair inc in my custer cassandra. I am doing my test over
> these tables
>
> CREATE TABLE pns_nonreg_bench.cf3 (
> s text,
> sp int,
> d text,
> dp int,
> m map<text, text>,
> t timestamp,
> PRIMARY KEY (s, sp, d, dp)
> ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
>
> CREATE TABLE pns_nonreg_bench.cf1 (
> ise text PRIMARY KEY,
> int_col int,
> text_col text,
> ts_col timestamp,
> uuid_col uuid
> ) WITH bloom_filter_fp_chance = 0.01
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
>
> table cf1
> Space used (live): 665.7 MB
> table cf2
> Space used (live): 697.03 MB
>
> It happens that when I do repair -inc -par on theses tables, cf2 got a pick
> of 3k sstables. When the repair finish, it takes 30 min or more to finish
> all the compactations and return to 6 sstable.
>
> I am a little concern about if this will happen on production. is it normal?
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
Re: 3k sstables during a repair incremental !!
Posted by Paulo Motta <pa...@gmail.com>.
Are you using vnodes by any chance? If so, how many? How many nodes and
what's the replication factor? How was data inserted (at what consistency
level)?
Streaming might create a large number of sstables with vnodes (see
CASSANDRA-10495), so in case data is inconsistent between nodes (detected
during the validation phase) a large number of sstables might be created
during the streaming phase of the repair. This might happen with or without
incremental repairs.
There is also a migration step from non-incremental to incremental repair
that must be followed for LCS tables to avoid recompacting all sstables
again. You may find more information in the migration procedure on
http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesMigration.html
2016-02-10 14:16 GMT-03:00 Jean Carlo <je...@gmail.com>:
> Hello Kai
>
> This is for *cf3*
>
> nodetool cfstats pns_nonreg_bench.cf3 -H
> Keyspace: pns_nonreg_bench
> Read Count: 23594
> Read Latency: 1.2980987539204882 ms.
> Write Count: 148161
> Write Latency: 0.04608940274431193 ms.
> Pending Flushes: 0
> Table: cf3
> SSTable count: 11489
> SSTables in each level: [11485/4, 4, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 954.89 MB
> Space used (total): 954.89 MB
> Space used by snapshots (total): 0 bytes
> Off heap memory used (total): 3.74 MB
> SSTable Compression Ratio: 0.4245371280396339
> Number of keys (estimate): 1051613
> Memtable cell count: 0
> Memtable data size: 0 bytes
> Memtable off heap memory used: 0 bytes
> Memtable switch count: 889
> Local read count: 23594
> Local read latency: 1.299 ms
> Local write count: 148161
> Local write latency: 0.047 ms
> Pending flushes: 0
> Bloom filter false positives: 16609
> Bloom filter false ratio: 0.00000
> Bloom filter space used: 1.64 MB
> Bloom filter off heap memory used: 1.55 MB
> Index summary off heap memory used: 1.88 MB
> Compression metadata off heap memory used: 318.56 KB
> Compacted partition minimum bytes: 643 bytes
> Compacted partition maximum bytes: 924 bytes
> Compacted partition mean bytes: 914 bytes
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1.0
> Average tombstones per slice (last five minutes): 0.0
> Maximum tombstones per slice (last five minutes): 0.0
>
> ----------------
>
> This is for *cf1*
>
> nodetool cfstats pns_nonreg_bench.cf1 -H
> Keyspace: pns_nonreg_bench
> Read Count: 24207
> Read Latency: 0.9072712851654481 ms.
> Write Count: 148380
> Write Latency: 0.04561746866154468 ms.
> Pending Flushes: 0
> Table: cf1
> SSTable count: 1942
>
> It seems that nodetool does not retrieve the rest of the information, I
> think it is due to compactions
>
> nodetool tpstats
> CompactionExecutor 2 83 1424
> 0 0
>
>
>
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Wed, Feb 10, 2016 at 6:00 PM, Kai Wang <de...@gmail.com> wrote:
>
>> Jean,
>>
>> What does your cfstats look like? Especially "SSTables in each level"
>> line.
>>
>> On Wed, Feb 10, 2016 at 8:33 AM, Jean Carlo <je...@gmail.com>
>> wrote:
>>
>>> Hello guys!
>>>
>>> I am testing the repair inc in my custer cassandra. I am doing my test
>>> over these tables
>>>
>>> *CREATE TABLE pns_nonreg_bench.cf3* (
>>> s text,
>>> sp int,
>>> d text,
>>> dp int,
>>> m map<text, text>,
>>> t timestamp,
>>> PRIMARY KEY (s, sp, d, dp)
>>> ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>>>
>>> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
>>> *LeveledCompactionStrategy*'}
>>> AND compression = {'sstable_compression':
>>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>>
>>> *CREATE TABLE pns_nonreg_bench.cf1* (
>>> ise text PRIMARY KEY,
>>> int_col int,
>>> text_col text,
>>> ts_col timestamp,
>>> uuid_col uuid
>>> ) WITH bloom_filter_fp_chance = 0.01
>>> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
>>> *LeveledCompactionStrategy*'}
>>> AND compression = {'sstable_compression':
>>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>>
>>>
>>>
>>> *table cf1 Space used (live): 665.7 MB*
>>>
>>> *table cf2*
>>> * Space used (live): 697.03 MB*
>>>
>>> It happens that when I do repair -inc -par on theses tables, *cf2 got a
>>> pick of 3k sstables*. When the repair finish, it takes 30 min or more
>>> to finish all the compactations and return to 6 sstable.
>>>
>>> I am a little concern about if this will happen on production. is it
>>> normal?
>>>
>>> Saludos
>>>
>>> Jean Carlo
>>>
>>> "The best way to predict the future is to invent it" Alan Kay
>>>
>>
>>
>
Re: 3k sstables during a repair incremental !!
Posted by Jean Carlo <je...@gmail.com>.
Hello Kai
This is for *cf3*
nodetool cfstats pns_nonreg_bench.cf3 -H
Keyspace: pns_nonreg_bench
Read Count: 23594
Read Latency: 1.2980987539204882 ms.
Write Count: 148161
Write Latency: 0.04608940274431193 ms.
Pending Flushes: 0
Table: cf3
SSTable count: 11489
SSTables in each level: [11485/4, 4, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 954.89 MB
Space used (total): 954.89 MB
Space used by snapshots (total): 0 bytes
Off heap memory used (total): 3.74 MB
SSTable Compression Ratio: 0.4245371280396339
Number of keys (estimate): 1051613
Memtable cell count: 0
Memtable data size: 0 bytes
Memtable off heap memory used: 0 bytes
Memtable switch count: 889
Local read count: 23594
Local read latency: 1.299 ms
Local write count: 148161
Local write latency: 0.047 ms
Pending flushes: 0
Bloom filter false positives: 16609
Bloom filter false ratio: 0.00000
Bloom filter space used: 1.64 MB
Bloom filter off heap memory used: 1.55 MB
Index summary off heap memory used: 1.88 MB
Compression metadata off heap memory used: 318.56 KB
Compacted partition minimum bytes: 643 bytes
Compacted partition maximum bytes: 924 bytes
Compacted partition mean bytes: 914 bytes
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
----------------
This is for *cf1*
nodetool cfstats pns_nonreg_bench.cf1 -H
Keyspace: pns_nonreg_bench
Read Count: 24207
Read Latency: 0.9072712851654481 ms.
Write Count: 148380
Write Latency: 0.04561746866154468 ms.
Pending Flushes: 0
Table: cf1
SSTable count: 1942
It seems that nodetool does not retrieve the rest of the information, I
think it is due to compactions
nodetool tpstats
CompactionExecutor 2 83 1424
0 0
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Feb 10, 2016 at 6:00 PM, Kai Wang <de...@gmail.com> wrote:
> Jean,
>
> What does your cfstats look like? Especially "SSTables in each level" line.
>
> On Wed, Feb 10, 2016 at 8:33 AM, Jean Carlo <je...@gmail.com>
> wrote:
>
>> Hello guys!
>>
>> I am testing the repair inc in my custer cassandra. I am doing my test
>> over these tables
>>
>> *CREATE TABLE pns_nonreg_bench.cf3* (
>> s text,
>> sp int,
>> d text,
>> dp int,
>> m map<text, text>,
>> t timestamp,
>> PRIMARY KEY (s, sp, d, dp)
>> ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>>
>> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
>> *LeveledCompactionStrategy*'}
>> AND compression = {'sstable_compression':
>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>
>> *CREATE TABLE pns_nonreg_bench.cf1* (
>> ise text PRIMARY KEY,
>> int_col int,
>> text_col text,
>> ts_col timestamp,
>> uuid_col uuid
>> ) WITH bloom_filter_fp_chance = 0.01
>> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
>> *LeveledCompactionStrategy*'}
>> AND compression = {'sstable_compression':
>> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>>
>>
>>
>> *table cf1 Space used (live): 665.7 MB*
>>
>> *table cf2*
>> * Space used (live): 697.03 MB*
>>
>> It happens that when I do repair -inc -par on theses tables, *cf2 got a
>> pick of 3k sstables*. When the repair finish, it takes 30 min or more to
>> finish all the compactations and return to 6 sstable.
>>
>> I am a little concern about if this will happen on production. is it
>> normal?
>>
>> Saludos
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>
>
Re: 3k sstables during a repair incremental !!
Posted by Kai Wang <de...@gmail.com>.
Jean,
What does your cfstats look like? Especially "SSTables in each level" line.
On Wed, Feb 10, 2016 at 8:33 AM, Jean Carlo <je...@gmail.com>
wrote:
> Hello guys!
>
> I am testing the repair inc in my custer cassandra. I am doing my test
> over these tables
>
> *CREATE TABLE pns_nonreg_bench.cf3* (
> s text,
> sp int,
> d text,
> dp int,
> m map<text, text>,
> t timestamp,
> PRIMARY KEY (s, sp, d, dp)
> ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
> *LeveledCompactionStrategy*'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>
> *CREATE TABLE pns_nonreg_bench.cf1* (
> ise text PRIMARY KEY,
> int_col int,
> text_col text,
> ts_col timestamp,
> uuid_col uuid
> ) WITH bloom_filter_fp_chance = 0.01
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
> *LeveledCompactionStrategy*'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>
>
>
> *table cf1 Space used (live): 665.7 MB*
>
> *table cf2*
> * Space used (live): 697.03 MB*
>
> It happens that when I do repair -inc -par on theses tables, *cf2 got a
> pick of 3k sstables*. When the repair finish, it takes 30 min or more to
> finish all the compactations and return to 6 sstable.
>
> I am a little concern about if this will happen on production. is it
> normal?
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
Re: 3k sstables during a repair incremental !!
Posted by Jean Carlo <je...@gmail.com>.
Correction:
*table cf3*
* Space used (live): 697.03 MB*
It happens that when I do repair -inc -par on theses tables, *cf3 got a
pick of 3k sstables*. When the repair finish, it takes 30 min or more to
finish all the compactations and return to 6 sstable.
Saludos
Jean Carlo
"The best way to predict the future is to invent it" Alan Kay
On Wed, Feb 10, 2016 at 2:33 PM, Jean Carlo <je...@gmail.com>
wrote:
> Hello guys!
>
> I am testing the repair inc in my custer cassandra. I am doing my test
> over these tables
>
> *CREATE TABLE pns_nonreg_bench.cf3* (
> s text,
> sp int,
> d text,
> dp int,
> m map<text, text>,
> t timestamp,
> PRIMARY KEY (s, sp, d, dp)
> ) WITH CLUSTERING ORDER BY (sp ASC, d ASC, dp ASC)
>
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
> *LeveledCompactionStrategy*'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>
> *CREATE TABLE pns_nonreg_bench.cf1* (
> ise text PRIMARY KEY,
> int_col int,
> text_col text,
> ts_col timestamp,
> uuid_col uuid
> ) WITH bloom_filter_fp_chance = 0.01
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.
> *LeveledCompactionStrategy*'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.*SnappyCompressor*'}
>
>
>
> *table cf1 Space used (live): 665.7 MB*
>
> *table cf2*
> * Space used (live): 697.03 MB*
>
> It happens that when I do repair -inc -par on theses tables, *cf2 got a
> pick of 3k sstables*. When the repair finish, it takes 30 min or more to
> finish all the compactations and return to 6 sstable.
>
> I am a little concern about if this will happen on production. is it
> normal?
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>