You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Cyril Scetbon (JIRA)" <ji...@apache.org> on 2014/05/16 17:07:14 UTC
[jira] [Updated] (CASSANDRA-7249) Too many threads associated with
parallel compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cyril Scetbon updated CASSANDRA-7249:
-------------------------------------
Description:
We have a lot of threads on some nodes as you can see :
node001: 560
node002: 529
node003: 4350
node004: 552
node005: 547
node006: 554
node007: 572
node008: 1444 <==
node009: 540
node010: 13691 <==
node011: 577
node012: 536
node013: 448
node014: 10295 <==
node015: 452
node016: 576
When I check what are those threads I see a lot of "Deserializer sstables".
Enabling DEBUG mode shows that a lot of actions are about parallel compaction. What is really surprising is that it tries to deserialize a huge number of times each sstable even if we only have 8 files for the concerned column family :
512690 /data/ks1/cf1/ks1-cf1-ic-616-Data.db
296623 /data/ks1/cf1/ks1-cf1-ic-637-Data.db
311904 /data/ks1/cf1/ks1-cf1-ic-642-Data.db
127061 /data/ks1/cf1/ks1-cf1-ic-643-Data.db
126921 /data/ks1/cf1/ks1-cf1-ic-644-Data.db
129815 /data/ks1/cf1/ks1-cf1-ic-645-Data.db
127862 /data/ks1/cf1/ks1-cf1-ic-646-Data.db
317069 /data/ks1/cf1/ks1-cf1-ic-647-Data.db
so, in a minute Cassandra execute 2 millions of times the following code :
{code}
else
{
logger.debug("parallel eager deserialize from " + iter.getPath());
queue.put(new RowContainer(new Row(iter.getKey(),
iter.getColumnFamilyWithColumns(ArrayBackedSortedColumns.factory()))));
}
{code}
It seems to be related to [CASSANDRA-5720|https://issues.apache.org/jira/browse/CASSANDRA-5720] cause we got the same error on the concerned column families before the number of threads raise. Upgrading to 2.0 is not a solution for now :(
was:
We have a lot of threads on some nodes as you can see :
node001: 560
node002: 529
node003: 4350
node004: 552
node005: 547
node006: 554
node007: 572
node008: 1444 <==
node009: 540
node010: 13691 <==
node011: 577
node012: 536
node013: 448
node014: 10295 <==
node015: 452
node016: 576
When I check what are those threads I see a lot of "Deserializer sstables".
Enabling DEBUG mode shows that a lot of actions are about parallel compaction. What is really surprising is that it tries to deserialize a huge number of times each sstable even if we only have 8 files for the concerned column family :
512690 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-616-Data.db
296623 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-637-Data.db
311904 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-642-Data.db
127061 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-643-Data.db
126921 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-644-Data.db
129815 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-645-Data.db
127862 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-646-Data.db
317069 /data/pns_fr_prod/syndic/pns_fr_prod-syndic-ic-647-Data.db
so, in a minute Cassandra execute 2 millions of times the following code :
{code}
else
{
logger.debug("parallel eager deserialize from " + iter.getPath());
queue.put(new RowContainer(new Row(iter.getKey(),
iter.getColumnFamilyWithColumns(ArrayBackedSortedColumns.factory()))));
}
{code}
It seems to be related to [CASSANDRA-5720|https://issues.apache.org/jira/browse/CASSANDRA-5720] cause we got the same error on the concerned column families before the number of threads raise. Upgrading to 2.0 is not a solution for now :(
> Too many threads associated with parallel compaction
> ----------------------------------------------------
>
> Key: CASSANDRA-7249
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7249
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 12.04.3 LTS
> 24 CPUs (hyper threading enabled)
> Reporter: Cyril Scetbon
> Labels: compaction, parallel, threads
>
> We have a lot of threads on some nodes as you can see :
> node001: 560
> node002: 529
> node003: 4350
> node004: 552
> node005: 547
> node006: 554
> node007: 572
> node008: 1444 <==
> node009: 540
> node010: 13691 <==
> node011: 577
> node012: 536
> node013: 448
> node014: 10295 <==
> node015: 452
> node016: 576
> When I check what are those threads I see a lot of "Deserializer sstables".
> Enabling DEBUG mode shows that a lot of actions are about parallel compaction. What is really surprising is that it tries to deserialize a huge number of times each sstable even if we only have 8 files for the concerned column family :
> 512690 /data/ks1/cf1/ks1-cf1-ic-616-Data.db
> 296623 /data/ks1/cf1/ks1-cf1-ic-637-Data.db
> 311904 /data/ks1/cf1/ks1-cf1-ic-642-Data.db
> 127061 /data/ks1/cf1/ks1-cf1-ic-643-Data.db
> 126921 /data/ks1/cf1/ks1-cf1-ic-644-Data.db
> 129815 /data/ks1/cf1/ks1-cf1-ic-645-Data.db
> 127862 /data/ks1/cf1/ks1-cf1-ic-646-Data.db
> 317069 /data/ks1/cf1/ks1-cf1-ic-647-Data.db
> so, in a minute Cassandra execute 2 millions of times the following code :
> {code}
> else
> {
> logger.debug("parallel eager deserialize from " + iter.getPath());
> queue.put(new RowContainer(new Row(iter.getKey(),
> iter.getColumnFamilyWithColumns(ArrayBackedSortedColumns.factory()))));
> }
> {code}
> It seems to be related to [CASSANDRA-5720|https://issues.apache.org/jira/browse/CASSANDRA-5720] cause we got the same error on the concerned column families before the number of threads raise. Upgrading to 2.0 is not a solution for now :(
--
This message was sent by Atlassian JIRA
(v6.2#6252)