You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Christian Winther (JIRA)" <ji...@apache.org> on 2015/09/24 12:44:07 UTC
[jira] [Updated] (CASSANDRA-10393) LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref)
[ https://issues.apache.org/jira/browse/CASSANDRA-10393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Winther updated CASSANDRA-10393:
------------------------------------------
Description:
When trying to repair full on a table with the following schema my nodes stall
and end up with spamming this
I've recently changed the table from SizeTieredCompactionStrategy to LeveledCompactionStrategy.
Coming from 2.1.9 -> 2.2.0 -> 2.2.1 i ran upgradesstable without issue as well
When trying to full repair post compaction change, I got "out of order" errors. A few google searches later, I was told to "scrub" the keyspace - did that during the night (no problems logged, and no data lost)
Now a repair just stalls and output memory leaks all over the place
{code}
CREATE KEYSPACE sessions WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;
CREATE TABLE sessions.sessions (
id text PRIMARY KEY,
client_ip text,
controller text,
controller_action text,
created timestamp,
data text,
expires timestamp,
http_host text,
modified timestamp,
request_agent text,
request_agent_bot boolean,
request_path text,
site_id int,
user_id int
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
{code}
{code}
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@4428a373) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@368dd97) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@66fb78be) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@9fdd2e6) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@84fcb91) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big was not released before the reference was garbage collected
{code}
was:
When trying to repair full on a table with the following schema my nodes stall
and end up with spamming this
I've recently changed the table from SizeTieredCompactionStrategy to LeveledCompactionStrategy.
Coming from 2.1.9 -> 2.2.0 -> 2.2.1 i ran upgradesstable without issue as well
When trying to full repair post compaction change, I got "out of order" errors. A few google searches later, I was told to "scrub" the keyspace - did that during the night (no problems logged, and no data lost)
Now a repair just stalls and output memory leaks all over the place
{code}
CREATE KEYSPACE sessions WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;
CREATE TABLE sessions.sessions (
id text PRIMARY KEY,
client_ip text,
controller text,
controller_action text,
created timestamp,
data text,
expires timestamp,
http_host text,
modified timestamp,
request_agent text,
request_agent_bot boolean,
request_path text,
site_id int,
user_id int
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
{code}
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@4428a373) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@368dd97) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@66fb78be) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@9fdd2e6) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@84fcb91) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big was not released before the reference was garbage collected
> LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref)
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-10393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10393
> Project: Cassandra
> Issue Type: Bug
> Environment: v 2.2.1 (from apt)
> -> lsb_release -a
> No LSB modules are available.
> Distributor ID: Debian
> Description: Debian GNU/Linux 7.8 (wheezy)
> Release: 7.8
> Codename: wheezy
> -> java -version
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
> Reporter: Christian Winther
> Labels: memory-leak, sstable
>
> When trying to repair full on a table with the following schema my nodes stall
> and end up with spamming this
> I've recently changed the table from SizeTieredCompactionStrategy to LeveledCompactionStrategy.
> Coming from 2.1.9 -> 2.2.0 -> 2.2.1 i ran upgradesstable without issue as well
> When trying to full repair post compaction change, I got "out of order" errors. A few google searches later, I was told to "scrub" the keyspace - did that during the night (no problems logged, and no data lost)
> Now a repair just stalls and output memory leaks all over the place
> {code}
> CREATE KEYSPACE sessions WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true;
> CREATE TABLE sessions.sessions (
> id text PRIMARY KEY,
> client_ip text,
> controller text,
> controller_action text,
> created timestamp,
> data text,
> expires timestamp,
> http_host text,
> modified timestamp,
> request_agent text,
> request_agent_bot boolean,
> request_path text,
> site_id int,
> user_id int
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> {code}
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@4428a373) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@368dd97) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@66fb78be) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1811114765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@9fdd2e6) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@84fcb91) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big was not released before the reference was garbage collected
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)