You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sylvain Lebresne (JIRA)" <ji...@apache.org> on 2012/06/08 09:18:23 UTC
[jira] [Commented] (CASSANDRA-4314) Index CF tombstones can cause
OOM
[ https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291592#comment-13291592 ]
Sylvain Lebresne commented on CASSANDRA-4314:
---------------------------------------------
+1
> Index CF tombstones can cause OOM
> ---------------------------------
>
> Key: CASSANDRA-4314
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
> Reporter: Wade Poziombka
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.0.11, 1.1.2
>
> Attachments: 2012-06-07-compact.zip, 2012-06-07.zip, 4314-1.0.txt, 4314-1.1.txt, oom.zip, yourkitsnapshot.png
>
>
> My database (now at 1.0.10) is in a state in which it goes out of memory with hardly any activity at all. A key slice nothing more.
> The logs attached are this including verbose gc in stdout. I started up cassandra and waited a bit to ensure that it was unperturbed.
> Then (about 15:46) I ran this slice (using Pelops), which in this case should return NO data. My client times out and the database goes OOM.
> ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in my cluster
> Selector s = new Selector(this.pool);
> List<IndexExpression> indexExpressions = new ArrayList<IndexExpression>();
> IndexExpression e = new IndexExpression(
> ByteBuffer.wrap("encryptionSettingsID".getBytes(ASCII)), IndexOperator.EQ,
> ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
> indexExpressions.add(e);
> IndexClause indexClause = new IndexClause(indexExpressions,
> ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
> SlicePredicate predicate = new SlicePredicate();
> predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
> { ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
> List<KeySlice> slices = s.getKeySlices(CF_TOKEN, indexClause, predicate, cl);
> Note that “encryptionSettingsID” is an indexed column. When this is executed there should be no columns with the supplied value.
> I suppose I may have some kind of blatant error in this query but it is not obvious to me. I’m relatively new to cassandra.
> My key space is defined as follows:
> KsDef(name:TB_UNIT, strategy_class:org.apache.cassandra.locator.SimpleStrategy, strategy_options:{replication_factor=3},
> cf_defs:[
> CfDef(keyspace:TB_UNIT, name:token, column_type:Standard, comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63, validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73, validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65, validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate), ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44, validation_class:UTF8Type, index_type:KEYS, index_name:EncryptionSettingsID)], caching:keys_only),
> CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6, column_type:Standard, comparator_type:BytesType, caching:keys_only),
> CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard, comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75 6E 74, validation_class:CounterColumnType)], default_validation_class:CounterColumnType, caching:keys_only)
> ])
> tpstats show pending tasks many minutes after time out:
> [root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
> Pool Name Active Pending Completed Blocked All time blocked
> ReadStage 3 3 107 0 0
> RequestResponseStage 0 0 56 0 0
> MutationStage 0 0 6 0 0
> ReadRepairStage 0 0 0 0 0
> ReplicateOnWriteStage 0 0 0 0 0
> GossipStage 0 0 2231 0 0
> AntiEntropyStage 0 0 0 0 0
> MigrationStage 0 0 0 0 0
> MemtablePostFlusher 0 0 3 0 0
> StreamStage 0 0 0 0 0
> FlushWriter 0 0 3 0 0
> MiscStage 0 0 0 0 0
> InternalResponseStage 0 0 0 0 0
> HintedHandoff 0 0 9 0 0
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 0
> BINARY 0
> READ 0
> MUTATION 0
> REQUEST_RESPONSE 0
> cfstats:
> Keyspace: keyspace
> Read Count: 118
> Read Latency: 0.14722033898305084 ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Column Family: token
> SSTable count: 7
> Space used (live): 4745885584
> Space used (total): 4745885584
> Number of Keys (estimate): 18626048
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 118
> Read Latency: 0.147 ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 0
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 55058352
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 150
> Compacted row maximum size: 258
> Compacted row mean size: 201
> Column Family: pan_2fef6478b62242dd94aecaa049b9d7bb
> SSTable count: 7
> Space used (live): 1987147156
> Space used (total): 1987147156
> Number of Keys (estimate): 14955264
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 0
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 28056224
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 104
> Compacted row maximum size: 124
> Compacted row mean size: 124
> Column Family: counters
> SSTable count: 11
> Space used (live): 3433469364
> Space used (total): 3433469364
> Number of Keys (estimate): 21475328
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 0
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 40271696
> Key cache capacity: 4652
> Key cache size: 4652
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 125
> Compacted row maximum size: 179
> Compacted row mean size: 150
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira