You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Chris Lowe (JIRA)" <ji...@apache.org> on 2015/07/22 13:45:05 UTC
[jira] [Commented] (CASSANDRA-6937) On large insert, node is being
marked down with Commit Log failure
[ https://issues.apache.org/jira/browse/CASSANDRA-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636759#comment-14636759 ]
Chris Lowe commented on CASSANDRA-6937:
---------------------------------------
This could be down to bad ownership of the files. It could be caused by Cassandra being executed under the cassandra:cassandra account, then under root:root, then cassandra:cassandra again.
The fix is to change the ownership of some of the files that cassandra creates, so that they are all owned by cassandra:cassandra. This can be done using
chown -R cassandra:cassandra .
within the cassandra-generated directory as specified in the conf file.
> On large insert, node is being marked down with Commit Log failure
> ------------------------------------------------------------------
>
> Key: CASSANDRA-6937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6937
> Project: Cassandra
> Issue Type: Bug
> Environment: 16 vcpu, 480GB SSD storage (used 7GB), 32GB RAM, Cassandra 2.0.5.24, hsha, datastax Java driver 2.0.1, vnodes
> Reporter: jonathan lacefield
> Assignee: Benedict
> Priority: Minor
> Attachments: cassandra.yaml copy, system.log
>
>
> during large insert, will get specs on insert load, nodes in the cluster are being marked down. creating defect for investigation.
> error message in log file of one of the down nodes:
> ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:06,666 StorageService.java (line 364) Stopping gossiper
> WARN [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:06,666 StorageService.java (line 278) Stopping gossip by operator request
> INFO [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:06,666 Gossiper.java (line 1251) Announcing shutdown
> INFO [FlushWriter:421] 2014-03-26 08:28:06,677 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data1/system/compactions_in_progress/system-compactions_in_progress-jb-198-Data.db (166 bytes) for commitlog position ReplayPosition(segmentId=1395644152684, position=9577531)
> INFO [CompactionExecutor:1880] 2014-03-26 08:28:06,679 CompactionTask.java (line 115) Compacting [SSTableReader(path='/var/lib/cassandra/data2/OpsCenter/rollups60/OpsCenter-rollups60-jb-18-Data.db'), SSTableReader(path='/var/lib/cassandra/data1/OpsCenter/rollups60/OpsCenter-rollups60-jb-19-Data.db'), SSTableReader(path='/var/lib/cassandra/data2/OpsCenter/rollups60/OpsCenter-rollups60-jb-17-Data.db'), SSTableReader(path='/var/lib/cassandra/data1/OpsCenter/rollups60/OpsCenter-rollups60-jb-20-Data.db')]
> INFO [OptionalTasks:1] 2014-03-26 08:28:07,322 MeteredFlusher.java (line 63) flushing high-traffic column family CFS(Keyspace='system', ColumnFamily='batchlog') (estimated 215899526 bytes)
> INFO [OptionalTasks:1] 2014-03-26 08:28:07,322 ColumnFamilyStore.java (line 784) Enqueuing flush of Memtable-batchlog@74009414(39242953/215899526 serialized/live bytes, 239 ops)
> INFO [FlushWriter:422] 2014-03-26 08:28:07,323 Memtable.java (line 331) Writing Memtable-batchlog@74009414(39242953/215899526 serialized/live bytes, 239 ops)
> INFO [FlushWriter:422] 2014-03-26 08:28:07,370 Memtable.java (line 371) Completed flushing /var/lib/cassandra/data1/system/batchlog/system-batchlog-jb-1-Data.db (204617 bytes) for commitlog position ReplayPosition(segmentId=1395644152684, position=17705519)
> INFO [MemoryMeter:1] 2014-03-26 08:28:08,296 Memtable.java (line 449) CFS(Keyspace='system', ColumnFamily='batchlog') liveRatio is 3.2515027780484544 (just-counted was 1.0013929201634204). calculation took 0ms for 12 cells
> ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,667 StorageService.java (line 369) Stopping RPC server
> INFO [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,667 ThriftServer.java (line 141) Stop listening to thrift clients
> ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,698 StorageService.java (line 374) Stopping native transport
> INFO [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,765 Server.java (line 181) Stop listening for CQL clients
> ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,766 CommitLog.java (line 379) Failed to allocate new commit log segments. Commit disk failure policy is stop; terminating thread
> java.lang.AssertionError: attempted to delete non-existing file CommitLog-3-1395644152652.log
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)