You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kasun Weranga <ka...@wso2.com> on 2012/08/08 08:26:56 UTC
Cassandra commitlog directory size increase on every restart -
Cassandra 1.1.0
Hi all,
I am facing the above issue in Cassandra 1.1.0, it will add 134.2MB
commitlog file in every restart, but it never delete it. We can't control
the commitlog dir size even
by explicitly setting commitlog_total_space_in_mb in cassandra.yaml.
I set commitlog_total_space_in_mb as 512 in cassandra.yaml and did some
testing, now my commitlog directory size reached 1.1 GB.
Also I turned on debug log for CommitLog, This is what I get when server
starts.
[2012-08-08 11:25:30,860] INFO
{org.apache.cassandra.config.DatabaseDescriptor} - Global memtable
threshold is enabled at 151MB
[2012-08-08 11:25:31,427] INFO {org.apache.cassandra.service.CacheService}
- Initializing key cache with capacity of 12 MBs.
[2012-08-08 11:25:31,443] INFO {org.apache.cassandra.service.CacheService}
- Scheduling key cache save to each 14400 seconds (going to save all keys).
[2012-08-08 11:25:31,445] INFO {org.apache.cassandra.service.CacheService}
- Initializing row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
[2012-08-08 11:25:31,448] INFO {org.apache.cassandra.service.CacheService}
- Scheduling row cache save to each 0 seconds (going to save all keys).
[2012-08-08 11:25:31,677] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-6
(1644 bytes)
[2012-08-08 11:25:31,677] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-5
(5967 bytes)
[2012-08-08 11:25:31,784] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/Versions/system-Versions-hc-5
(247 bytes)
[2012-08-08 11:25:31,789] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/Versions/system-Versions-hc-6
(247 bytes)
[2012-08-08 11:25:31,887] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-1
(242 bytes)
[2012-08-08 11:25:31,897] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-2
(244 bytes)
[2012-08-08 11:25:31,935] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14
(80 bytes)
[2012-08-08 11:25:31,935] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13
(346 bytes)
[2012-08-08 11:25:31,935] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15
(163 bytes)
[2012-08-08 11:25:32,980] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-2
(312328 bytes)
[2012-08-08 11:25:32,980] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-3
(312992 bytes)
[2012-08-08 11:25:32,984] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-1
(647552 bytes)
[2012-08-08 11:25:33,183] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/EVENT_KS/bam_service_data_publisher/EVENT_KS-bam_service_data_publisher-hc-1
(249143 bytes)
[2012-08-08 11:25:33,266] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-1
(152 bytes)
[2012-08-08 11:25:33,266] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-2
(150 bytes)
[2012-08-08 11:25:33,362] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-2
(319 bytes)
[2012-08-08 11:25:33,363] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-1
(484 bytes)
[2012-08-08 11:25:33,412] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-1
(163 bytes)
[2012-08-08 11:25:33,422] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-2
(161 bytes)
[2012-08-08 11:25:33,447] INFO
{org.apache.cassandra.service.AbstractCassandraDaemon} - completed
pre-loading (6 keys) key cache.
[2012-08-08 11:25:34,227] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809612463142.log,
./repository/database/cassandra/commitlog/CommitLog-15809986125630.log,
./repository/database/cassandra/commitlog/CommitLog-15809986875007.log,
./repository/database/cassandra/commitlog/CommitLog-15809986654744.log,
./repository/database/cassandra/commitlog/CommitLog-15809986776035.log,
./repository/database/cassandra/commitlog/CommitLog-15809986520160.log,
./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
[2012-08-08 11:25:34,230] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
[2012-08-08 11:25:34,255] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
starting at 0
[2012-08-08 11:25:34,255] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,295] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.6275696c64: {ColumnFamily(Versions [version:false:5@1344363219522000
,])}
[2012-08-08 11:25:34,322] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 91
[2012-08-08 11:25:34,322] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.63716c: {ColumnFamily(Versions [version:false:5@1344363219527000,])}
[2012-08-08 11:25:34,322] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 180
[2012-08-08 11:25:34,323] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.746872696674: {ColumnFamily(Versions
[version:false:7@1344363219528000,])}
[2012-08-08 11:25:34,323] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 274
[2012-08-08 11:25:34,324] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.4c: {ColumnFamily(LocationInfo
[47656e65726174696f6e:false:4@1344363219567000,])}
[2012-08-08 11:25:34,325] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 363
[2012-08-08 11:25:34,325] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.426f6f747374726170: {ColumnFamily(LocationInfo
[42:false:1@1344363219785000,])}
[2012-08-08 11:25:34,325] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 448
[2012-08-08 11:25:34,325] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.4c: {ColumnFamily(LocationInfo [546f6b656e:false:16@1344363219786000
,])}
[2012-08-08 11:25:34,326] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 544
[2012-08-08 11:25:34,326] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 548
[2012-08-08 11:25:34,326] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
[2012-08-08 11:25:34,326] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
[2012-08-08 11:25:34,359] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
starting at 0
[2012-08-08 11:25:34,359] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,359] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,360] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
[2012-08-08 11:25:34,360] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
[2012-08-08 11:25:34,392] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
starting at 0
[2012-08-08 11:25:34,392] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,393] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,393] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
[2012-08-08 11:25:34,393] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
[2012-08-08 11:25:34,429] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
starting at 0
[2012-08-08 11:25:34,429] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,429] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,430] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
[2012-08-08 11:25:34,430] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
[2012-08-08 11:25:34,449] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
starting at 0
[2012-08-08 11:25:34,450] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,450] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,451] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
[2012-08-08 11:25:34,451] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
[2012-08-08 11:25:34,493] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
starting at 0
[2012-08-08 11:25:34,493] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,494] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,494] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
[2012-08-08 11:25:34,494] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
[2012-08-08 11:25:34,515] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
starting at 0
[2012-08-08 11:25:34,515] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,515] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,516] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
[2012-08-08 11:25:34,516] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Finished waiting on
mutations from recovery
[2012-08-08 11:25:34,519] INFO {org.apache.cassandra.db.ColumnFamilyStore}
- Enqueuing flush of Memtable-Versions@507924927(83/103 serialized/live
bytes, 3 ops)
[2012-08-08 11:25:34,520] INFO {org.apache.cassandra.db.Memtable} -
Writing Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
[2012-08-08 11:25:34,840] INFO {org.apache.cassandra.db.Memtable} -
Completed flushing
./repository/database/cassandra/data/system/Versions/system-Versions-hc-7-Data.db
(247 bytes)
[2012-08-08 11:25:34,858] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
segments for ReplayPosition(segmentId=2233031438605, position=0), column
family 7
[2012-08-08 11:25:34,859] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete commit
log
CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
dirty is ; hasNext: false*
[2012-08-08 11:25:34,859] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Log replay complete, 3
replayed mutations
[2012-08-08 11:25:34,879] INFO
{org.apache.cassandra.service.StorageService} - Cassandra version: 1.1.0
[2012-08-08 11:25:34,879] INFO
{org.apache.cassandra.service.StorageService} - Thrift API version: 19.30.0
[2012-08-08 11:25:34,881] INFO
{org.apache.cassandra.service.StorageService} - CQL supported versions:
2.0.0,3.0.0-beta1 (default: 2.0.0)
[2012-08-08 11:25:34,904] INFO
{org.apache.cassandra.service.StorageService} - Loading persisted ring
state
[2012-08-08 11:25:34,906] INFO
{org.apache.cassandra.service.StorageService} - Starting up server gossip
[2012-08-08 11:25:34,917] INFO {org.apache.cassandra.db.ColumnFamilyStore}
- Enqueuing flush of Memtable-LocationInfo@1885256814(29/36
serialized/live bytes, 1 ops)
[2012-08-08 11:25:34,918] INFO {org.apache.cassandra.db.Memtable} -
Writing Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1
ops)
[2012-08-08 11:25:35,182] INFO {org.apache.cassandra.db.Memtable} -
Completed flushing
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db
(80 bytes)
[2012-08-08 11:25:35,183] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
segments for ReplayPosition(segmentId=2233031438605, position=363), column
family 0
[2012-08-08 11:25:35,184] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - * Not safe to delete commit
log
CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
dirty is Versions (7), ; hasNext: false*
[2012-08-08 11:25:35,192] INFO
{org.apache.cassandra.db.compaction.CompactionTask} - Compacting
[SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13-Data.db'),
SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14-Data.db'),
SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db'),
SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15-Data.db')]
[2012-08-08 11:25:35,390] INFO {org.apache.cassandra.net.MessagingService}
- Starting Messaging Service on port 7000
[2012-08-08 11:25:35,399] INFO
{org.apache.cassandra.service.StorageService} - Using saved token
83957406514855090039949046883112789193
[2012-08-08 11:25:35,401] INFO {org.apache.cassandra.db.ColumnFamilyStore}
- Enqueuing flush of Memtable-LocationInfo@1345718067(53/66
serialized/live bytes, 2 ops)
[2012-08-08 11:25:35,401] INFO {org.apache.cassandra.db.Memtable} -
Writing Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2
ops)
[2012-08-08 11:25:35,959] INFO
{org.apache.cassandra.db.compaction.CompactionTask} - Compacted to
[./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-17-Data.db,].
669 to 346 (~51% of original) bytes for 3 keys at 0.000502MB/s. Time:
657ms.
[2012-08-08 11:25:36,001] INFO {org.apache.cassandra.db.Memtable} -
Completed flushing
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-18-Data.db
(163 bytes)
[2012-08-08 11:25:36,002] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
segments for ReplayPosition(segmentId=2233031438605, position=544), column
family 0
[2012-08-08 11:25:36,002] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete commit
log
CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
dirty is Versions (7), ; hasNext: false*
[2012-08-08 11:25:36,003] INFO
{org.apache.cassandra.service.StorageService} - Node
localhost/127.0.0.1state jump to normal
[2012-08-08 11:25:36,009] INFO
{org.apache.cassandra.service.StorageService} - Bootstrap/Replace/Move
completed! Now serving reads.
[2012-08-08 11:25:36,010] INFO {org.apache.cassandra.utils.Mx4jTool} -
Will not load MX4J, mx4j-tools.jar is not in the classpath
[2012-08-08 11:25:36,058] INFO
{org.apache.axis2.deployment.ClusterBuilder} - Clustering has been disabled
[2012-08-08 11:25:36,173] INFO
{org.apache.cassandra.thrift.CassandraDaemon} - Binding thrift service to
localhost/127.0.0.1:9160
[2012-08-08 11:25:36,254] INFO
{org.apache.cassandra.thrift.CassandraDaemon} - Using TFastFramedTransport
with a max frame size of 15728640 bytes.
[2012-08-08 11:25:36,257] INFO
{org.apache.cassandra.thrift.CassandraDaemon} - Using
synchronous/threadpool thrift server on localhost/127.0.0.1 : 9160
[2012-08-08 11:25:36,258] INFO
{org.wso2.carbon.cassandra.server.CassandraServerController} - Cassandra
Server Controller Thread was destroyed successfully
[2012-08-08 11:25:36,258] INFO
{org.apache.cassandra.thrift.CassandraDaemon} - Listening for thrift
clients...
What is the reason for keeping old commitlogs eventhough I have
set commitlog_total_space_in_mb in cassandra.yaml?
Why is this printing "Not safe to delete" message?
Appreciate your help on fixing this issue.
Thanks,
Kasun.
Re: Cassandra commitlog directory size increase on every restart -
Cassandra 1.1.0
Posted by Tyler Hobbs <ty...@datastax.com>.
There have been some commitlog-related fixes in later versions of 1.1, so
it's worth trying an upgrade. If that doesn't resolve the issue, open a
JIRA ticket with these details.
On Thu, Aug 9, 2012 at 9:15 AM, Kasun Weranga <ka...@wso2.com> wrote:
> Any idea on how to fix this?
>
> Thanks,
> Kasun
>
>
> On Wed, Aug 8, 2012 at 11:56 AM, Kasun Weranga <ka...@wso2.com> wrote:
>
>> Hi all,
>>
>> I am facing the above issue in Cassandra 1.1.0, it will add 134.2MB
>> commitlog file in every restart, but it never delete it. We can't control
>> the commitlog dir size even
>> by explicitly setting commitlog_total_space_in_mb in cassandra.yaml.
>> I set commitlog_total_space_in_mb as 512 in cassandra.yaml and did some
>> testing, now my commitlog directory size reached 1.1 GB.
>>
>> Also I turned on debug log for CommitLog, This is what I get when server
>> starts.
>>
>>
>> [2012-08-08 11:25:30,860] INFO
>> {org.apache.cassandra.config.DatabaseDescriptor} - Global memtable
>> threshold is enabled at 151MB
>> [2012-08-08 11:25:31,427] INFO
>> {org.apache.cassandra.service.CacheService} - Initializing key cache with
>> capacity of 12 MBs.
>> [2012-08-08 11:25:31,443] INFO
>> {org.apache.cassandra.service.CacheService} - Scheduling key cache save to
>> each 14400 seconds (going to save all keys).
>> [2012-08-08 11:25:31,445] INFO
>> {org.apache.cassandra.service.CacheService} - Initializing row cache with
>> capacity of 0 MBs and provider
>> org.apache.cassandra.cache.SerializingCacheProvider
>> [2012-08-08 11:25:31,448] INFO
>> {org.apache.cassandra.service.CacheService} - Scheduling row cache save to
>> each 0 seconds (going to save all keys).
>> [2012-08-08 11:25:31,677] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-6
>> (1644 bytes)
>> [2012-08-08 11:25:31,677] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-5
>> (5967 bytes)
>> [2012-08-08 11:25:31,784] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-5
>> (247 bytes)
>> [2012-08-08 11:25:31,789] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-6
>> (247 bytes)
>> [2012-08-08 11:25:31,887] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-1
>> (242 bytes)
>> [2012-08-08 11:25:31,897] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-2
>> (244 bytes)
>> [2012-08-08 11:25:31,935] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14
>> (80 bytes)
>> [2012-08-08 11:25:31,935] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13
>> (346 bytes)
>> [2012-08-08 11:25:31,935] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15
>> (163 bytes)
>> [2012-08-08 11:25:32,980] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-2
>> (312328 bytes)
>> [2012-08-08 11:25:32,980] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-3
>> (312992 bytes)
>> [2012-08-08 11:25:32,984] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-1
>> (647552 bytes)
>> [2012-08-08 11:25:33,183] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/EVENT_KS/bam_service_data_publisher/EVENT_KS-bam_service_data_publisher-hc-1
>> (249143 bytes)
>> [2012-08-08 11:25:33,266] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-1
>> (152 bytes)
>> [2012-08-08 11:25:33,266] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-2
>> (150 bytes)
>> [2012-08-08 11:25:33,362] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-2
>> (319 bytes)
>> [2012-08-08 11:25:33,363] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-1
>> (484 bytes)
>> [2012-08-08 11:25:33,412] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-1
>> (163 bytes)
>> [2012-08-08 11:25:33,422] INFO
>> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
>> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-2
>> (161 bytes)
>> [2012-08-08 11:25:33,447] INFO
>> {org.apache.cassandra.service.AbstractCassandraDaemon} - completed
>> pre-loading (6 keys) key cache.
>> [2012-08-08 11:25:34,227] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log,
>> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log,
>> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log,
>> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log,
>> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log,
>> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log,
>> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
>> [2012-08-08 11:25:34,230] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
>> [2012-08-08 11:25:34,255] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
>> starting at 0
>> [2012-08-08 11:25:34,255] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
>> [2012-08-08 11:25:34,295] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
>> system.6275696c64: {ColumnFamily(Versions [version:false:5@1344363219522000
>> ,])}
>> [2012-08-08 11:25:34,322] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 91
>> [2012-08-08 11:25:34,322] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
>> system.63716c: {ColumnFamily(Versions [version:false:5@1344363219527000
>> ,])}
>> [2012-08-08 11:25:34,322] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 180
>> [2012-08-08 11:25:34,323] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
>> system.746872696674: {ColumnFamily(Versions
>> [version:false:7@1344363219528000,])}
>> [2012-08-08 11:25:34,323] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 274
>> [2012-08-08 11:25:34,324] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
>> system.4c: {ColumnFamily(LocationInfo
>> [47656e65726174696f6e:false:4@1344363219567000,])}
>> [2012-08-08 11:25:34,325] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 363
>> [2012-08-08 11:25:34,325] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
>> system.426f6f747374726170: {ColumnFamily(LocationInfo
>> [42:false:1@1344363219785000,])}
>> [2012-08-08 11:25:34,325] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 448
>> [2012-08-08 11:25:34,325] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
>> system.4c: {ColumnFamily(LocationInfo [546f6b656e:false:16@1344363219786000
>> ,])}
>> [2012-08-08 11:25:34,326] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 544
>> [2012-08-08 11:25:34,326] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
>> marker at 548
>> [2012-08-08 11:25:34,326] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
>> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
>> [2012-08-08 11:25:34,326] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
>> [2012-08-08 11:25:34,359] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
>> starting at 0
>> [2012-08-08 11:25:34,359] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
>> [2012-08-08 11:25:34,359] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
>> marker at 4
>> [2012-08-08 11:25:34,360] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
>> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
>> [2012-08-08 11:25:34,360] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
>> [2012-08-08 11:25:34,392] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
>> starting at 0
>> [2012-08-08 11:25:34,392] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
>> [2012-08-08 11:25:34,393] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
>> marker at 4
>> [2012-08-08 11:25:34,393] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
>> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
>> [2012-08-08 11:25:34,393] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
>> [2012-08-08 11:25:34,429] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
>> starting at 0
>> [2012-08-08 11:25:34,429] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
>> [2012-08-08 11:25:34,429] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
>> marker at 4
>> [2012-08-08 11:25:34,430] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
>> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
>> [2012-08-08 11:25:34,430] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
>> [2012-08-08 11:25:34,449] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
>> starting at 0
>> [2012-08-08 11:25:34,450] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
>> [2012-08-08 11:25:34,450] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
>> marker at 4
>> [2012-08-08 11:25:34,451] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
>> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
>> [2012-08-08 11:25:34,451] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
>> [2012-08-08 11:25:34,493] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
>> starting at 0
>> [2012-08-08 11:25:34,493] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
>> [2012-08-08 11:25:34,494] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
>> marker at 4
>> [2012-08-08 11:25:34,494] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
>> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
>> [2012-08-08 11:25:34,494] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
>> [2012-08-08 11:25:34,515] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
>> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
>> starting at 0
>> [2012-08-08 11:25:34,515] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
>> [2012-08-08 11:25:34,515] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
>> marker at 4
>> [2012-08-08 11:25:34,516] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
>> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
>> [2012-08-08 11:25:34,516] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - Finished waiting on
>> mutations from recovery
>> [2012-08-08 11:25:34,519] INFO
>> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of
>> Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
>> [2012-08-08 11:25:34,520] INFO {org.apache.cassandra.db.Memtable} -
>> Writing Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
>> [2012-08-08 11:25:34,840] INFO {org.apache.cassandra.db.Memtable} -
>> Completed flushing
>> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-7-Data.db
>> (247 bytes)
>> [2012-08-08 11:25:34,858] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
>> segments for ReplayPosition(segmentId=2233031438605, position=0), column
>> family 7
>> [2012-08-08 11:25:34,859] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete
>> commit log
>> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
>> dirty is ; hasNext: false*
>> [2012-08-08 11:25:34,859] INFO
>> {org.apache.cassandra.db.commitlog.CommitLog} - Log replay complete, 3
>> replayed mutations
>> [2012-08-08 11:25:34,879] INFO
>> {org.apache.cassandra.service.StorageService} - Cassandra version: 1.1.0
>> [2012-08-08 11:25:34,879] INFO
>> {org.apache.cassandra.service.StorageService} - Thrift API version: 19.30.0
>> [2012-08-08 11:25:34,881] INFO
>> {org.apache.cassandra.service.StorageService} - CQL supported versions:
>> 2.0.0,3.0.0-beta1 (default: 2.0.0)
>> [2012-08-08 11:25:34,904] INFO
>> {org.apache.cassandra.service.StorageService} - Loading persisted ring
>> state
>> [2012-08-08 11:25:34,906] INFO
>> {org.apache.cassandra.service.StorageService} - Starting up server gossip
>> [2012-08-08 11:25:34,917] INFO
>> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of
>> Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1 ops)
>> [2012-08-08 11:25:34,918] INFO {org.apache.cassandra.db.Memtable} -
>> Writing Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1
>> ops)
>> [2012-08-08 11:25:35,182] INFO {org.apache.cassandra.db.Memtable} -
>> Completed flushing
>> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db
>> (80 bytes)
>> [2012-08-08 11:25:35,183] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
>> segments for ReplayPosition(segmentId=2233031438605, position=363), column
>> family 0
>> [2012-08-08 11:25:35,184] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - * Not safe to delete
>> commit log
>> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
>> dirty is Versions (7), ; hasNext: false*
>> [2012-08-08 11:25:35,192] INFO
>> {org.apache.cassandra.db.compaction.CompactionTask} - Compacting
>> [SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13-Data.db'),
>> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14-Data.db'),
>> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db'),
>> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15-Data.db')]
>> [2012-08-08 11:25:35,390] INFO
>> {org.apache.cassandra.net.MessagingService} - Starting Messaging Service
>> on port 7000
>> [2012-08-08 11:25:35,399] INFO
>> {org.apache.cassandra.service.StorageService} - Using saved token
>> 83957406514855090039949046883112789193
>> [2012-08-08 11:25:35,401] INFO
>> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of
>> Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2 ops)
>> [2012-08-08 11:25:35,401] INFO {org.apache.cassandra.db.Memtable} -
>> Writing Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2
>> ops)
>> [2012-08-08 11:25:35,959] INFO
>> {org.apache.cassandra.db.compaction.CompactionTask} - Compacted to
>> [./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-17-Data.db,].
>> 669 to 346 (~51% of original) bytes for 3 keys at 0.000502MB/s. Time:
>> 657ms.
>> [2012-08-08 11:25:36,001] INFO {org.apache.cassandra.db.Memtable} -
>> Completed flushing
>> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-18-Data.db
>> (163 bytes)
>> [2012-08-08 11:25:36,002] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
>> segments for ReplayPosition(segmentId=2233031438605, position=544), column
>> family 0
>> [2012-08-08 11:25:36,002] DEBUG
>> {org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete
>> commit log
>> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
>> dirty is Versions (7), ; hasNext: false*
>> [2012-08-08 11:25:36,003] INFO
>> {org.apache.cassandra.service.StorageService} - Node localhost/127.0.0.1state jump to normal
>> [2012-08-08 11:25:36,009] INFO
>> {org.apache.cassandra.service.StorageService} - Bootstrap/Replace/Move
>> completed! Now serving reads.
>> [2012-08-08 11:25:36,010] INFO {org.apache.cassandra.utils.Mx4jTool} -
>> Will not load MX4J, mx4j-tools.jar is not in the classpath
>> [2012-08-08 11:25:36,058] INFO
>> {org.apache.axis2.deployment.ClusterBuilder} - Clustering has been disabled
>> [2012-08-08 11:25:36,173] INFO
>> {org.apache.cassandra.thrift.CassandraDaemon} - Binding thrift service to
>> localhost/127.0.0.1:9160
>> [2012-08-08 11:25:36,254] INFO
>> {org.apache.cassandra.thrift.CassandraDaemon} - Using TFastFramedTransport
>> with a max frame size of 15728640 bytes.
>> [2012-08-08 11:25:36,257] INFO
>> {org.apache.cassandra.thrift.CassandraDaemon} - Using
>> synchronous/threadpool thrift server on localhost/127.0.0.1 : 9160
>> [2012-08-08 11:25:36,258] INFO
>> {org.wso2.carbon.cassandra.server.CassandraServerController} - Cassandra
>> Server Controller Thread was destroyed successfully
>> [2012-08-08 11:25:36,258] INFO
>> {org.apache.cassandra.thrift.CassandraDaemon} - Listening for thrift
>> clients...
>>
>>
>> What is the reason for keeping old commitlogs eventhough I have
>> set commitlog_total_space_in_mb in cassandra.yaml?
>> Why is this printing "Not safe to delete" message?
>>
>>
>> Appreciate your help on fixing this issue.
>>
>>
>> Thanks,
>> Kasun.
>>
>
>
>
>
--
Tyler Hobbs
DataStax <http://datastax.com/>
Re: Cassandra commitlog directory size increase on every restart -
Cassandra 1.1.0
Posted by Kasun Weranga <ka...@wso2.com>.
Any idea on how to fix this?
Thanks,
Kasun
On Wed, Aug 8, 2012 at 11:56 AM, Kasun Weranga <ka...@wso2.com> wrote:
> Hi all,
>
> I am facing the above issue in Cassandra 1.1.0, it will add 134.2MB
> commitlog file in every restart, but it never delete it. We can't control
> the commitlog dir size even
> by explicitly setting commitlog_total_space_in_mb in cassandra.yaml.
> I set commitlog_total_space_in_mb as 512 in cassandra.yaml and did some
> testing, now my commitlog directory size reached 1.1 GB.
>
> Also I turned on debug log for CommitLog, This is what I get when server
> starts.
>
>
> [2012-08-08 11:25:30,860] INFO
> {org.apache.cassandra.config.DatabaseDescriptor} - Global memtable
> threshold is enabled at 151MB
> [2012-08-08 11:25:31,427] INFO
> {org.apache.cassandra.service.CacheService} - Initializing key cache with
> capacity of 12 MBs.
> [2012-08-08 11:25:31,443] INFO
> {org.apache.cassandra.service.CacheService} - Scheduling key cache save to
> each 14400 seconds (going to save all keys).
> [2012-08-08 11:25:31,445] INFO
> {org.apache.cassandra.service.CacheService} - Initializing row cache with
> capacity of 0 MBs and provider
> org.apache.cassandra.cache.SerializingCacheProvider
> [2012-08-08 11:25:31,448] INFO
> {org.apache.cassandra.service.CacheService} - Scheduling row cache save to
> each 0 seconds (going to save all keys).
> [2012-08-08 11:25:31,677] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-6
> (1644 bytes)
> [2012-08-08 11:25:31,677] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-5
> (5967 bytes)
> [2012-08-08 11:25:31,784] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-5
> (247 bytes)
> [2012-08-08 11:25:31,789] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-6
> (247 bytes)
> [2012-08-08 11:25:31,887] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-1
> (242 bytes)
> [2012-08-08 11:25:31,897] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-2
> (244 bytes)
> [2012-08-08 11:25:31,935] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14
> (80 bytes)
> [2012-08-08 11:25:31,935] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13
> (346 bytes)
> [2012-08-08 11:25:31,935] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15
> (163 bytes)
> [2012-08-08 11:25:32,980] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-2
> (312328 bytes)
> [2012-08-08 11:25:32,980] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-3
> (312992 bytes)
> [2012-08-08 11:25:32,984] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-1
> (647552 bytes)
> [2012-08-08 11:25:33,183] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/EVENT_KS/bam_service_data_publisher/EVENT_KS-bam_service_data_publisher-hc-1
> (249143 bytes)
> [2012-08-08 11:25:33,266] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-1
> (152 bytes)
> [2012-08-08 11:25:33,266] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-2
> (150 bytes)
> [2012-08-08 11:25:33,362] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-2
> (319 bytes)
> [2012-08-08 11:25:33,363] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-1
> (484 bytes)
> [2012-08-08 11:25:33,412] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-1
> (163 bytes)
> [2012-08-08 11:25:33,422] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-2
> (161 bytes)
> [2012-08-08 11:25:33,447] INFO
> {org.apache.cassandra.service.AbstractCassandraDaemon} - completed
> pre-loading (6 keys) key cache.
> [2012-08-08 11:25:34,227] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> [2012-08-08 11:25:34,230] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
> [2012-08-08 11:25:34,255] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
> starting at 0
> [2012-08-08 11:25:34,255] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,295] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.6275696c64: {ColumnFamily(Versions [version:false:5@1344363219522000
> ,])}
> [2012-08-08 11:25:34,322] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 91
> [2012-08-08 11:25:34,322] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.63716c: {ColumnFamily(Versions [version:false:5@1344363219527000
> ,])}
> [2012-08-08 11:25:34,322] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 180
> [2012-08-08 11:25:34,323] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.746872696674: {ColumnFamily(Versions
> [version:false:7@1344363219528000,])}
> [2012-08-08 11:25:34,323] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 274
> [2012-08-08 11:25:34,324] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.4c: {ColumnFamily(LocationInfo
> [47656e65726174696f6e:false:4@1344363219567000,])}
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 363
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.426f6f747374726170: {ColumnFamily(LocationInfo
> [42:false:1@1344363219785000,])}
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 448
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.4c: {ColumnFamily(LocationInfo [546f6b656e:false:16@1344363219786000
> ,])}
> [2012-08-08 11:25:34,326] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 544
> [2012-08-08 11:25:34,326] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 548
> [2012-08-08 11:25:34,326] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
> [2012-08-08 11:25:34,326] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
> [2012-08-08 11:25:34,359] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
> starting at 0
> [2012-08-08 11:25:34,359] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,359] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,360] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
> [2012-08-08 11:25:34,360] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
> [2012-08-08 11:25:34,392] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
> starting at 0
> [2012-08-08 11:25:34,392] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,393] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,393] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
> [2012-08-08 11:25:34,393] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
> [2012-08-08 11:25:34,429] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
> starting at 0
> [2012-08-08 11:25:34,429] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,429] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,430] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
> [2012-08-08 11:25:34,430] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
> [2012-08-08 11:25:34,449] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
> starting at 0
> [2012-08-08 11:25:34,450] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,450] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,451] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
> [2012-08-08 11:25:34,451] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
> [2012-08-08 11:25:34,493] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
> starting at 0
> [2012-08-08 11:25:34,493] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,494] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,494] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
> [2012-08-08 11:25:34,494] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> [2012-08-08 11:25:34,515] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> starting at 0
> [2012-08-08 11:25:34,515] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,515] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,516] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> [2012-08-08 11:25:34,516] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished waiting on
> mutations from recovery
> [2012-08-08 11:25:34,519] INFO
> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of
> Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
> [2012-08-08 11:25:34,520] INFO {org.apache.cassandra.db.Memtable} -
> Writing Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
> [2012-08-08 11:25:34,840] INFO {org.apache.cassandra.db.Memtable} -
> Completed flushing
> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-7-Data.db
> (247 bytes)
> [2012-08-08 11:25:34,858] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
> segments for ReplayPosition(segmentId=2233031438605, position=0), column
> family 7
> [2012-08-08 11:25:34,859] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete
> commit log
> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
> dirty is ; hasNext: false*
> [2012-08-08 11:25:34,859] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Log replay complete, 3
> replayed mutations
> [2012-08-08 11:25:34,879] INFO
> {org.apache.cassandra.service.StorageService} - Cassandra version: 1.1.0
> [2012-08-08 11:25:34,879] INFO
> {org.apache.cassandra.service.StorageService} - Thrift API version: 19.30.0
> [2012-08-08 11:25:34,881] INFO
> {org.apache.cassandra.service.StorageService} - CQL supported versions:
> 2.0.0,3.0.0-beta1 (default: 2.0.0)
> [2012-08-08 11:25:34,904] INFO
> {org.apache.cassandra.service.StorageService} - Loading persisted ring
> state
> [2012-08-08 11:25:34,906] INFO
> {org.apache.cassandra.service.StorageService} - Starting up server gossip
> [2012-08-08 11:25:34,917] INFO
> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of
> Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1 ops)
> [2012-08-08 11:25:34,918] INFO {org.apache.cassandra.db.Memtable} -
> Writing Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1
> ops)
> [2012-08-08 11:25:35,182] INFO {org.apache.cassandra.db.Memtable} -
> Completed flushing
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db
> (80 bytes)
> [2012-08-08 11:25:35,183] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
> segments for ReplayPosition(segmentId=2233031438605, position=363), column
> family 0
> [2012-08-08 11:25:35,184] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - * Not safe to delete
> commit log
> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
> dirty is Versions (7), ; hasNext: false*
> [2012-08-08 11:25:35,192] INFO
> {org.apache.cassandra.db.compaction.CompactionTask} - Compacting
> [SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13-Data.db'),
> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14-Data.db'),
> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db'),
> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15-Data.db')]
> [2012-08-08 11:25:35,390] INFO
> {org.apache.cassandra.net.MessagingService} - Starting Messaging Service
> on port 7000
> [2012-08-08 11:25:35,399] INFO
> {org.apache.cassandra.service.StorageService} - Using saved token
> 83957406514855090039949046883112789193
> [2012-08-08 11:25:35,401] INFO
> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of
> Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2 ops)
> [2012-08-08 11:25:35,401] INFO {org.apache.cassandra.db.Memtable} -
> Writing Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2
> ops)
> [2012-08-08 11:25:35,959] INFO
> {org.apache.cassandra.db.compaction.CompactionTask} - Compacted to
> [./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-17-Data.db,].
> 669 to 346 (~51% of original) bytes for 3 keys at 0.000502MB/s. Time:
> 657ms.
> [2012-08-08 11:25:36,001] INFO {org.apache.cassandra.db.Memtable} -
> Completed flushing
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-18-Data.db
> (163 bytes)
> [2012-08-08 11:25:36,002] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
> segments for ReplayPosition(segmentId=2233031438605, position=544), column
> family 0
> [2012-08-08 11:25:36,002] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete
> commit log
> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
> dirty is Versions (7), ; hasNext: false*
> [2012-08-08 11:25:36,003] INFO
> {org.apache.cassandra.service.StorageService} - Node localhost/127.0.0.1state jump to normal
> [2012-08-08 11:25:36,009] INFO
> {org.apache.cassandra.service.StorageService} - Bootstrap/Replace/Move
> completed! Now serving reads.
> [2012-08-08 11:25:36,010] INFO {org.apache.cassandra.utils.Mx4jTool} -
> Will not load MX4J, mx4j-tools.jar is not in the classpath
> [2012-08-08 11:25:36,058] INFO
> {org.apache.axis2.deployment.ClusterBuilder} - Clustering has been disabled
> [2012-08-08 11:25:36,173] INFO
> {org.apache.cassandra.thrift.CassandraDaemon} - Binding thrift service to
> localhost/127.0.0.1:9160
> [2012-08-08 11:25:36,254] INFO
> {org.apache.cassandra.thrift.CassandraDaemon} - Using TFastFramedTransport
> with a max frame size of 15728640 bytes.
> [2012-08-08 11:25:36,257] INFO
> {org.apache.cassandra.thrift.CassandraDaemon} - Using
> synchronous/threadpool thrift server on localhost/127.0.0.1 : 9160
> [2012-08-08 11:25:36,258] INFO
> {org.wso2.carbon.cassandra.server.CassandraServerController} - Cassandra
> Server Controller Thread was destroyed successfully
> [2012-08-08 11:25:36,258] INFO
> {org.apache.cassandra.thrift.CassandraDaemon} - Listening for thrift
> clients...
>
>
> What is the reason for keeping old commitlogs eventhough I have
> set commitlog_total_space_in_mb in cassandra.yaml?
> Why is this printing "Not safe to delete" message?
>
>
> Appreciate your help on fixing this issue.
>
>
> Thanks,
> Kasun.
>