You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Bo Finnerup Madsen <bo...@gmail.com> on 2016/04/12 19:53:44 UTC
Debugging out of memory
Hi,
We have an application that reads data from a set of external sources and
loads them into our cassandra cluster. The load goes ok for some time
(~24h) and then some servers in the cluster starts flapping between being
down and up, and finally they go out of memory.
The cluster consists of 5 m4.xlarge machines with 16gb memory, cassandra
has an 8gb heap. All machines have a high load while data is being written,
with a load between 6 and 20.
I have tried sifting through the information available from nodetool, but I
am unable to find anything helping me determine what is causing the oom. I
am quite new to cassandra, so I might very well overlook the obvious. So
any pointers on how to proceed with identifying the problem will be much
appriciated :)
In the following I have included information from 10.61.70.110 when it was
flapping.
Status for ddp keyspace(only keyspace containing any real data):
-----------------------------------------------------------------------------
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.61.70.108 65.97 GB 256 59,1%
de79a554-9296-4575-8b79-2089f92069cd rack1
UN 10.61.70.110 58.95 GB 256 63,3%
310460f6-b7ce-45a7-be63-a7dd409f6b17 rack1
UN 10.61.70.72 58.17 GB 256 60,3%
44fd4f8e-18cd-4487-8174-3a22fb9ed24f rack1
UN 10.61.70.107 58.69 GB 256 58,5%
f8118fc2-e340-45db-a06e-a5842107d6c8 rack1
UN 10.61.70.64 68 GB 256 58,7%
84bee9fe-2adc-48aa-915c-f43d972f5a2f rack1
-----------------------------------------------------------------------------
Snippet from system.log:
-----------------------------------------------------------------------------
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,547 MessagingService.java:980
- MUTATION messages were dropped in last 5000 ms: 5776 for internal timeout
and 0 for cross node timeout
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,547 StatusLogger.java:52 -
Pool Name Active Pending Completed Blocked All
Time Blocked
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,551 StatusLogger.java:56 -
MutationStage 32 4881705 17826061870 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,552 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,552 StatusLogger.java:56 -
ReadStage 0 0 3266887 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,553 StatusLogger.java:56 -
RequestResponseStage 0 0 389429305 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,553 StatusLogger.java:56 -
ReadRepairStage 0 0 322804 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,554 StatusLogger.java:56 -
CounterMutationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,554 StatusLogger.java:56 -
MiscStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,554 StatusLogger.java:56 -
CompactionExecutor 4 54 31305 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,555 StatusLogger.java:56 -
MemtableReclaimMemory 0 0 3310 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,555 StatusLogger.java:56 -
PendingRangeCalculator 0 0 10 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,555 StatusLogger.java:56 -
GossipStage 0 0 338170 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,555 StatusLogger.java:56 -
SecondaryIndexManagement 0 0 0 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,556 StatusLogger.java:56 -
HintsDispatcher 1 4 6264 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,558 StatusLogger.java:56 -
MigrationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,559 StatusLogger.java:56 -
MemtablePostFlush 0 0 3451 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,560 StatusLogger.java:56 -
ValidationExecutor 0 0 0 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,561 StatusLogger.java:56 -
Sampler 0 0 0 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,563 StatusLogger.java:56 -
MemtableFlushWriter 0 0 3310 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,564 StatusLogger.java:56 -
InternalResponseStage 0 0 1873184 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,565 StatusLogger.java:56 -
AntiEntropyStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,565 StatusLogger.java:56 -
CacheCleanupExecutor 0 0 0 0
0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,566 StatusLogger.java:56 -
Native-Transport-Requests 3 2 212872837 0
1796
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,566 StatusLogger.java:66 -
CompactionManager 4 32
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,566 StatusLogger.java:78 -
MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,567 StatusLogger.java:88 -
Cache Type Size Capacity
KeysToSave
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,568 StatusLogger.java:90 -
KeyCache 88654200 104857600
all
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,568 StatusLogger.java:96 -
RowCache 0 0
all
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,568 StatusLogger.java:103 -
Table Memtable ops,data
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,569 StatusLogger.java:106 -
ddp.fingerprint_by_content_uuid_mv 99624,17726363
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,569 StatusLogger.java:106 -
ddp.sync 4747,1551
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,569 StatusLogger.java:106 -
ddp.log_portal 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,570 StatusLogger.java:106 -
ddp.meta_data 2083,3829424
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,570 StatusLogger.java:106 -
ddp.configuration 199,13915
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,570 StatusLogger.java:106 -
ddp.uuids_by_related_uuid 414431,17721698
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,570 StatusLogger.java:106 -
ddp.file_by_file_id 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,571 StatusLogger.java:106 -
ddp.fingerprint_by_content_type_mv 83007,3317002
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,571 StatusLogger.java:106 -
ddp.heartbeat 35855,13407
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,574 StatusLogger.java:106 -
ddp.concept 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,575 StatusLogger.java:106 -
ddp.classification_scheme 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,576 StatusLogger.java:106 -
ddp.rendering_relations_mv 9665,1201123
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,576 StatusLogger.java:106 -
ddp.file 41,19826
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,576 StatusLogger.java:106 -
ddp.rendering_relations 276049,13465909
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,576 StatusLogger.java:106 -
ddp.semantic_group 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,577 StatusLogger.java:106 -
ddp.rendering 23344,68861340
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,577 StatusLogger.java:106 -
ddp.thesauri 44,8
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,577 StatusLogger.java:106 -
ddp.file_download 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,577 StatusLogger.java:106 -
ddp.uuids_by_related_uuid_mv 172394,14524712
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,578 StatusLogger.java:106 -
ddp.fingerprint 124884,54955730
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,578 StatusLogger.java:106 -
ddp.ddp_status 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,578 StatusLogger.java:106 -
ddp.log_portal_mv 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,578 StatusLogger.java:106 -
system_distributed.parent_repair_history 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,579 StatusLogger.java:106 -
system_distributed.repair_history 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,579 StatusLogger.java:106 -
system.compaction_history 20,4595
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,580 StatusLogger.java:106 -
system.hints 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,580 StatusLogger.java:106 -
system.schema_aggregates 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,580 StatusLogger.java:106 -
system.IndexInfo 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,581 StatusLogger.java:106 -
system.schema_columnfamilies 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,581 StatusLogger.java:106 -
system.schema_triggers 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,582 StatusLogger.java:106 -
system.size_estimates 50400,764904
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,582 StatusLogger.java:106 -
system.schema_functions 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,582 StatusLogger.java:106 -
system.paxos 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,582 StatusLogger.java:106 -
system.views_builds_in_progress 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,583 StatusLogger.java:106 -
system.built_views 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,583 StatusLogger.java:106 -
system.peer_events 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,583 StatusLogger.java:106 -
system.range_xfers 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,583 StatusLogger.java:106 -
system.peers 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,584 StatusLogger.java:106 -
system.batches 188441,33975764
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,584 StatusLogger.java:106 -
system.schema_keyspaces 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,585 StatusLogger.java:106 -
system.schema_usertypes 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,585 StatusLogger.java:106 -
system.local 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,585 StatusLogger.java:106 -
system.sstable_activity 1376,24807
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,587 StatusLogger.java:106 -
system.available_ranges 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,587 StatusLogger.java:106 -
system.batchlog 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,588 StatusLogger.java:106 -
system.schema_columns 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,589 StatusLogger.java:106 -
system_schema.columns 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,589 StatusLogger.java:106 -
system_schema.types 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,590 StatusLogger.java:106 -
system_schema.indexes 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,590 StatusLogger.java:106 -
system_schema.keyspaces 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,590 StatusLogger.java:106 -
system_schema.dropped_columns 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,592 StatusLogger.java:106 -
system_schema.aggregates 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,592 StatusLogger.java:106 -
system_schema.triggers 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,592 StatusLogger.java:106 -
system_schema.tables 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,592 StatusLogger.java:106 -
system_schema.views 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,593 StatusLogger.java:106 -
system_schema.functions 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,593 StatusLogger.java:106 -
system_auth.roles 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,593 StatusLogger.java:106 -
system_auth.role_members 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,593 StatusLogger.java:106 -
system_auth.resource_role_permissons_index 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,594 StatusLogger.java:106 -
system_auth.role_permissions 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,594 StatusLogger.java:106 -
system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2016-04-12 17:35:35,594 StatusLogger.java:106 -
system_traces.events 0,0
-----------------------------------------------------------------------------
compactionstats:
-----------------------------------------------------------------------------
pending tasks: 32
id compaction type keyspace
table completed total unit progress
3bf89310-00d1-11e6-a5a3-f125ce747d55 Compaction ddp
rendering_relations 1,09 GB 5,43 GB bytes 20,10%
9c5bd6b1-00d4-11e6-a5a3-f125ce747d55 Compaction ddp
meta_data 243,64 MB 338,56 MB bytes 71,96%
50308cb0-00d2-11e6-a5a3-f125ce747d55 Compaction ddp
uuids_by_related_uuid 1,37 GB 2,17 GB bytes 63,11%
e8965d90-00c3-11e6-a5a3-f125ce747d55 Compaction ddp
rendering 19,02 GB 24,09 GB bytes 78,96%
-----------------------------------------------------------------------------
Thank you in advance :)
Yours sincerely,
Bo Madsen
Re: Debugging out of memory
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi,
This one is old, do you still need help there? Sorry we missed it.
1. What Cassandra version do you use?
2. What does "nodetool tpstats" show you. Any dropped or pending message?
3. Is your error a full heap memory issue or native one?
4. What configurations did you change from default and you think might
be related to this (memtable size, GC config, bloom filters...)?
the cluster starts flapping between being down and up
Using AWS, even more with small instances make sure to use
phi_convict_threshold set to about 12, just in case. This will prevent node
from flapping that much.
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-04-12 19:53 GMT+02:00 Bo Finnerup Madsen <bo...@gmail.com>:
> Hi,
>
> We have an application that reads data from a set of external sources and
> loads them into our cassandra cluster. The load goes ok for some time
> (~24h) and then some servers in the cluster starts flapping between being
> down and up, and finally they go out of memory.
> The cluster consists of 5 m4.xlarge machines with 16gb memory, cassandra
> has an 8gb heap. All machines have a high load while data is being written,
> with a load between 6 and 20.
>
> I have tried sifting through the information available from nodetool, but
> I am unable to find anything helping me determine what is causing the oom.
> I am quite new to cassandra, so I might very well overlook the obvious. So
> any pointers on how to proceed with identifying the problem will be much
> appriciated :)
>
> In the following I have included information from 10.61.70.110 when it was
> flapping.
>
> Status for ddp keyspace(only keyspace containing any real data):
>
> -----------------------------------------------------------------------------
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.61.70.108 65.97 GB 256 59,1%
> de79a554-9296-4575-8b79-2089f92069cd rack1
> UN 10.61.70.110 58.95 GB 256 63,3%
> 310460f6-b7ce-45a7-be63-a7dd409f6b17 rack1
> UN 10.61.70.72 58.17 GB 256 60,3%
> 44fd4f8e-18cd-4487-8174-3a22fb9ed24f rack1
> UN 10.61.70.107 58.69 GB 256 58,5%
> f8118fc2-e340-45db-a06e-a5842107d6c8 rack1
> UN 10.61.70.64 68 GB 256 58,7%
> 84bee9fe-2adc-48aa-915c-f43d972f5a2f rack1
>
> -----------------------------------------------------------------------------
>
>
> Snippet from system.log:
>
> -----------------------------------------------------------------------------
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,547 MessagingService.java:980
> - MUTATION messages were dropped in last 5000 ms: 5776 for internal timeout
> and 0 for cross node timeout
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,547 StatusLogger.java:52 -
> Pool Name Active Pending Completed Blocked All
> Time Blocked
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,551 StatusLogger.java:56 -
> MutationStage 32 4881705 17826061870 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,552 StatusLogger.java:56 -
> ViewMutationStage 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,552 StatusLogger.java:56 -
> ReadStage 0 0 3266887 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,553 StatusLogger.java:56 -
> RequestResponseStage 0 0 389429305 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,553 StatusLogger.java:56 -
> ReadRepairStage 0 0 322804 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,554 StatusLogger.java:56 -
> CounterMutationStage 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,554 StatusLogger.java:56 -
> MiscStage 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,554 StatusLogger.java:56 -
> CompactionExecutor 4 54 31305 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,555 StatusLogger.java:56 -
> MemtableReclaimMemory 0 0 3310 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,555 StatusLogger.java:56 -
> PendingRangeCalculator 0 0 10 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,555 StatusLogger.java:56 -
> GossipStage 0 0 338170 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,555 StatusLogger.java:56 -
> SecondaryIndexManagement 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,556 StatusLogger.java:56 -
> HintsDispatcher 1 4 6264 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,558 StatusLogger.java:56 -
> MigrationStage 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,559 StatusLogger.java:56 -
> MemtablePostFlush 0 0 3451 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,560 StatusLogger.java:56 -
> ValidationExecutor 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,561 StatusLogger.java:56 -
> Sampler 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,563 StatusLogger.java:56 -
> MemtableFlushWriter 0 0 3310 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,564 StatusLogger.java:56 -
> InternalResponseStage 0 0 1873184 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,565 StatusLogger.java:56 -
> AntiEntropyStage 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,565 StatusLogger.java:56 -
> CacheCleanupExecutor 0 0 0 0
> 0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,566 StatusLogger.java:56 -
> Native-Transport-Requests 3 2 212872837 0
> 1796
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,566 StatusLogger.java:66 -
> CompactionManager 4 32
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,566 StatusLogger.java:78 -
> MessagingService n/a 0/0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,567 StatusLogger.java:88 -
> Cache Type Size Capacity
> KeysToSave
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,568 StatusLogger.java:90 -
> KeyCache 88654200 104857600
> all
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,568 StatusLogger.java:96 -
> RowCache 0 0
> all
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,568 StatusLogger.java:103 -
> Table Memtable ops,data
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,569 StatusLogger.java:106 -
> ddp.fingerprint_by_content_uuid_mv 99624,17726363
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,569 StatusLogger.java:106 -
> ddp.sync 4747,1551
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,569 StatusLogger.java:106 -
> ddp.log_portal 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,570 StatusLogger.java:106 -
> ddp.meta_data 2083,3829424
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,570 StatusLogger.java:106 -
> ddp.configuration 199,13915
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,570 StatusLogger.java:106 -
> ddp.uuids_by_related_uuid 414431,17721698
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,570 StatusLogger.java:106 -
> ddp.file_by_file_id 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,571 StatusLogger.java:106 -
> ddp.fingerprint_by_content_type_mv 83007,3317002
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,571 StatusLogger.java:106 -
> ddp.heartbeat 35855,13407
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,574 StatusLogger.java:106 -
> ddp.concept 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,575 StatusLogger.java:106 -
> ddp.classification_scheme 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,576 StatusLogger.java:106 -
> ddp.rendering_relations_mv 9665,1201123
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,576 StatusLogger.java:106 -
> ddp.file 41,19826
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,576 StatusLogger.java:106 -
> ddp.rendering_relations 276049,13465909
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,576 StatusLogger.java:106 -
> ddp.semantic_group 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,577 StatusLogger.java:106 -
> ddp.rendering 23344,68861340
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,577 StatusLogger.java:106 -
> ddp.thesauri 44,8
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,577 StatusLogger.java:106 -
> ddp.file_download 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,577 StatusLogger.java:106 -
> ddp.uuids_by_related_uuid_mv 172394,14524712
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,578 StatusLogger.java:106 -
> ddp.fingerprint 124884,54955730
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,578 StatusLogger.java:106 -
> ddp.ddp_status 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,578 StatusLogger.java:106 -
> ddp.log_portal_mv 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,578 StatusLogger.java:106 -
> system_distributed.parent_repair_history 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,579 StatusLogger.java:106 -
> system_distributed.repair_history 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,579 StatusLogger.java:106 -
> system.compaction_history 20,4595
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,580 StatusLogger.java:106 -
> system.hints 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,580 StatusLogger.java:106 -
> system.schema_aggregates 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,580 StatusLogger.java:106 -
> system.IndexInfo 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,581 StatusLogger.java:106 -
> system.schema_columnfamilies 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,581 StatusLogger.java:106 -
> system.schema_triggers 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,582 StatusLogger.java:106 -
> system.size_estimates 50400,764904
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,582 StatusLogger.java:106 -
> system.schema_functions 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,582 StatusLogger.java:106 -
> system.paxos 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,582 StatusLogger.java:106 -
> system.views_builds_in_progress 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,583 StatusLogger.java:106 -
> system.built_views 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,583 StatusLogger.java:106 -
> system.peer_events 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,583 StatusLogger.java:106 -
> system.range_xfers 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,583 StatusLogger.java:106 -
> system.peers 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,584 StatusLogger.java:106 -
> system.batches 188441,33975764
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,584 StatusLogger.java:106 -
> system.schema_keyspaces 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,585 StatusLogger.java:106 -
> system.schema_usertypes 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,585 StatusLogger.java:106 -
> system.local 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,585 StatusLogger.java:106 -
> system.sstable_activity 1376,24807
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,587 StatusLogger.java:106 -
> system.available_ranges 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,587 StatusLogger.java:106 -
> system.batchlog 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,588 StatusLogger.java:106 -
> system.schema_columns 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,589 StatusLogger.java:106 -
> system_schema.columns 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,589 StatusLogger.java:106 -
> system_schema.types 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,590 StatusLogger.java:106 -
> system_schema.indexes 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,590 StatusLogger.java:106 -
> system_schema.keyspaces 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,590 StatusLogger.java:106 -
> system_schema.dropped_columns 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,592 StatusLogger.java:106 -
> system_schema.aggregates 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,592 StatusLogger.java:106 -
> system_schema.triggers 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,592 StatusLogger.java:106 -
> system_schema.tables 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,592 StatusLogger.java:106 -
> system_schema.views 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,593 StatusLogger.java:106 -
> system_schema.functions 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,593 StatusLogger.java:106 -
> system_auth.roles 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,593 StatusLogger.java:106 -
> system_auth.role_members 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,593 StatusLogger.java:106 -
> system_auth.resource_role_permissons_index 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,594 StatusLogger.java:106 -
> system_auth.role_permissions 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,594 StatusLogger.java:106 -
> system_traces.sessions 0,0
> INFO [ScheduledTasks:1] 2016-04-12 17:35:35,594 StatusLogger.java:106 -
> system_traces.events 0,0
>
> -----------------------------------------------------------------------------
>
> compactionstats:
>
> -----------------------------------------------------------------------------
> pending tasks: 32
> id compaction type keyspace
> table completed total unit progress
> 3bf89310-00d1-11e6-a5a3-f125ce747d55 Compaction ddp
> rendering_relations 1,09 GB 5,43 GB bytes 20,10%
> 9c5bd6b1-00d4-11e6-a5a3-f125ce747d55 Compaction ddp
> meta_data 243,64 MB 338,56 MB bytes 71,96%
> 50308cb0-00d2-11e6-a5a3-f125ce747d55 Compaction ddp
> uuids_by_related_uuid 1,37 GB 2,17 GB bytes 63,11%
> e8965d90-00c3-11e6-a5a3-f125ce747d55 Compaction ddp
> rendering 19,02 GB 24,09 GB bytes 78,96%
>
> -----------------------------------------------------------------------------
>
> Thank you in advance :)
>
> Yours sincerely,
> Bo Madsen
>