You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Sonny Heer <so...@gmail.com> on 2010/04/21 19:02:37 UTC
Import using cassandra 0.6.1
Currently running on a single node with intensive write operations.
After running for a while...
Client starts outputting:
TimedOutException()
at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232)
at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670)
at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643)
Cassandra starts outputting:
INFO 08:08:49,864 Cassandra starting up...
INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving
220554976 used; max is 1190723584
INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving
230245792 used; max is 1190723584
INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving
250127792 used; max is 1190723584
INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving
261657720 used; max is 1190723584
INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving
273764040 used; max is 1190723584
INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving
282912656 used; max is 1190723584
INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving
294861824 used; max is 1190723584
INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving
304045640 used; max is 1190723584
INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving
312964344 used; max is 1190723584
INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving
324961320 used; max is 1190723584
INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving
345874144 used; max is 1190723584
INFO 08:20:27,357 Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1271866827357.log
INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving
366587984 used; max is 1190723584
INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving
384059832 used; max is 1190723584
INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving
383631432 used; max is 1190723584
INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving
399697888 used; max is 1190723584
INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving
415428952 used; max is 1190723584
INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving
433542656 used; max is 1190723584
INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving
452030560 used; max is 1190723584
INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving
464013992 used; max is 1190723584
INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving
471978840 used; max is 1190723584
INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving
475912120 used; max is 1190723584
INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving
483742344 used; max is 1190723584
INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving
491761672 used; max is 1190723584
INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving
507442800 used; max is 1190723584
INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving
523028304 used; max is 1190723584
INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving
530865568 used; max is 1190723584
INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving
542690216 used; max is 1190723584
INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving
550728232 used; max is 1190723584
INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving
573981584 used; max is 1190723584
INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving
563891472 used; max is 1190723584
INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving
575756040 used; max is 1190723584
INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving
583631432 used; max is 1190723584
INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving
595405816 used; max is 1190723584
INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving
603389432 used; max is 1190723584
INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving
615150584 used; max is 1190723584
INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving
623125104 used; max is 1190723584
INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving
634661008 used; max is 1190723584
INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving
646272840 used; max is 1190723584
INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving
653978688 used; max is 1190723584
INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving
665462760 used; max is 1190723584
INFO 08:26:44,895 Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1271867204895.log
INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving
673423168 used; max is 1190723584
INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving
685277616 used; max is 1190723584
INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving
692803112 used; max is 1190723584
INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving
700213488 used; max is 1190723584
INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving
708002984 used; max is 1190723584
INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving
719547672 used; max is 1190723584
INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving
727261696 used; max is 1190723584
INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving
738760480 used; max is 1190723584
INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving
746376392 used; max is 1190723584
INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving
754102376 used; max is 1190723584
INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving
757926896 used; max is 1190723584
INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving
763964048 used; max is 1190723584
INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving
771496792 used; max is 1190723584
INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving
779088496 used; max is 1190723584
INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving
786858536 used; max is 1190723584
INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving
794395528 used; max is 1190723584
INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving
805862000 used; max is 1190723584
INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving
813708392 used; max is 1190723584
INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving
821245568 used; max is 1190723584
INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving
828614768 used; max is 1190723584
INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving
835962544 used; max is 1190723584
INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving
844183384 used; max is 1190723584
INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving
847924056 used; max is 1190723584
INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving
841384040 used; max is 1190723584
INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving
839554448 used; max is 1190723584
INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving
850851600 used; max is 1190723584
INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving
869886032 used; max is 1190723584
INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving
881346312 used; max is 1190723584
INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving
892677840 used; max is 1190723584
INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving
911084096 used; max is 1190723584
INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving
922370800 used; max is 1190723584
INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving
929916416 used; max is 1190723584
INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving
941201152 used; max is 1190723584
INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving
948631600 used; max is 1190723584
INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving
956424384 used; max is 1190723584
INFO 08:33:14,864 Creating new commitlog segment
/var/lib/cassandra/commitlog/CommitLog-1271867594864.log
INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving
972976288 used; max is 1190723584
INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving
981829584 used; max is 1190723584
INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving
995907192 used; max is 1190723584
INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving
1011914464 used; max is 1190723584
INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408
reclaimed leaving 1015729984 used; max is 1190723584
INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912
reclaimed leaving 1019176200 used; max is 1190723584
INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088
reclaimed leaving 1022673936 used; max is 1190723584
INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904
reclaimed leaving 1025979696 used; max is 1190723584
INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072
reclaimed leaving 1028992352 used; max is 1190723584
INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440
reclaimed leaving 1031961224 used; max is 1190723584
INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568
reclaimed leaving 1034536576 used; max is 1190723584
INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880
reclaimed leaving 1037266456 used; max is 1190723584
INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432
reclaimed leaving 1039709528 used; max is 1190723584
INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800
reclaimed leaving 1042230248 used; max is 1190723584
INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936
reclaimed leaving 1044527600 used; max is 1190723584
INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008
reclaimed leaving 1046621520 used; max is 1190723584
INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336
reclaimed leaving 1048677040 used; max is 1190723584
INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368
reclaimed leaving 1050563688 used; max is 1190723584
INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112
reclaimed leaving 1052423824 used; max is 1190723584
INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048
reclaimed leaving 1054138992 used; max is 1190723584
INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560
reclaimed leaving 1055761280 used; max is 1190723584
INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728
reclaimed leaving 1057201248 used; max is 1190723584
INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792
reclaimed leaving 1058496000 used; max is 1190723584
INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136
reclaimed leaving 1060011424 used; max is 1190723584
INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792
reclaimed leaving 1061388688 used; max is 1190723584
INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096
reclaimed leaving 1062658344 used; max is 1190723584
INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488
reclaimed leaving 1063834408 used; max is 1190723584
INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232
reclaimed leaving 1064800992 used; max is 1190723584
INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288
reclaimed leaving 1065798256 used; max is 1190723584
INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680
reclaimed leaving 1066867192 used; max is 1190723584
INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304
reclaimed leaving 1067925416 used; max is 1190723584
INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632
reclaimed leaving 1068886912 used; max is 1190723584
INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184
reclaimed leaving 1069705264 used; max is 1190723584
INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544
reclaimed leaving 1070568224 used; max is 1190723584
INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752
reclaimed leaving 1071366040 used; max is 1190723584
INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104
reclaimed leaving 1072168400 used; max is 1190723584
INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264
reclaimed leaving 1072743152 used; max is 1190723584
INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160
reclaimed leaving 1073334720 used; max is 1190723584
INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152
reclaimed leaving 1073930576 used; max is 1190723584
INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416
reclaimed leaving 1074548160 used; max is 1190723584
INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544
reclaimed leaving 1075122368 used; max is 1190723584
INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272
reclaimed leaving 1075699904 used; max is 1190723584
INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552
reclaimed leaving 1076227648 used; max is 1190723584
INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976
reclaimed leaving 1076747648 used; max is 1190723584
INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584
reclaimed leaving 1077152296 used; max is 1190723584
INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168
reclaimed leaving 1077567128 used; max is 1190723584
INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704
reclaimed leaving 1077987824 used; max is 1190723584
INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848
reclaimed leaving 1078393448 used; max is 1190723584
INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136
reclaimed leaving 1078708784 used; max is 1190723584
INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344
reclaimed leaving 1079074464 used; max is 1190723584
INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992
reclaimed leaving 1079409016 used; max is 1190723584
INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952
reclaimed leaving 1079749672 used; max is 1190723584
INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768
reclaimed leaving 1080014112 used; max is 1190723584
INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216
reclaimed leaving 1080278992 used; max is 1190723584
INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912
reclaimed leaving 1080574456 used; max is 1190723584
INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376
reclaimed leaving 1080884096 used; max is 1190723584
INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512
reclaimed leaving 1081076888 used; max is 1190723584
INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168
reclaimed leaving 1081313872 used; max is 1190723584
INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168
reclaimed leaving 1081528880 used; max is 1190723584
INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152
reclaimed leaving 1081704648 used; max is 1190723584
INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216
reclaimed leaving 1081889512 used; max is 1190723584
INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392
reclaimed leaving 1082072568 used; max is 1190723584
INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376
reclaimed leaving 1082270680 used; max is 1190723584
INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920
reclaimed leaving 1082522872 used; max is 1190723584
INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392
reclaimed leaving 1082632368 used; max is 1190723584
INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560
reclaimed leaving 1082794472 used; max is 1190723584
INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024
reclaimed leaving 1082963008 used; max is 1190723584
INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504
reclaimed leaving 1083117048 used; max is 1190723584
INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192
reclaimed leaving 1083213304 used; max is 1190723584
INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392
reclaimed leaving 1083332280 used; max is 1190723584
INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808
reclaimed leaving 1083448744 used; max is 1190723584
INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864
reclaimed leaving 1083601864 used; max is 1190723584
INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520
reclaimed leaving 1083680168 used; max is 1190723584
INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872
reclaimed leaving 1083814848 used; max is 1190723584
INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648
reclaimed leaving 1083871152 used; max is 1190723584
INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720
reclaimed leaving 1083984728 used; max is 1190723584
INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056
reclaimed leaving 1084063504 used; max is 1190723584
Re: Import using cassandra 0.6.1
Posted by Sonny Heer <so...@gmail.com>.
Gotcha. No i don't see anything particularly interesting in the log.
Do i need to turn on higher logging in log4j?
here it is after i killed the client:
INFO [main] 2010-04-21 14:25:52,166 DatabaseDescriptor.java (line
229) Auto DiskAccessMode determined to be standard
INFO [main] 2010-04-21 14:25:52,416 SystemTable.java (line 139) Saved
Token not found. Using LXzz63jDw6A1wZGz
INFO [main] 2010-04-21 14:25:52,416 SystemTable.java (line 145) Saved
ClusterName not found. Using NGram Cluster
INFO [main] 2010-04-21 14:25:52,416 CommitLogSegment.java (line 50)
Creating new commitlog segment
/var/lib/cassandra/commitlog\CommitLog-1271885152416.log
INFO [main] 2010-04-21 14:25:52,448 StorageService.java (line 317)
Starting up server gossip
INFO [main] 2010-04-21 14:25:52,495 CassandraDaemon.java (line 108)
Binding thrift service to localhost/127.0.0.1:9160
INFO [main] 2010-04-21 14:25:52,495 CassandraDaemon.java (line 148)
Cassandra starting up...
INFO [COMMIT-LOG-WRITER] 2010-04-21 14:30:44,248
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog\CommitLog-1271885444248.log
INFO [COMMIT-LOG-WRITER] 2010-04-21 14:34:37,267
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog\CommitLog-1271885677267.log
INFO [COMMIT-LOG-WRITER] 2010-04-21 14:39:08,723
CommitLogSegment.java (line 50) Creating new commitlog segment
/var/lib/cassandra/commitlog\CommitLog-1271885948723.log
INFO [GC inspection] 2010-04-21 14:41:48,757 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 11665 ms, 34488656 reclaimed leaving
1051534024 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:42:00,913 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 9219 ms, 31272728 reclaimed leaving
1053427416 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:42:13,054 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 9498 ms, 31116952 reclaimed leaving
1055163128 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:42:25,195 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 10318 ms, 29605880 reclaimed leaving
1056674184 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:42:37,289 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 10051 ms, 28492128 reclaimed leaving
1057795048 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:42:49,398 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 9904 ms, 27056840 reclaimed leaving
1059227696 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:43:01,336 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 9865 ms, 25652224 reclaimed leaving
1060635120 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:43:12,977 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 11515 ms, 25644312 reclaimed leaving
1060642984 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:43:23,758 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 10754 ms, 25640552 reclaimed leaving
1060645864 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:43:35,446 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 11602 ms, 25642952 reclaimed leaving
1060647384 used; max is 1174208512
INFO [GC inspection] 2010-04-21 14:43:47,790 GCInspector.java (line
110) GC for ConcurrentMarkSweep: 12313 ms, 25638616 reclaimed leaving
1060647272 used; max is 1174208512
On Wed, Apr 21, 2010 at 3:16 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Wed, Apr 21, 2010 at 5:05 PM, Sonny Heer <so...@gmail.com> wrote:
>> What does OOM stand for?
>
> out of memory
>
>> for a given insert the size is small (meaning the a single insert
>> operation only has about a sentence of data) although as the insert
>> process continues, the columns under a given row key could potentially
>> grow to be large. Is that what you mean?
>
> yes.
>
> look in your log for warnings about row size during compaction.
>
Re: Import using cassandra 0.6.1
Posted by Jonathan Ellis <jb...@gmail.com>.
On Wed, Apr 21, 2010 at 5:05 PM, Sonny Heer <so...@gmail.com> wrote:
> What does OOM stand for?
out of memory
> for a given insert the size is small (meaning the a single insert
> operation only has about a sentence of data) although as the insert
> process continues, the columns under a given row key could potentially
> grow to be large. Is that what you mean?
yes.
look in your log for warnings about row size during compaction.
Re: Import using cassandra 0.6.1
Posted by Sonny Heer <so...@gmail.com>.
what i mean by as data is processed is that the column size will grow
in cassandra, but my client isn't ever writing large column size under
a given row...
Any idea whats going on here?
On Wed, Apr 21, 2010 at 3:05 PM, Sonny Heer <so...@gmail.com> wrote:
> What does OOM stand for?
>
> for a given insert the size is small (meaning the a single insert
> operation only has about a sentence of data) although as the insert
> process continues, the columns under a given row key could potentially
> grow to be large. Is that what you mean?
>
> An operation entails:
> Read
> Insert row (IE: rowkey: FOO columnName: BLAH value: 10)
> Delete a row
> Insert different row (IE: rowkey: FOO columnName: 0000010|Blah value: 10)
>
> millions of these operations are performed in sequence as files are
> read from a directory source.
>
> On Wed, Apr 21, 2010 at 2:37 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> then that's not the problem.
>>
>> are you writing large rows that OOM during compaction?
>>
>> On Wed, Apr 21, 2010 at 4:34 PM, Sonny Heer <so...@gmail.com> wrote:
>>> They are showing up as completed? Is this correct:
>>>
>>>
>>> Pool Name Active Pending Completed
>>> STREAM-STAGE 0 0 0
>>> RESPONSE-STAGE 0 0 0
>>> ROW-READ-STAGE 0 0 517446
>>> LB-OPERATIONS 0 0 0
>>> MESSAGE-DESERIALIZER-POOL 0 0 0
>>> GMFD 0 0 0
>>> LB-TARGET 0 0 0
>>> CONSISTENCY-MANAGER 0 0 0
>>> ROW-MUTATION-STAGE 0 0 1353622
>>> MESSAGE-STREAMING-POOL 0 0 0
>>> LOAD-BALANCER-STAGE 0 0 0
>>> FLUSH-SORTER-POOL 0 0 0
>>> MEMTABLE-POST-FLUSHER 0 0 0
>>> FLUSH-WRITER-POOL 0 0 0
>>> AE-SERVICE-STAGE 0 0 0
>>>
>>>
>>> On Wed, Apr 21, 2010 at 2:09 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>> you need to figure out where the memory is going. check tpstats, if
>>>> the pending ops are large somewhere that means you're just generating
>>>> insert ops faster than it can handle.
>>>>
>>>> On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer <so...@gmail.com> wrote:
>>>>> note: I'm using the Thrift API to insert. The commitLog directory
>>>>> continues to grow. The heap size continues to grow as well.
>>>>>
>>>>> I decreased MemtableSizeInMB size, but noticed no changes. Any idea
>>>>> what is causing this, and/or what property i need to tweek to
>>>>> alleviate this? What is the "insert threshold"?
>>>>>
>>>>> I moved to a more powerful node as well, it still ended up failing
>>>>> just after a longer period.
>>>>>
>>>>> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>>>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
>>>>>>
>>>>>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <so...@gmail.com> wrote:
>>>>>>> Currently running on a single node with intensive write operations.
>>>>>>>
>>>>>>>
>>>>>>> After running for a while...
>>>>>>>
>>>>>>> Client starts outputting:
>>>>>>>
>>>>>>> TimedOutException()
>>>>>>> at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232)
>>>>>>> at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670)
>>>>>>> at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643)
>>>>>>>
>>>>>>> Cassandra starts outputting:
>>>>>>>
>>>>>>> INFO 08:08:49,864 Cassandra starting up...
>>>>>>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving
>>>>>>> 220554976 used; max is 1190723584
>>>>>>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving
>>>>>>> 230245792 used; max is 1190723584
>>>>>>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving
>>>>>>> 250127792 used; max is 1190723584
>>>>>>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving
>>>>>>> 261657720 used; max is 1190723584
>>>>>>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving
>>>>>>> 273764040 used; max is 1190723584
>>>>>>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving
>>>>>>> 282912656 used; max is 1190723584
>>>>>>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving
>>>>>>> 294861824 used; max is 1190723584
>>>>>>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving
>>>>>>> 304045640 used; max is 1190723584
>>>>>>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving
>>>>>>> 312964344 used; max is 1190723584
>>>>>>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving
>>>>>>> 324961320 used; max is 1190723584
>>>>>>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving
>>>>>>> 345874144 used; max is 1190723584
>>>>>>> INFO 08:20:27,357 Creating new commitlog segment
>>>>>>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log
>>>>>>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving
>>>>>>> 366587984 used; max is 1190723584
>>>>>>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving
>>>>>>> 384059832 used; max is 1190723584
>>>>>>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving
>>>>>>> 383631432 used; max is 1190723584
>>>>>>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving
>>>>>>> 399697888 used; max is 1190723584
>>>>>>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving
>>>>>>> 415428952 used; max is 1190723584
>>>>>>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving
>>>>>>> 433542656 used; max is 1190723584
>>>>>>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving
>>>>>>> 452030560 used; max is 1190723584
>>>>>>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving
>>>>>>> 464013992 used; max is 1190723584
>>>>>>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving
>>>>>>> 471978840 used; max is 1190723584
>>>>>>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving
>>>>>>> 475912120 used; max is 1190723584
>>>>>>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving
>>>>>>> 483742344 used; max is 1190723584
>>>>>>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving
>>>>>>> 491761672 used; max is 1190723584
>>>>>>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving
>>>>>>> 507442800 used; max is 1190723584
>>>>>>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving
>>>>>>> 523028304 used; max is 1190723584
>>>>>>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving
>>>>>>> 530865568 used; max is 1190723584
>>>>>>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving
>>>>>>> 542690216 used; max is 1190723584
>>>>>>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving
>>>>>>> 550728232 used; max is 1190723584
>>>>>>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving
>>>>>>> 573981584 used; max is 1190723584
>>>>>>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving
>>>>>>> 563891472 used; max is 1190723584
>>>>>>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving
>>>>>>> 575756040 used; max is 1190723584
>>>>>>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving
>>>>>>> 583631432 used; max is 1190723584
>>>>>>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving
>>>>>>> 595405816 used; max is 1190723584
>>>>>>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving
>>>>>>> 603389432 used; max is 1190723584
>>>>>>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving
>>>>>>> 615150584 used; max is 1190723584
>>>>>>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving
>>>>>>> 623125104 used; max is 1190723584
>>>>>>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving
>>>>>>> 634661008 used; max is 1190723584
>>>>>>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving
>>>>>>> 646272840 used; max is 1190723584
>>>>>>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving
>>>>>>> 653978688 used; max is 1190723584
>>>>>>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving
>>>>>>> 665462760 used; max is 1190723584
>>>>>>> INFO 08:26:44,895 Creating new commitlog segment
>>>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log
>>>>>>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving
>>>>>>> 673423168 used; max is 1190723584
>>>>>>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving
>>>>>>> 685277616 used; max is 1190723584
>>>>>>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving
>>>>>>> 692803112 used; max is 1190723584
>>>>>>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving
>>>>>>> 700213488 used; max is 1190723584
>>>>>>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving
>>>>>>> 708002984 used; max is 1190723584
>>>>>>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving
>>>>>>> 719547672 used; max is 1190723584
>>>>>>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving
>>>>>>> 727261696 used; max is 1190723584
>>>>>>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving
>>>>>>> 738760480 used; max is 1190723584
>>>>>>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving
>>>>>>> 746376392 used; max is 1190723584
>>>>>>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving
>>>>>>> 754102376 used; max is 1190723584
>>>>>>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving
>>>>>>> 757926896 used; max is 1190723584
>>>>>>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving
>>>>>>> 763964048 used; max is 1190723584
>>>>>>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving
>>>>>>> 771496792 used; max is 1190723584
>>>>>>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving
>>>>>>> 779088496 used; max is 1190723584
>>>>>>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving
>>>>>>> 786858536 used; max is 1190723584
>>>>>>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving
>>>>>>> 794395528 used; max is 1190723584
>>>>>>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving
>>>>>>> 805862000 used; max is 1190723584
>>>>>>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving
>>>>>>> 813708392 used; max is 1190723584
>>>>>>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving
>>>>>>> 821245568 used; max is 1190723584
>>>>>>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving
>>>>>>> 828614768 used; max is 1190723584
>>>>>>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving
>>>>>>> 835962544 used; max is 1190723584
>>>>>>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving
>>>>>>> 844183384 used; max is 1190723584
>>>>>>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving
>>>>>>> 847924056 used; max is 1190723584
>>>>>>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving
>>>>>>> 841384040 used; max is 1190723584
>>>>>>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving
>>>>>>> 839554448 used; max is 1190723584
>>>>>>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving
>>>>>>> 850851600 used; max is 1190723584
>>>>>>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving
>>>>>>> 869886032 used; max is 1190723584
>>>>>>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving
>>>>>>> 881346312 used; max is 1190723584
>>>>>>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving
>>>>>>> 892677840 used; max is 1190723584
>>>>>>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving
>>>>>>> 911084096 used; max is 1190723584
>>>>>>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving
>>>>>>> 922370800 used; max is 1190723584
>>>>>>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving
>>>>>>> 929916416 used; max is 1190723584
>>>>>>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving
>>>>>>> 941201152 used; max is 1190723584
>>>>>>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving
>>>>>>> 948631600 used; max is 1190723584
>>>>>>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving
>>>>>>> 956424384 used; max is 1190723584
>>>>>>> INFO 08:33:14,864 Creating new commitlog segment
>>>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log
>>>>>>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving
>>>>>>> 972976288 used; max is 1190723584
>>>>>>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving
>>>>>>> 981829584 used; max is 1190723584
>>>>>>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving
>>>>>>> 995907192 used; max is 1190723584
>>>>>>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving
>>>>>>> 1011914464 used; max is 1190723584
>>>>>>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408
>>>>>>> reclaimed leaving 1015729984 used; max is 1190723584
>>>>>>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912
>>>>>>> reclaimed leaving 1019176200 used; max is 1190723584
>>>>>>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088
>>>>>>> reclaimed leaving 1022673936 used; max is 1190723584
>>>>>>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904
>>>>>>> reclaimed leaving 1025979696 used; max is 1190723584
>>>>>>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072
>>>>>>> reclaimed leaving 1028992352 used; max is 1190723584
>>>>>>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440
>>>>>>> reclaimed leaving 1031961224 used; max is 1190723584
>>>>>>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568
>>>>>>> reclaimed leaving 1034536576 used; max is 1190723584
>>>>>>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880
>>>>>>> reclaimed leaving 1037266456 used; max is 1190723584
>>>>>>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432
>>>>>>> reclaimed leaving 1039709528 used; max is 1190723584
>>>>>>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800
>>>>>>> reclaimed leaving 1042230248 used; max is 1190723584
>>>>>>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936
>>>>>>> reclaimed leaving 1044527600 used; max is 1190723584
>>>>>>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008
>>>>>>> reclaimed leaving 1046621520 used; max is 1190723584
>>>>>>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336
>>>>>>> reclaimed leaving 1048677040 used; max is 1190723584
>>>>>>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368
>>>>>>> reclaimed leaving 1050563688 used; max is 1190723584
>>>>>>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112
>>>>>>> reclaimed leaving 1052423824 used; max is 1190723584
>>>>>>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048
>>>>>>> reclaimed leaving 1054138992 used; max is 1190723584
>>>>>>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560
>>>>>>> reclaimed leaving 1055761280 used; max is 1190723584
>>>>>>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728
>>>>>>> reclaimed leaving 1057201248 used; max is 1190723584
>>>>>>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792
>>>>>>> reclaimed leaving 1058496000 used; max is 1190723584
>>>>>>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136
>>>>>>> reclaimed leaving 1060011424 used; max is 1190723584
>>>>>>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792
>>>>>>> reclaimed leaving 1061388688 used; max is 1190723584
>>>>>>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096
>>>>>>> reclaimed leaving 1062658344 used; max is 1190723584
>>>>>>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488
>>>>>>> reclaimed leaving 1063834408 used; max is 1190723584
>>>>>>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232
>>>>>>> reclaimed leaving 1064800992 used; max is 1190723584
>>>>>>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288
>>>>>>> reclaimed leaving 1065798256 used; max is 1190723584
>>>>>>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680
>>>>>>> reclaimed leaving 1066867192 used; max is 1190723584
>>>>>>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304
>>>>>>> reclaimed leaving 1067925416 used; max is 1190723584
>>>>>>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632
>>>>>>> reclaimed leaving 1068886912 used; max is 1190723584
>>>>>>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184
>>>>>>> reclaimed leaving 1069705264 used; max is 1190723584
>>>>>>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544
>>>>>>> reclaimed leaving 1070568224 used; max is 1190723584
>>>>>>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752
>>>>>>> reclaimed leaving 1071366040 used; max is 1190723584
>>>>>>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104
>>>>>>> reclaimed leaving 1072168400 used; max is 1190723584
>>>>>>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264
>>>>>>> reclaimed leaving 1072743152 used; max is 1190723584
>>>>>>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160
>>>>>>> reclaimed leaving 1073334720 used; max is 1190723584
>>>>>>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152
>>>>>>> reclaimed leaving 1073930576 used; max is 1190723584
>>>>>>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416
>>>>>>> reclaimed leaving 1074548160 used; max is 1190723584
>>>>>>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544
>>>>>>> reclaimed leaving 1075122368 used; max is 1190723584
>>>>>>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272
>>>>>>> reclaimed leaving 1075699904 used; max is 1190723584
>>>>>>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552
>>>>>>> reclaimed leaving 1076227648 used; max is 1190723584
>>>>>>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976
>>>>>>> reclaimed leaving 1076747648 used; max is 1190723584
>>>>>>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584
>>>>>>> reclaimed leaving 1077152296 used; max is 1190723584
>>>>>>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168
>>>>>>> reclaimed leaving 1077567128 used; max is 1190723584
>>>>>>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704
>>>>>>> reclaimed leaving 1077987824 used; max is 1190723584
>>>>>>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848
>>>>>>> reclaimed leaving 1078393448 used; max is 1190723584
>>>>>>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136
>>>>>>> reclaimed leaving 1078708784 used; max is 1190723584
>>>>>>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344
>>>>>>> reclaimed leaving 1079074464 used; max is 1190723584
>>>>>>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992
>>>>>>> reclaimed leaving 1079409016 used; max is 1190723584
>>>>>>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952
>>>>>>> reclaimed leaving 1079749672 used; max is 1190723584
>>>>>>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768
>>>>>>> reclaimed leaving 1080014112 used; max is 1190723584
>>>>>>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216
>>>>>>> reclaimed leaving 1080278992 used; max is 1190723584
>>>>>>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912
>>>>>>> reclaimed leaving 1080574456 used; max is 1190723584
>>>>>>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376
>>>>>>> reclaimed leaving 1080884096 used; max is 1190723584
>>>>>>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512
>>>>>>> reclaimed leaving 1081076888 used; max is 1190723584
>>>>>>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168
>>>>>>> reclaimed leaving 1081313872 used; max is 1190723584
>>>>>>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168
>>>>>>> reclaimed leaving 1081528880 used; max is 1190723584
>>>>>>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152
>>>>>>> reclaimed leaving 1081704648 used; max is 1190723584
>>>>>>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216
>>>>>>> reclaimed leaving 1081889512 used; max is 1190723584
>>>>>>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392
>>>>>>> reclaimed leaving 1082072568 used; max is 1190723584
>>>>>>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376
>>>>>>> reclaimed leaving 1082270680 used; max is 1190723584
>>>>>>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920
>>>>>>> reclaimed leaving 1082522872 used; max is 1190723584
>>>>>>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392
>>>>>>> reclaimed leaving 1082632368 used; max is 1190723584
>>>>>>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560
>>>>>>> reclaimed leaving 1082794472 used; max is 1190723584
>>>>>>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024
>>>>>>> reclaimed leaving 1082963008 used; max is 1190723584
>>>>>>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504
>>>>>>> reclaimed leaving 1083117048 used; max is 1190723584
>>>>>>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192
>>>>>>> reclaimed leaving 1083213304 used; max is 1190723584
>>>>>>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392
>>>>>>> reclaimed leaving 1083332280 used; max is 1190723584
>>>>>>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808
>>>>>>> reclaimed leaving 1083448744 used; max is 1190723584
>>>>>>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864
>>>>>>> reclaimed leaving 1083601864 used; max is 1190723584
>>>>>>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520
>>>>>>> reclaimed leaving 1083680168 used; max is 1190723584
>>>>>>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872
>>>>>>> reclaimed leaving 1083814848 used; max is 1190723584
>>>>>>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648
>>>>>>> reclaimed leaving 1083871152 used; max is 1190723584
>>>>>>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720
>>>>>>> reclaimed leaving 1083984728 used; max is 1190723584
>>>>>>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056
>>>>>>> reclaimed leaving 1084063504 used; max is 1190723584
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Import using cassandra 0.6.1
Posted by Sonny Heer <so...@gmail.com>.
What does OOM stand for?
for a given insert the size is small (meaning the a single insert
operation only has about a sentence of data) although as the insert
process continues, the columns under a given row key could potentially
grow to be large. Is that what you mean?
An operation entails:
Read
Insert row (IE: rowkey: FOO columnName: BLAH value: 10)
Delete a row
Insert different row (IE: rowkey: FOO columnName: 0000010|Blah value: 10)
millions of these operations are performed in sequence as files are
read from a directory source.
On Wed, Apr 21, 2010 at 2:37 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> then that's not the problem.
>
> are you writing large rows that OOM during compaction?
>
> On Wed, Apr 21, 2010 at 4:34 PM, Sonny Heer <so...@gmail.com> wrote:
>> They are showing up as completed? Is this correct:
>>
>>
>> Pool Name Active Pending Completed
>> STREAM-STAGE 0 0 0
>> RESPONSE-STAGE 0 0 0
>> ROW-READ-STAGE 0 0 517446
>> LB-OPERATIONS 0 0 0
>> MESSAGE-DESERIALIZER-POOL 0 0 0
>> GMFD 0 0 0
>> LB-TARGET 0 0 0
>> CONSISTENCY-MANAGER 0 0 0
>> ROW-MUTATION-STAGE 0 0 1353622
>> MESSAGE-STREAMING-POOL 0 0 0
>> LOAD-BALANCER-STAGE 0 0 0
>> FLUSH-SORTER-POOL 0 0 0
>> MEMTABLE-POST-FLUSHER 0 0 0
>> FLUSH-WRITER-POOL 0 0 0
>> AE-SERVICE-STAGE 0 0 0
>>
>>
>> On Wed, Apr 21, 2010 at 2:09 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>> you need to figure out where the memory is going. check tpstats, if
>>> the pending ops are large somewhere that means you're just generating
>>> insert ops faster than it can handle.
>>>
>>> On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer <so...@gmail.com> wrote:
>>>> note: I'm using the Thrift API to insert. The commitLog directory
>>>> continues to grow. The heap size continues to grow as well.
>>>>
>>>> I decreased MemtableSizeInMB size, but noticed no changes. Any idea
>>>> what is causing this, and/or what property i need to tweek to
>>>> alleviate this? What is the "insert threshold"?
>>>>
>>>> I moved to a more powerful node as well, it still ended up failing
>>>> just after a longer period.
>>>>
>>>> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
>>>>>
>>>>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <so...@gmail.com> wrote:
>>>>>> Currently running on a single node with intensive write operations.
>>>>>>
>>>>>>
>>>>>> After running for a while...
>>>>>>
>>>>>> Client starts outputting:
>>>>>>
>>>>>> TimedOutException()
>>>>>> at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232)
>>>>>> at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670)
>>>>>> at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643)
>>>>>>
>>>>>> Cassandra starts outputting:
>>>>>>
>>>>>> INFO 08:08:49,864 Cassandra starting up...
>>>>>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving
>>>>>> 220554976 used; max is 1190723584
>>>>>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving
>>>>>> 230245792 used; max is 1190723584
>>>>>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving
>>>>>> 250127792 used; max is 1190723584
>>>>>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving
>>>>>> 261657720 used; max is 1190723584
>>>>>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving
>>>>>> 273764040 used; max is 1190723584
>>>>>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving
>>>>>> 282912656 used; max is 1190723584
>>>>>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving
>>>>>> 294861824 used; max is 1190723584
>>>>>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving
>>>>>> 304045640 used; max is 1190723584
>>>>>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving
>>>>>> 312964344 used; max is 1190723584
>>>>>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving
>>>>>> 324961320 used; max is 1190723584
>>>>>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving
>>>>>> 345874144 used; max is 1190723584
>>>>>> INFO 08:20:27,357 Creating new commitlog segment
>>>>>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log
>>>>>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving
>>>>>> 366587984 used; max is 1190723584
>>>>>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving
>>>>>> 384059832 used; max is 1190723584
>>>>>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving
>>>>>> 383631432 used; max is 1190723584
>>>>>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving
>>>>>> 399697888 used; max is 1190723584
>>>>>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving
>>>>>> 415428952 used; max is 1190723584
>>>>>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving
>>>>>> 433542656 used; max is 1190723584
>>>>>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving
>>>>>> 452030560 used; max is 1190723584
>>>>>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving
>>>>>> 464013992 used; max is 1190723584
>>>>>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving
>>>>>> 471978840 used; max is 1190723584
>>>>>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving
>>>>>> 475912120 used; max is 1190723584
>>>>>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving
>>>>>> 483742344 used; max is 1190723584
>>>>>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving
>>>>>> 491761672 used; max is 1190723584
>>>>>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving
>>>>>> 507442800 used; max is 1190723584
>>>>>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving
>>>>>> 523028304 used; max is 1190723584
>>>>>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving
>>>>>> 530865568 used; max is 1190723584
>>>>>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving
>>>>>> 542690216 used; max is 1190723584
>>>>>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving
>>>>>> 550728232 used; max is 1190723584
>>>>>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving
>>>>>> 573981584 used; max is 1190723584
>>>>>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving
>>>>>> 563891472 used; max is 1190723584
>>>>>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving
>>>>>> 575756040 used; max is 1190723584
>>>>>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving
>>>>>> 583631432 used; max is 1190723584
>>>>>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving
>>>>>> 595405816 used; max is 1190723584
>>>>>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving
>>>>>> 603389432 used; max is 1190723584
>>>>>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving
>>>>>> 615150584 used; max is 1190723584
>>>>>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving
>>>>>> 623125104 used; max is 1190723584
>>>>>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving
>>>>>> 634661008 used; max is 1190723584
>>>>>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving
>>>>>> 646272840 used; max is 1190723584
>>>>>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving
>>>>>> 653978688 used; max is 1190723584
>>>>>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving
>>>>>> 665462760 used; max is 1190723584
>>>>>> INFO 08:26:44,895 Creating new commitlog segment
>>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log
>>>>>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving
>>>>>> 673423168 used; max is 1190723584
>>>>>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving
>>>>>> 685277616 used; max is 1190723584
>>>>>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving
>>>>>> 692803112 used; max is 1190723584
>>>>>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving
>>>>>> 700213488 used; max is 1190723584
>>>>>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving
>>>>>> 708002984 used; max is 1190723584
>>>>>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving
>>>>>> 719547672 used; max is 1190723584
>>>>>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving
>>>>>> 727261696 used; max is 1190723584
>>>>>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving
>>>>>> 738760480 used; max is 1190723584
>>>>>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving
>>>>>> 746376392 used; max is 1190723584
>>>>>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving
>>>>>> 754102376 used; max is 1190723584
>>>>>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving
>>>>>> 757926896 used; max is 1190723584
>>>>>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving
>>>>>> 763964048 used; max is 1190723584
>>>>>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving
>>>>>> 771496792 used; max is 1190723584
>>>>>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving
>>>>>> 779088496 used; max is 1190723584
>>>>>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving
>>>>>> 786858536 used; max is 1190723584
>>>>>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving
>>>>>> 794395528 used; max is 1190723584
>>>>>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving
>>>>>> 805862000 used; max is 1190723584
>>>>>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving
>>>>>> 813708392 used; max is 1190723584
>>>>>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving
>>>>>> 821245568 used; max is 1190723584
>>>>>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving
>>>>>> 828614768 used; max is 1190723584
>>>>>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving
>>>>>> 835962544 used; max is 1190723584
>>>>>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving
>>>>>> 844183384 used; max is 1190723584
>>>>>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving
>>>>>> 847924056 used; max is 1190723584
>>>>>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving
>>>>>> 841384040 used; max is 1190723584
>>>>>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving
>>>>>> 839554448 used; max is 1190723584
>>>>>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving
>>>>>> 850851600 used; max is 1190723584
>>>>>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving
>>>>>> 869886032 used; max is 1190723584
>>>>>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving
>>>>>> 881346312 used; max is 1190723584
>>>>>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving
>>>>>> 892677840 used; max is 1190723584
>>>>>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving
>>>>>> 911084096 used; max is 1190723584
>>>>>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving
>>>>>> 922370800 used; max is 1190723584
>>>>>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving
>>>>>> 929916416 used; max is 1190723584
>>>>>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving
>>>>>> 941201152 used; max is 1190723584
>>>>>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving
>>>>>> 948631600 used; max is 1190723584
>>>>>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving
>>>>>> 956424384 used; max is 1190723584
>>>>>> INFO 08:33:14,864 Creating new commitlog segment
>>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log
>>>>>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving
>>>>>> 972976288 used; max is 1190723584
>>>>>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving
>>>>>> 981829584 used; max is 1190723584
>>>>>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving
>>>>>> 995907192 used; max is 1190723584
>>>>>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving
>>>>>> 1011914464 used; max is 1190723584
>>>>>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408
>>>>>> reclaimed leaving 1015729984 used; max is 1190723584
>>>>>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912
>>>>>> reclaimed leaving 1019176200 used; max is 1190723584
>>>>>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088
>>>>>> reclaimed leaving 1022673936 used; max is 1190723584
>>>>>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904
>>>>>> reclaimed leaving 1025979696 used; max is 1190723584
>>>>>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072
>>>>>> reclaimed leaving 1028992352 used; max is 1190723584
>>>>>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440
>>>>>> reclaimed leaving 1031961224 used; max is 1190723584
>>>>>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568
>>>>>> reclaimed leaving 1034536576 used; max is 1190723584
>>>>>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880
>>>>>> reclaimed leaving 1037266456 used; max is 1190723584
>>>>>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432
>>>>>> reclaimed leaving 1039709528 used; max is 1190723584
>>>>>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800
>>>>>> reclaimed leaving 1042230248 used; max is 1190723584
>>>>>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936
>>>>>> reclaimed leaving 1044527600 used; max is 1190723584
>>>>>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008
>>>>>> reclaimed leaving 1046621520 used; max is 1190723584
>>>>>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336
>>>>>> reclaimed leaving 1048677040 used; max is 1190723584
>>>>>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368
>>>>>> reclaimed leaving 1050563688 used; max is 1190723584
>>>>>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112
>>>>>> reclaimed leaving 1052423824 used; max is 1190723584
>>>>>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048
>>>>>> reclaimed leaving 1054138992 used; max is 1190723584
>>>>>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560
>>>>>> reclaimed leaving 1055761280 used; max is 1190723584
>>>>>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728
>>>>>> reclaimed leaving 1057201248 used; max is 1190723584
>>>>>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792
>>>>>> reclaimed leaving 1058496000 used; max is 1190723584
>>>>>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136
>>>>>> reclaimed leaving 1060011424 used; max is 1190723584
>>>>>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792
>>>>>> reclaimed leaving 1061388688 used; max is 1190723584
>>>>>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096
>>>>>> reclaimed leaving 1062658344 used; max is 1190723584
>>>>>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488
>>>>>> reclaimed leaving 1063834408 used; max is 1190723584
>>>>>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232
>>>>>> reclaimed leaving 1064800992 used; max is 1190723584
>>>>>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288
>>>>>> reclaimed leaving 1065798256 used; max is 1190723584
>>>>>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680
>>>>>> reclaimed leaving 1066867192 used; max is 1190723584
>>>>>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304
>>>>>> reclaimed leaving 1067925416 used; max is 1190723584
>>>>>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632
>>>>>> reclaimed leaving 1068886912 used; max is 1190723584
>>>>>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184
>>>>>> reclaimed leaving 1069705264 used; max is 1190723584
>>>>>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544
>>>>>> reclaimed leaving 1070568224 used; max is 1190723584
>>>>>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752
>>>>>> reclaimed leaving 1071366040 used; max is 1190723584
>>>>>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104
>>>>>> reclaimed leaving 1072168400 used; max is 1190723584
>>>>>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264
>>>>>> reclaimed leaving 1072743152 used; max is 1190723584
>>>>>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160
>>>>>> reclaimed leaving 1073334720 used; max is 1190723584
>>>>>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152
>>>>>> reclaimed leaving 1073930576 used; max is 1190723584
>>>>>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416
>>>>>> reclaimed leaving 1074548160 used; max is 1190723584
>>>>>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544
>>>>>> reclaimed leaving 1075122368 used; max is 1190723584
>>>>>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272
>>>>>> reclaimed leaving 1075699904 used; max is 1190723584
>>>>>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552
>>>>>> reclaimed leaving 1076227648 used; max is 1190723584
>>>>>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976
>>>>>> reclaimed leaving 1076747648 used; max is 1190723584
>>>>>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584
>>>>>> reclaimed leaving 1077152296 used; max is 1190723584
>>>>>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168
>>>>>> reclaimed leaving 1077567128 used; max is 1190723584
>>>>>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704
>>>>>> reclaimed leaving 1077987824 used; max is 1190723584
>>>>>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848
>>>>>> reclaimed leaving 1078393448 used; max is 1190723584
>>>>>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136
>>>>>> reclaimed leaving 1078708784 used; max is 1190723584
>>>>>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344
>>>>>> reclaimed leaving 1079074464 used; max is 1190723584
>>>>>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992
>>>>>> reclaimed leaving 1079409016 used; max is 1190723584
>>>>>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952
>>>>>> reclaimed leaving 1079749672 used; max is 1190723584
>>>>>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768
>>>>>> reclaimed leaving 1080014112 used; max is 1190723584
>>>>>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216
>>>>>> reclaimed leaving 1080278992 used; max is 1190723584
>>>>>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912
>>>>>> reclaimed leaving 1080574456 used; max is 1190723584
>>>>>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376
>>>>>> reclaimed leaving 1080884096 used; max is 1190723584
>>>>>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512
>>>>>> reclaimed leaving 1081076888 used; max is 1190723584
>>>>>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168
>>>>>> reclaimed leaving 1081313872 used; max is 1190723584
>>>>>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168
>>>>>> reclaimed leaving 1081528880 used; max is 1190723584
>>>>>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152
>>>>>> reclaimed leaving 1081704648 used; max is 1190723584
>>>>>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216
>>>>>> reclaimed leaving 1081889512 used; max is 1190723584
>>>>>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392
>>>>>> reclaimed leaving 1082072568 used; max is 1190723584
>>>>>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376
>>>>>> reclaimed leaving 1082270680 used; max is 1190723584
>>>>>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920
>>>>>> reclaimed leaving 1082522872 used; max is 1190723584
>>>>>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392
>>>>>> reclaimed leaving 1082632368 used; max is 1190723584
>>>>>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560
>>>>>> reclaimed leaving 1082794472 used; max is 1190723584
>>>>>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024
>>>>>> reclaimed leaving 1082963008 used; max is 1190723584
>>>>>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504
>>>>>> reclaimed leaving 1083117048 used; max is 1190723584
>>>>>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192
>>>>>> reclaimed leaving 1083213304 used; max is 1190723584
>>>>>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392
>>>>>> reclaimed leaving 1083332280 used; max is 1190723584
>>>>>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808
>>>>>> reclaimed leaving 1083448744 used; max is 1190723584
>>>>>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864
>>>>>> reclaimed leaving 1083601864 used; max is 1190723584
>>>>>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520
>>>>>> reclaimed leaving 1083680168 used; max is 1190723584
>>>>>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872
>>>>>> reclaimed leaving 1083814848 used; max is 1190723584
>>>>>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648
>>>>>> reclaimed leaving 1083871152 used; max is 1190723584
>>>>>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720
>>>>>> reclaimed leaving 1083984728 used; max is 1190723584
>>>>>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056
>>>>>> reclaimed leaving 1084063504 used; max is 1190723584
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Import using cassandra 0.6.1
Posted by Jonathan Ellis <jb...@gmail.com>.
then that's not the problem.
are you writing large rows that OOM during compaction?
On Wed, Apr 21, 2010 at 4:34 PM, Sonny Heer <so...@gmail.com> wrote:
> They are showing up as completed? Is this correct:
>
>
> Pool Name Active Pending Completed
> STREAM-STAGE 0 0 0
> RESPONSE-STAGE 0 0 0
> ROW-READ-STAGE 0 0 517446
> LB-OPERATIONS 0 0 0
> MESSAGE-DESERIALIZER-POOL 0 0 0
> GMFD 0 0 0
> LB-TARGET 0 0 0
> CONSISTENCY-MANAGER 0 0 0
> ROW-MUTATION-STAGE 0 0 1353622
> MESSAGE-STREAMING-POOL 0 0 0
> LOAD-BALANCER-STAGE 0 0 0
> FLUSH-SORTER-POOL 0 0 0
> MEMTABLE-POST-FLUSHER 0 0 0
> FLUSH-WRITER-POOL 0 0 0
> AE-SERVICE-STAGE 0 0 0
>
>
> On Wed, Apr 21, 2010 at 2:09 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> you need to figure out where the memory is going. check tpstats, if
>> the pending ops are large somewhere that means you're just generating
>> insert ops faster than it can handle.
>>
>> On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer <so...@gmail.com> wrote:
>>> note: I'm using the Thrift API to insert. The commitLog directory
>>> continues to grow. The heap size continues to grow as well.
>>>
>>> I decreased MemtableSizeInMB size, but noticed no changes. Any idea
>>> what is causing this, and/or what property i need to tweek to
>>> alleviate this? What is the "insert threshold"?
>>>
>>> I moved to a more powerful node as well, it still ended up failing
>>> just after a longer period.
>>>
>>> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>>>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
>>>>
>>>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <so...@gmail.com> wrote:
>>>>> Currently running on a single node with intensive write operations.
>>>>>
>>>>>
>>>>> After running for a while...
>>>>>
>>>>> Client starts outputting:
>>>>>
>>>>> TimedOutException()
>>>>> at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232)
>>>>> at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670)
>>>>> at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643)
>>>>>
>>>>> Cassandra starts outputting:
>>>>>
>>>>> INFO 08:08:49,864 Cassandra starting up...
>>>>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving
>>>>> 220554976 used; max is 1190723584
>>>>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving
>>>>> 230245792 used; max is 1190723584
>>>>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving
>>>>> 250127792 used; max is 1190723584
>>>>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving
>>>>> 261657720 used; max is 1190723584
>>>>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving
>>>>> 273764040 used; max is 1190723584
>>>>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving
>>>>> 282912656 used; max is 1190723584
>>>>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving
>>>>> 294861824 used; max is 1190723584
>>>>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving
>>>>> 304045640 used; max is 1190723584
>>>>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving
>>>>> 312964344 used; max is 1190723584
>>>>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving
>>>>> 324961320 used; max is 1190723584
>>>>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving
>>>>> 345874144 used; max is 1190723584
>>>>> INFO 08:20:27,357 Creating new commitlog segment
>>>>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log
>>>>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving
>>>>> 366587984 used; max is 1190723584
>>>>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving
>>>>> 384059832 used; max is 1190723584
>>>>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving
>>>>> 383631432 used; max is 1190723584
>>>>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving
>>>>> 399697888 used; max is 1190723584
>>>>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving
>>>>> 415428952 used; max is 1190723584
>>>>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving
>>>>> 433542656 used; max is 1190723584
>>>>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving
>>>>> 452030560 used; max is 1190723584
>>>>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving
>>>>> 464013992 used; max is 1190723584
>>>>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving
>>>>> 471978840 used; max is 1190723584
>>>>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving
>>>>> 475912120 used; max is 1190723584
>>>>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving
>>>>> 483742344 used; max is 1190723584
>>>>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving
>>>>> 491761672 used; max is 1190723584
>>>>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving
>>>>> 507442800 used; max is 1190723584
>>>>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving
>>>>> 523028304 used; max is 1190723584
>>>>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving
>>>>> 530865568 used; max is 1190723584
>>>>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving
>>>>> 542690216 used; max is 1190723584
>>>>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving
>>>>> 550728232 used; max is 1190723584
>>>>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving
>>>>> 573981584 used; max is 1190723584
>>>>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving
>>>>> 563891472 used; max is 1190723584
>>>>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving
>>>>> 575756040 used; max is 1190723584
>>>>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving
>>>>> 583631432 used; max is 1190723584
>>>>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving
>>>>> 595405816 used; max is 1190723584
>>>>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving
>>>>> 603389432 used; max is 1190723584
>>>>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving
>>>>> 615150584 used; max is 1190723584
>>>>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving
>>>>> 623125104 used; max is 1190723584
>>>>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving
>>>>> 634661008 used; max is 1190723584
>>>>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving
>>>>> 646272840 used; max is 1190723584
>>>>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving
>>>>> 653978688 used; max is 1190723584
>>>>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving
>>>>> 665462760 used; max is 1190723584
>>>>> INFO 08:26:44,895 Creating new commitlog segment
>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log
>>>>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving
>>>>> 673423168 used; max is 1190723584
>>>>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving
>>>>> 685277616 used; max is 1190723584
>>>>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving
>>>>> 692803112 used; max is 1190723584
>>>>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving
>>>>> 700213488 used; max is 1190723584
>>>>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving
>>>>> 708002984 used; max is 1190723584
>>>>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving
>>>>> 719547672 used; max is 1190723584
>>>>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving
>>>>> 727261696 used; max is 1190723584
>>>>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving
>>>>> 738760480 used; max is 1190723584
>>>>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving
>>>>> 746376392 used; max is 1190723584
>>>>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving
>>>>> 754102376 used; max is 1190723584
>>>>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving
>>>>> 757926896 used; max is 1190723584
>>>>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving
>>>>> 763964048 used; max is 1190723584
>>>>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving
>>>>> 771496792 used; max is 1190723584
>>>>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving
>>>>> 779088496 used; max is 1190723584
>>>>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving
>>>>> 786858536 used; max is 1190723584
>>>>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving
>>>>> 794395528 used; max is 1190723584
>>>>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving
>>>>> 805862000 used; max is 1190723584
>>>>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving
>>>>> 813708392 used; max is 1190723584
>>>>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving
>>>>> 821245568 used; max is 1190723584
>>>>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving
>>>>> 828614768 used; max is 1190723584
>>>>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving
>>>>> 835962544 used; max is 1190723584
>>>>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving
>>>>> 844183384 used; max is 1190723584
>>>>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving
>>>>> 847924056 used; max is 1190723584
>>>>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving
>>>>> 841384040 used; max is 1190723584
>>>>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving
>>>>> 839554448 used; max is 1190723584
>>>>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving
>>>>> 850851600 used; max is 1190723584
>>>>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving
>>>>> 869886032 used; max is 1190723584
>>>>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving
>>>>> 881346312 used; max is 1190723584
>>>>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving
>>>>> 892677840 used; max is 1190723584
>>>>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving
>>>>> 911084096 used; max is 1190723584
>>>>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving
>>>>> 922370800 used; max is 1190723584
>>>>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving
>>>>> 929916416 used; max is 1190723584
>>>>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving
>>>>> 941201152 used; max is 1190723584
>>>>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving
>>>>> 948631600 used; max is 1190723584
>>>>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving
>>>>> 956424384 used; max is 1190723584
>>>>> INFO 08:33:14,864 Creating new commitlog segment
>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log
>>>>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving
>>>>> 972976288 used; max is 1190723584
>>>>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving
>>>>> 981829584 used; max is 1190723584
>>>>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving
>>>>> 995907192 used; max is 1190723584
>>>>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving
>>>>> 1011914464 used; max is 1190723584
>>>>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408
>>>>> reclaimed leaving 1015729984 used; max is 1190723584
>>>>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912
>>>>> reclaimed leaving 1019176200 used; max is 1190723584
>>>>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088
>>>>> reclaimed leaving 1022673936 used; max is 1190723584
>>>>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904
>>>>> reclaimed leaving 1025979696 used; max is 1190723584
>>>>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072
>>>>> reclaimed leaving 1028992352 used; max is 1190723584
>>>>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440
>>>>> reclaimed leaving 1031961224 used; max is 1190723584
>>>>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568
>>>>> reclaimed leaving 1034536576 used; max is 1190723584
>>>>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880
>>>>> reclaimed leaving 1037266456 used; max is 1190723584
>>>>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432
>>>>> reclaimed leaving 1039709528 used; max is 1190723584
>>>>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800
>>>>> reclaimed leaving 1042230248 used; max is 1190723584
>>>>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936
>>>>> reclaimed leaving 1044527600 used; max is 1190723584
>>>>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008
>>>>> reclaimed leaving 1046621520 used; max is 1190723584
>>>>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336
>>>>> reclaimed leaving 1048677040 used; max is 1190723584
>>>>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368
>>>>> reclaimed leaving 1050563688 used; max is 1190723584
>>>>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112
>>>>> reclaimed leaving 1052423824 used; max is 1190723584
>>>>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048
>>>>> reclaimed leaving 1054138992 used; max is 1190723584
>>>>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560
>>>>> reclaimed leaving 1055761280 used; max is 1190723584
>>>>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728
>>>>> reclaimed leaving 1057201248 used; max is 1190723584
>>>>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792
>>>>> reclaimed leaving 1058496000 used; max is 1190723584
>>>>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136
>>>>> reclaimed leaving 1060011424 used; max is 1190723584
>>>>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792
>>>>> reclaimed leaving 1061388688 used; max is 1190723584
>>>>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096
>>>>> reclaimed leaving 1062658344 used; max is 1190723584
>>>>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488
>>>>> reclaimed leaving 1063834408 used; max is 1190723584
>>>>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232
>>>>> reclaimed leaving 1064800992 used; max is 1190723584
>>>>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288
>>>>> reclaimed leaving 1065798256 used; max is 1190723584
>>>>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680
>>>>> reclaimed leaving 1066867192 used; max is 1190723584
>>>>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304
>>>>> reclaimed leaving 1067925416 used; max is 1190723584
>>>>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632
>>>>> reclaimed leaving 1068886912 used; max is 1190723584
>>>>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184
>>>>> reclaimed leaving 1069705264 used; max is 1190723584
>>>>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544
>>>>> reclaimed leaving 1070568224 used; max is 1190723584
>>>>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752
>>>>> reclaimed leaving 1071366040 used; max is 1190723584
>>>>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104
>>>>> reclaimed leaving 1072168400 used; max is 1190723584
>>>>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264
>>>>> reclaimed leaving 1072743152 used; max is 1190723584
>>>>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160
>>>>> reclaimed leaving 1073334720 used; max is 1190723584
>>>>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152
>>>>> reclaimed leaving 1073930576 used; max is 1190723584
>>>>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416
>>>>> reclaimed leaving 1074548160 used; max is 1190723584
>>>>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544
>>>>> reclaimed leaving 1075122368 used; max is 1190723584
>>>>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272
>>>>> reclaimed leaving 1075699904 used; max is 1190723584
>>>>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552
>>>>> reclaimed leaving 1076227648 used; max is 1190723584
>>>>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976
>>>>> reclaimed leaving 1076747648 used; max is 1190723584
>>>>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584
>>>>> reclaimed leaving 1077152296 used; max is 1190723584
>>>>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168
>>>>> reclaimed leaving 1077567128 used; max is 1190723584
>>>>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704
>>>>> reclaimed leaving 1077987824 used; max is 1190723584
>>>>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848
>>>>> reclaimed leaving 1078393448 used; max is 1190723584
>>>>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136
>>>>> reclaimed leaving 1078708784 used; max is 1190723584
>>>>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344
>>>>> reclaimed leaving 1079074464 used; max is 1190723584
>>>>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992
>>>>> reclaimed leaving 1079409016 used; max is 1190723584
>>>>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952
>>>>> reclaimed leaving 1079749672 used; max is 1190723584
>>>>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768
>>>>> reclaimed leaving 1080014112 used; max is 1190723584
>>>>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216
>>>>> reclaimed leaving 1080278992 used; max is 1190723584
>>>>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912
>>>>> reclaimed leaving 1080574456 used; max is 1190723584
>>>>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376
>>>>> reclaimed leaving 1080884096 used; max is 1190723584
>>>>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512
>>>>> reclaimed leaving 1081076888 used; max is 1190723584
>>>>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168
>>>>> reclaimed leaving 1081313872 used; max is 1190723584
>>>>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168
>>>>> reclaimed leaving 1081528880 used; max is 1190723584
>>>>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152
>>>>> reclaimed leaving 1081704648 used; max is 1190723584
>>>>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216
>>>>> reclaimed leaving 1081889512 used; max is 1190723584
>>>>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392
>>>>> reclaimed leaving 1082072568 used; max is 1190723584
>>>>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376
>>>>> reclaimed leaving 1082270680 used; max is 1190723584
>>>>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920
>>>>> reclaimed leaving 1082522872 used; max is 1190723584
>>>>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392
>>>>> reclaimed leaving 1082632368 used; max is 1190723584
>>>>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560
>>>>> reclaimed leaving 1082794472 used; max is 1190723584
>>>>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024
>>>>> reclaimed leaving 1082963008 used; max is 1190723584
>>>>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504
>>>>> reclaimed leaving 1083117048 used; max is 1190723584
>>>>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192
>>>>> reclaimed leaving 1083213304 used; max is 1190723584
>>>>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392
>>>>> reclaimed leaving 1083332280 used; max is 1190723584
>>>>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808
>>>>> reclaimed leaving 1083448744 used; max is 1190723584
>>>>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864
>>>>> reclaimed leaving 1083601864 used; max is 1190723584
>>>>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520
>>>>> reclaimed leaving 1083680168 used; max is 1190723584
>>>>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872
>>>>> reclaimed leaving 1083814848 used; max is 1190723584
>>>>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648
>>>>> reclaimed leaving 1083871152 used; max is 1190723584
>>>>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720
>>>>> reclaimed leaving 1083984728 used; max is 1190723584
>>>>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056
>>>>> reclaimed leaving 1084063504 used; max is 1190723584
>>>>>
>>>>
>>>
>>
>
Re: Import using cassandra 0.6.1
Posted by Sonny Heer <so...@gmail.com>.
They are showing up as completed? Is this correct:
Pool Name Active Pending Completed
STREAM-STAGE 0 0 0
RESPONSE-STAGE 0 0 0
ROW-READ-STAGE 0 0 517446
LB-OPERATIONS 0 0 0
MESSAGE-DESERIALIZER-POOL 0 0 0
GMFD 0 0 0
LB-TARGET 0 0 0
CONSISTENCY-MANAGER 0 0 0
ROW-MUTATION-STAGE 0 0 1353622
MESSAGE-STREAMING-POOL 0 0 0
LOAD-BALANCER-STAGE 0 0 0
FLUSH-SORTER-POOL 0 0 0
MEMTABLE-POST-FLUSHER 0 0 0
FLUSH-WRITER-POOL 0 0 0
AE-SERVICE-STAGE 0 0 0
On Wed, Apr 21, 2010 at 2:09 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> you need to figure out where the memory is going. check tpstats, if
> the pending ops are large somewhere that means you're just generating
> insert ops faster than it can handle.
>
> On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer <so...@gmail.com> wrote:
>> note: I'm using the Thrift API to insert. The commitLog directory
>> continues to grow. The heap size continues to grow as well.
>>
>> I decreased MemtableSizeInMB size, but noticed no changes. Any idea
>> what is causing this, and/or what property i need to tweek to
>> alleviate this? What is the "insert threshold"?
>>
>> I moved to a more powerful node as well, it still ended up failing
>> just after a longer period.
>>
>> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
>>>
>>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <so...@gmail.com> wrote:
>>>> Currently running on a single node with intensive write operations.
>>>>
>>>>
>>>> After running for a while...
>>>>
>>>> Client starts outputting:
>>>>
>>>> TimedOutException()
>>>> at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232)
>>>> at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670)
>>>> at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643)
>>>>
>>>> Cassandra starts outputting:
>>>>
>>>> INFO 08:08:49,864 Cassandra starting up...
>>>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving
>>>> 220554976 used; max is 1190723584
>>>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving
>>>> 230245792 used; max is 1190723584
>>>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving
>>>> 250127792 used; max is 1190723584
>>>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving
>>>> 261657720 used; max is 1190723584
>>>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving
>>>> 273764040 used; max is 1190723584
>>>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving
>>>> 282912656 used; max is 1190723584
>>>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving
>>>> 294861824 used; max is 1190723584
>>>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving
>>>> 304045640 used; max is 1190723584
>>>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving
>>>> 312964344 used; max is 1190723584
>>>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving
>>>> 324961320 used; max is 1190723584
>>>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving
>>>> 345874144 used; max is 1190723584
>>>> INFO 08:20:27,357 Creating new commitlog segment
>>>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log
>>>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving
>>>> 366587984 used; max is 1190723584
>>>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving
>>>> 384059832 used; max is 1190723584
>>>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving
>>>> 383631432 used; max is 1190723584
>>>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving
>>>> 399697888 used; max is 1190723584
>>>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving
>>>> 415428952 used; max is 1190723584
>>>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving
>>>> 433542656 used; max is 1190723584
>>>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving
>>>> 452030560 used; max is 1190723584
>>>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving
>>>> 464013992 used; max is 1190723584
>>>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving
>>>> 471978840 used; max is 1190723584
>>>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving
>>>> 475912120 used; max is 1190723584
>>>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving
>>>> 483742344 used; max is 1190723584
>>>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving
>>>> 491761672 used; max is 1190723584
>>>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving
>>>> 507442800 used; max is 1190723584
>>>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving
>>>> 523028304 used; max is 1190723584
>>>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving
>>>> 530865568 used; max is 1190723584
>>>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving
>>>> 542690216 used; max is 1190723584
>>>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving
>>>> 550728232 used; max is 1190723584
>>>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving
>>>> 573981584 used; max is 1190723584
>>>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving
>>>> 563891472 used; max is 1190723584
>>>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving
>>>> 575756040 used; max is 1190723584
>>>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving
>>>> 583631432 used; max is 1190723584
>>>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving
>>>> 595405816 used; max is 1190723584
>>>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving
>>>> 603389432 used; max is 1190723584
>>>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving
>>>> 615150584 used; max is 1190723584
>>>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving
>>>> 623125104 used; max is 1190723584
>>>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving
>>>> 634661008 used; max is 1190723584
>>>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving
>>>> 646272840 used; max is 1190723584
>>>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving
>>>> 653978688 used; max is 1190723584
>>>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving
>>>> 665462760 used; max is 1190723584
>>>> INFO 08:26:44,895 Creating new commitlog segment
>>>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log
>>>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving
>>>> 673423168 used; max is 1190723584
>>>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving
>>>> 685277616 used; max is 1190723584
>>>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving
>>>> 692803112 used; max is 1190723584
>>>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving
>>>> 700213488 used; max is 1190723584
>>>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving
>>>> 708002984 used; max is 1190723584
>>>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving
>>>> 719547672 used; max is 1190723584
>>>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving
>>>> 727261696 used; max is 1190723584
>>>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving
>>>> 738760480 used; max is 1190723584
>>>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving
>>>> 746376392 used; max is 1190723584
>>>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving
>>>> 754102376 used; max is 1190723584
>>>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving
>>>> 757926896 used; max is 1190723584
>>>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving
>>>> 763964048 used; max is 1190723584
>>>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving
>>>> 771496792 used; max is 1190723584
>>>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving
>>>> 779088496 used; max is 1190723584
>>>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving
>>>> 786858536 used; max is 1190723584
>>>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving
>>>> 794395528 used; max is 1190723584
>>>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving
>>>> 805862000 used; max is 1190723584
>>>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving
>>>> 813708392 used; max is 1190723584
>>>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving
>>>> 821245568 used; max is 1190723584
>>>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving
>>>> 828614768 used; max is 1190723584
>>>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving
>>>> 835962544 used; max is 1190723584
>>>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving
>>>> 844183384 used; max is 1190723584
>>>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving
>>>> 847924056 used; max is 1190723584
>>>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving
>>>> 841384040 used; max is 1190723584
>>>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving
>>>> 839554448 used; max is 1190723584
>>>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving
>>>> 850851600 used; max is 1190723584
>>>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving
>>>> 869886032 used; max is 1190723584
>>>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving
>>>> 881346312 used; max is 1190723584
>>>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving
>>>> 892677840 used; max is 1190723584
>>>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving
>>>> 911084096 used; max is 1190723584
>>>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving
>>>> 922370800 used; max is 1190723584
>>>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving
>>>> 929916416 used; max is 1190723584
>>>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving
>>>> 941201152 used; max is 1190723584
>>>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving
>>>> 948631600 used; max is 1190723584
>>>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving
>>>> 956424384 used; max is 1190723584
>>>> INFO 08:33:14,864 Creating new commitlog segment
>>>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log
>>>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving
>>>> 972976288 used; max is 1190723584
>>>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving
>>>> 981829584 used; max is 1190723584
>>>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving
>>>> 995907192 used; max is 1190723584
>>>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving
>>>> 1011914464 used; max is 1190723584
>>>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408
>>>> reclaimed leaving 1015729984 used; max is 1190723584
>>>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912
>>>> reclaimed leaving 1019176200 used; max is 1190723584
>>>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088
>>>> reclaimed leaving 1022673936 used; max is 1190723584
>>>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904
>>>> reclaimed leaving 1025979696 used; max is 1190723584
>>>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072
>>>> reclaimed leaving 1028992352 used; max is 1190723584
>>>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440
>>>> reclaimed leaving 1031961224 used; max is 1190723584
>>>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568
>>>> reclaimed leaving 1034536576 used; max is 1190723584
>>>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880
>>>> reclaimed leaving 1037266456 used; max is 1190723584
>>>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432
>>>> reclaimed leaving 1039709528 used; max is 1190723584
>>>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800
>>>> reclaimed leaving 1042230248 used; max is 1190723584
>>>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936
>>>> reclaimed leaving 1044527600 used; max is 1190723584
>>>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008
>>>> reclaimed leaving 1046621520 used; max is 1190723584
>>>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336
>>>> reclaimed leaving 1048677040 used; max is 1190723584
>>>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368
>>>> reclaimed leaving 1050563688 used; max is 1190723584
>>>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112
>>>> reclaimed leaving 1052423824 used; max is 1190723584
>>>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048
>>>> reclaimed leaving 1054138992 used; max is 1190723584
>>>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560
>>>> reclaimed leaving 1055761280 used; max is 1190723584
>>>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728
>>>> reclaimed leaving 1057201248 used; max is 1190723584
>>>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792
>>>> reclaimed leaving 1058496000 used; max is 1190723584
>>>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136
>>>> reclaimed leaving 1060011424 used; max is 1190723584
>>>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792
>>>> reclaimed leaving 1061388688 used; max is 1190723584
>>>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096
>>>> reclaimed leaving 1062658344 used; max is 1190723584
>>>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488
>>>> reclaimed leaving 1063834408 used; max is 1190723584
>>>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232
>>>> reclaimed leaving 1064800992 used; max is 1190723584
>>>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288
>>>> reclaimed leaving 1065798256 used; max is 1190723584
>>>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680
>>>> reclaimed leaving 1066867192 used; max is 1190723584
>>>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304
>>>> reclaimed leaving 1067925416 used; max is 1190723584
>>>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632
>>>> reclaimed leaving 1068886912 used; max is 1190723584
>>>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184
>>>> reclaimed leaving 1069705264 used; max is 1190723584
>>>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544
>>>> reclaimed leaving 1070568224 used; max is 1190723584
>>>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752
>>>> reclaimed leaving 1071366040 used; max is 1190723584
>>>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104
>>>> reclaimed leaving 1072168400 used; max is 1190723584
>>>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264
>>>> reclaimed leaving 1072743152 used; max is 1190723584
>>>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160
>>>> reclaimed leaving 1073334720 used; max is 1190723584
>>>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152
>>>> reclaimed leaving 1073930576 used; max is 1190723584
>>>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416
>>>> reclaimed leaving 1074548160 used; max is 1190723584
>>>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544
>>>> reclaimed leaving 1075122368 used; max is 1190723584
>>>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272
>>>> reclaimed leaving 1075699904 used; max is 1190723584
>>>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552
>>>> reclaimed leaving 1076227648 used; max is 1190723584
>>>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976
>>>> reclaimed leaving 1076747648 used; max is 1190723584
>>>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584
>>>> reclaimed leaving 1077152296 used; max is 1190723584
>>>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168
>>>> reclaimed leaving 1077567128 used; max is 1190723584
>>>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704
>>>> reclaimed leaving 1077987824 used; max is 1190723584
>>>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848
>>>> reclaimed leaving 1078393448 used; max is 1190723584
>>>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136
>>>> reclaimed leaving 1078708784 used; max is 1190723584
>>>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344
>>>> reclaimed leaving 1079074464 used; max is 1190723584
>>>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992
>>>> reclaimed leaving 1079409016 used; max is 1190723584
>>>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952
>>>> reclaimed leaving 1079749672 used; max is 1190723584
>>>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768
>>>> reclaimed leaving 1080014112 used; max is 1190723584
>>>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216
>>>> reclaimed leaving 1080278992 used; max is 1190723584
>>>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912
>>>> reclaimed leaving 1080574456 used; max is 1190723584
>>>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376
>>>> reclaimed leaving 1080884096 used; max is 1190723584
>>>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512
>>>> reclaimed leaving 1081076888 used; max is 1190723584
>>>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168
>>>> reclaimed leaving 1081313872 used; max is 1190723584
>>>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168
>>>> reclaimed leaving 1081528880 used; max is 1190723584
>>>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152
>>>> reclaimed leaving 1081704648 used; max is 1190723584
>>>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216
>>>> reclaimed leaving 1081889512 used; max is 1190723584
>>>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392
>>>> reclaimed leaving 1082072568 used; max is 1190723584
>>>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376
>>>> reclaimed leaving 1082270680 used; max is 1190723584
>>>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920
>>>> reclaimed leaving 1082522872 used; max is 1190723584
>>>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392
>>>> reclaimed leaving 1082632368 used; max is 1190723584
>>>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560
>>>> reclaimed leaving 1082794472 used; max is 1190723584
>>>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024
>>>> reclaimed leaving 1082963008 used; max is 1190723584
>>>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504
>>>> reclaimed leaving 1083117048 used; max is 1190723584
>>>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192
>>>> reclaimed leaving 1083213304 used; max is 1190723584
>>>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392
>>>> reclaimed leaving 1083332280 used; max is 1190723584
>>>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808
>>>> reclaimed leaving 1083448744 used; max is 1190723584
>>>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864
>>>> reclaimed leaving 1083601864 used; max is 1190723584
>>>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520
>>>> reclaimed leaving 1083680168 used; max is 1190723584
>>>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872
>>>> reclaimed leaving 1083814848 used; max is 1190723584
>>>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648
>>>> reclaimed leaving 1083871152 used; max is 1190723584
>>>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720
>>>> reclaimed leaving 1083984728 used; max is 1190723584
>>>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056
>>>> reclaimed leaving 1084063504 used; max is 1190723584
>>>>
>>>
>>
>
Re: Import using cassandra 0.6.1
Posted by Jonathan Ellis <jb...@gmail.com>.
you need to figure out where the memory is going. check tpstats, if
the pending ops are large somewhere that means you're just generating
insert ops faster than it can handle.
On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer <so...@gmail.com> wrote:
> note: I'm using the Thrift API to insert. The commitLog directory
> continues to grow. The heap size continues to grow as well.
>
> I decreased MemtableSizeInMB size, but noticed no changes. Any idea
> what is causing this, and/or what property i need to tweek to
> alleviate this? What is the "insert threshold"?
>
> I moved to a more powerful node as well, it still ended up failing
> just after a longer period.
>
> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
>>
>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <so...@gmail.com> wrote:
>>> Currently running on a single node with intensive write operations.
>>>
>>>
>>> After running for a while...
>>>
>>> Client starts outputting:
>>>
>>> TimedOutException()
>>> at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232)
>>> at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670)
>>> at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643)
>>>
>>> Cassandra starts outputting:
>>>
>>> INFO 08:08:49,864 Cassandra starting up...
>>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving
>>> 220554976 used; max is 1190723584
>>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving
>>> 230245792 used; max is 1190723584
>>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving
>>> 250127792 used; max is 1190723584
>>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving
>>> 261657720 used; max is 1190723584
>>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving
>>> 273764040 used; max is 1190723584
>>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving
>>> 282912656 used; max is 1190723584
>>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving
>>> 294861824 used; max is 1190723584
>>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving
>>> 304045640 used; max is 1190723584
>>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving
>>> 312964344 used; max is 1190723584
>>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving
>>> 324961320 used; max is 1190723584
>>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving
>>> 345874144 used; max is 1190723584
>>> INFO 08:20:27,357 Creating new commitlog segment
>>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log
>>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving
>>> 366587984 used; max is 1190723584
>>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving
>>> 384059832 used; max is 1190723584
>>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving
>>> 383631432 used; max is 1190723584
>>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving
>>> 399697888 used; max is 1190723584
>>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving
>>> 415428952 used; max is 1190723584
>>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving
>>> 433542656 used; max is 1190723584
>>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving
>>> 452030560 used; max is 1190723584
>>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving
>>> 464013992 used; max is 1190723584
>>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving
>>> 471978840 used; max is 1190723584
>>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving
>>> 475912120 used; max is 1190723584
>>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving
>>> 483742344 used; max is 1190723584
>>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving
>>> 491761672 used; max is 1190723584
>>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving
>>> 507442800 used; max is 1190723584
>>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving
>>> 523028304 used; max is 1190723584
>>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving
>>> 530865568 used; max is 1190723584
>>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving
>>> 542690216 used; max is 1190723584
>>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving
>>> 550728232 used; max is 1190723584
>>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving
>>> 573981584 used; max is 1190723584
>>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving
>>> 563891472 used; max is 1190723584
>>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving
>>> 575756040 used; max is 1190723584
>>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving
>>> 583631432 used; max is 1190723584
>>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving
>>> 595405816 used; max is 1190723584
>>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving
>>> 603389432 used; max is 1190723584
>>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving
>>> 615150584 used; max is 1190723584
>>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving
>>> 623125104 used; max is 1190723584
>>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving
>>> 634661008 used; max is 1190723584
>>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving
>>> 646272840 used; max is 1190723584
>>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving
>>> 653978688 used; max is 1190723584
>>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving
>>> 665462760 used; max is 1190723584
>>> INFO 08:26:44,895 Creating new commitlog segment
>>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log
>>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving
>>> 673423168 used; max is 1190723584
>>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving
>>> 685277616 used; max is 1190723584
>>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving
>>> 692803112 used; max is 1190723584
>>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving
>>> 700213488 used; max is 1190723584
>>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving
>>> 708002984 used; max is 1190723584
>>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving
>>> 719547672 used; max is 1190723584
>>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving
>>> 727261696 used; max is 1190723584
>>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving
>>> 738760480 used; max is 1190723584
>>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving
>>> 746376392 used; max is 1190723584
>>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving
>>> 754102376 used; max is 1190723584
>>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving
>>> 757926896 used; max is 1190723584
>>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving
>>> 763964048 used; max is 1190723584
>>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving
>>> 771496792 used; max is 1190723584
>>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving
>>> 779088496 used; max is 1190723584
>>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving
>>> 786858536 used; max is 1190723584
>>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving
>>> 794395528 used; max is 1190723584
>>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving
>>> 805862000 used; max is 1190723584
>>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving
>>> 813708392 used; max is 1190723584
>>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving
>>> 821245568 used; max is 1190723584
>>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving
>>> 828614768 used; max is 1190723584
>>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving
>>> 835962544 used; max is 1190723584
>>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving
>>> 844183384 used; max is 1190723584
>>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving
>>> 847924056 used; max is 1190723584
>>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving
>>> 841384040 used; max is 1190723584
>>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving
>>> 839554448 used; max is 1190723584
>>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving
>>> 850851600 used; max is 1190723584
>>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving
>>> 869886032 used; max is 1190723584
>>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving
>>> 881346312 used; max is 1190723584
>>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving
>>> 892677840 used; max is 1190723584
>>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving
>>> 911084096 used; max is 1190723584
>>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving
>>> 922370800 used; max is 1190723584
>>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving
>>> 929916416 used; max is 1190723584
>>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving
>>> 941201152 used; max is 1190723584
>>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving
>>> 948631600 used; max is 1190723584
>>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving
>>> 956424384 used; max is 1190723584
>>> INFO 08:33:14,864 Creating new commitlog segment
>>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log
>>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving
>>> 972976288 used; max is 1190723584
>>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving
>>> 981829584 used; max is 1190723584
>>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving
>>> 995907192 used; max is 1190723584
>>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving
>>> 1011914464 used; max is 1190723584
>>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408
>>> reclaimed leaving 1015729984 used; max is 1190723584
>>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912
>>> reclaimed leaving 1019176200 used; max is 1190723584
>>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088
>>> reclaimed leaving 1022673936 used; max is 1190723584
>>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904
>>> reclaimed leaving 1025979696 used; max is 1190723584
>>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072
>>> reclaimed leaving 1028992352 used; max is 1190723584
>>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440
>>> reclaimed leaving 1031961224 used; max is 1190723584
>>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568
>>> reclaimed leaving 1034536576 used; max is 1190723584
>>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880
>>> reclaimed leaving 1037266456 used; max is 1190723584
>>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432
>>> reclaimed leaving 1039709528 used; max is 1190723584
>>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800
>>> reclaimed leaving 1042230248 used; max is 1190723584
>>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936
>>> reclaimed leaving 1044527600 used; max is 1190723584
>>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008
>>> reclaimed leaving 1046621520 used; max is 1190723584
>>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336
>>> reclaimed leaving 1048677040 used; max is 1190723584
>>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368
>>> reclaimed leaving 1050563688 used; max is 1190723584
>>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112
>>> reclaimed leaving 1052423824 used; max is 1190723584
>>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048
>>> reclaimed leaving 1054138992 used; max is 1190723584
>>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560
>>> reclaimed leaving 1055761280 used; max is 1190723584
>>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728
>>> reclaimed leaving 1057201248 used; max is 1190723584
>>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792
>>> reclaimed leaving 1058496000 used; max is 1190723584
>>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136
>>> reclaimed leaving 1060011424 used; max is 1190723584
>>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792
>>> reclaimed leaving 1061388688 used; max is 1190723584
>>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096
>>> reclaimed leaving 1062658344 used; max is 1190723584
>>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488
>>> reclaimed leaving 1063834408 used; max is 1190723584
>>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232
>>> reclaimed leaving 1064800992 used; max is 1190723584
>>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288
>>> reclaimed leaving 1065798256 used; max is 1190723584
>>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680
>>> reclaimed leaving 1066867192 used; max is 1190723584
>>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304
>>> reclaimed leaving 1067925416 used; max is 1190723584
>>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632
>>> reclaimed leaving 1068886912 used; max is 1190723584
>>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184
>>> reclaimed leaving 1069705264 used; max is 1190723584
>>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544
>>> reclaimed leaving 1070568224 used; max is 1190723584
>>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752
>>> reclaimed leaving 1071366040 used; max is 1190723584
>>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104
>>> reclaimed leaving 1072168400 used; max is 1190723584
>>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264
>>> reclaimed leaving 1072743152 used; max is 1190723584
>>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160
>>> reclaimed leaving 1073334720 used; max is 1190723584
>>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152
>>> reclaimed leaving 1073930576 used; max is 1190723584
>>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416
>>> reclaimed leaving 1074548160 used; max is 1190723584
>>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544
>>> reclaimed leaving 1075122368 used; max is 1190723584
>>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272
>>> reclaimed leaving 1075699904 used; max is 1190723584
>>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552
>>> reclaimed leaving 1076227648 used; max is 1190723584
>>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976
>>> reclaimed leaving 1076747648 used; max is 1190723584
>>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584
>>> reclaimed leaving 1077152296 used; max is 1190723584
>>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168
>>> reclaimed leaving 1077567128 used; max is 1190723584
>>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704
>>> reclaimed leaving 1077987824 used; max is 1190723584
>>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848
>>> reclaimed leaving 1078393448 used; max is 1190723584
>>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136
>>> reclaimed leaving 1078708784 used; max is 1190723584
>>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344
>>> reclaimed leaving 1079074464 used; max is 1190723584
>>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992
>>> reclaimed leaving 1079409016 used; max is 1190723584
>>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952
>>> reclaimed leaving 1079749672 used; max is 1190723584
>>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768
>>> reclaimed leaving 1080014112 used; max is 1190723584
>>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216
>>> reclaimed leaving 1080278992 used; max is 1190723584
>>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912
>>> reclaimed leaving 1080574456 used; max is 1190723584
>>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376
>>> reclaimed leaving 1080884096 used; max is 1190723584
>>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512
>>> reclaimed leaving 1081076888 used; max is 1190723584
>>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168
>>> reclaimed leaving 1081313872 used; max is 1190723584
>>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168
>>> reclaimed leaving 1081528880 used; max is 1190723584
>>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152
>>> reclaimed leaving 1081704648 used; max is 1190723584
>>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216
>>> reclaimed leaving 1081889512 used; max is 1190723584
>>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392
>>> reclaimed leaving 1082072568 used; max is 1190723584
>>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376
>>> reclaimed leaving 1082270680 used; max is 1190723584
>>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920
>>> reclaimed leaving 1082522872 used; max is 1190723584
>>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392
>>> reclaimed leaving 1082632368 used; max is 1190723584
>>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560
>>> reclaimed leaving 1082794472 used; max is 1190723584
>>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024
>>> reclaimed leaving 1082963008 used; max is 1190723584
>>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504
>>> reclaimed leaving 1083117048 used; max is 1190723584
>>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192
>>> reclaimed leaving 1083213304 used; max is 1190723584
>>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392
>>> reclaimed leaving 1083332280 used; max is 1190723584
>>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808
>>> reclaimed leaving 1083448744 used; max is 1190723584
>>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864
>>> reclaimed leaving 1083601864 used; max is 1190723584
>>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520
>>> reclaimed leaving 1083680168 used; max is 1190723584
>>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872
>>> reclaimed leaving 1083814848 used; max is 1190723584
>>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648
>>> reclaimed leaving 1083871152 used; max is 1190723584
>>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720
>>> reclaimed leaving 1083984728 used; max is 1190723584
>>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056
>>> reclaimed leaving 1084063504 used; max is 1190723584
>>>
>>
>
Re: Import using cassandra 0.6.1
Posted by Sonny Heer <so...@gmail.com>.
note: I'm using the Thrift API to insert. The commitLog directory
continues to grow. The heap size continues to grow as well.
I decreased MemtableSizeInMB size, but noticed no changes. Any idea
what is causing this, and/or what property i need to tweek to
alleviate this? What is the "insert threshold"?
I moved to a more powerful node as well, it still ended up failing
just after a longer period.
On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
>
> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <so...@gmail.com> wrote:
>> Currently running on a single node with intensive write operations.
>>
>>
>> After running for a while...
>>
>> Client starts outputting:
>>
>> TimedOutException()
>> at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232)
>> at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670)
>> at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643)
>>
>> Cassandra starts outputting:
>>
>> INFO 08:08:49,864 Cassandra starting up...
>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving
>> 220554976 used; max is 1190723584
>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving
>> 230245792 used; max is 1190723584
>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving
>> 250127792 used; max is 1190723584
>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving
>> 261657720 used; max is 1190723584
>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving
>> 273764040 used; max is 1190723584
>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving
>> 282912656 used; max is 1190723584
>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving
>> 294861824 used; max is 1190723584
>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving
>> 304045640 used; max is 1190723584
>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving
>> 312964344 used; max is 1190723584
>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving
>> 324961320 used; max is 1190723584
>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving
>> 345874144 used; max is 1190723584
>> INFO 08:20:27,357 Creating new commitlog segment
>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log
>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving
>> 366587984 used; max is 1190723584
>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving
>> 384059832 used; max is 1190723584
>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving
>> 383631432 used; max is 1190723584
>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving
>> 399697888 used; max is 1190723584
>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving
>> 415428952 used; max is 1190723584
>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving
>> 433542656 used; max is 1190723584
>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving
>> 452030560 used; max is 1190723584
>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving
>> 464013992 used; max is 1190723584
>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving
>> 471978840 used; max is 1190723584
>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving
>> 475912120 used; max is 1190723584
>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving
>> 483742344 used; max is 1190723584
>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving
>> 491761672 used; max is 1190723584
>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving
>> 507442800 used; max is 1190723584
>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving
>> 523028304 used; max is 1190723584
>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving
>> 530865568 used; max is 1190723584
>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving
>> 542690216 used; max is 1190723584
>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving
>> 550728232 used; max is 1190723584
>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving
>> 573981584 used; max is 1190723584
>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving
>> 563891472 used; max is 1190723584
>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving
>> 575756040 used; max is 1190723584
>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving
>> 583631432 used; max is 1190723584
>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving
>> 595405816 used; max is 1190723584
>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving
>> 603389432 used; max is 1190723584
>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving
>> 615150584 used; max is 1190723584
>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving
>> 623125104 used; max is 1190723584
>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving
>> 634661008 used; max is 1190723584
>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving
>> 646272840 used; max is 1190723584
>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving
>> 653978688 used; max is 1190723584
>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving
>> 665462760 used; max is 1190723584
>> INFO 08:26:44,895 Creating new commitlog segment
>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log
>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving
>> 673423168 used; max is 1190723584
>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving
>> 685277616 used; max is 1190723584
>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving
>> 692803112 used; max is 1190723584
>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving
>> 700213488 used; max is 1190723584
>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving
>> 708002984 used; max is 1190723584
>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving
>> 719547672 used; max is 1190723584
>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving
>> 727261696 used; max is 1190723584
>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving
>> 738760480 used; max is 1190723584
>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving
>> 746376392 used; max is 1190723584
>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving
>> 754102376 used; max is 1190723584
>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving
>> 757926896 used; max is 1190723584
>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving
>> 763964048 used; max is 1190723584
>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving
>> 771496792 used; max is 1190723584
>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving
>> 779088496 used; max is 1190723584
>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving
>> 786858536 used; max is 1190723584
>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving
>> 794395528 used; max is 1190723584
>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving
>> 805862000 used; max is 1190723584
>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving
>> 813708392 used; max is 1190723584
>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving
>> 821245568 used; max is 1190723584
>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving
>> 828614768 used; max is 1190723584
>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving
>> 835962544 used; max is 1190723584
>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving
>> 844183384 used; max is 1190723584
>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving
>> 847924056 used; max is 1190723584
>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving
>> 841384040 used; max is 1190723584
>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving
>> 839554448 used; max is 1190723584
>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving
>> 850851600 used; max is 1190723584
>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving
>> 869886032 used; max is 1190723584
>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving
>> 881346312 used; max is 1190723584
>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving
>> 892677840 used; max is 1190723584
>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving
>> 911084096 used; max is 1190723584
>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving
>> 922370800 used; max is 1190723584
>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving
>> 929916416 used; max is 1190723584
>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving
>> 941201152 used; max is 1190723584
>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving
>> 948631600 used; max is 1190723584
>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving
>> 956424384 used; max is 1190723584
>> INFO 08:33:14,864 Creating new commitlog segment
>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log
>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving
>> 972976288 used; max is 1190723584
>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving
>> 981829584 used; max is 1190723584
>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving
>> 995907192 used; max is 1190723584
>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving
>> 1011914464 used; max is 1190723584
>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408
>> reclaimed leaving 1015729984 used; max is 1190723584
>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912
>> reclaimed leaving 1019176200 used; max is 1190723584
>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088
>> reclaimed leaving 1022673936 used; max is 1190723584
>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904
>> reclaimed leaving 1025979696 used; max is 1190723584
>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072
>> reclaimed leaving 1028992352 used; max is 1190723584
>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440
>> reclaimed leaving 1031961224 used; max is 1190723584
>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568
>> reclaimed leaving 1034536576 used; max is 1190723584
>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880
>> reclaimed leaving 1037266456 used; max is 1190723584
>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432
>> reclaimed leaving 1039709528 used; max is 1190723584
>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800
>> reclaimed leaving 1042230248 used; max is 1190723584
>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936
>> reclaimed leaving 1044527600 used; max is 1190723584
>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008
>> reclaimed leaving 1046621520 used; max is 1190723584
>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336
>> reclaimed leaving 1048677040 used; max is 1190723584
>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368
>> reclaimed leaving 1050563688 used; max is 1190723584
>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112
>> reclaimed leaving 1052423824 used; max is 1190723584
>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048
>> reclaimed leaving 1054138992 used; max is 1190723584
>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560
>> reclaimed leaving 1055761280 used; max is 1190723584
>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728
>> reclaimed leaving 1057201248 used; max is 1190723584
>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792
>> reclaimed leaving 1058496000 used; max is 1190723584
>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136
>> reclaimed leaving 1060011424 used; max is 1190723584
>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792
>> reclaimed leaving 1061388688 used; max is 1190723584
>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096
>> reclaimed leaving 1062658344 used; max is 1190723584
>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488
>> reclaimed leaving 1063834408 used; max is 1190723584
>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232
>> reclaimed leaving 1064800992 used; max is 1190723584
>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288
>> reclaimed leaving 1065798256 used; max is 1190723584
>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680
>> reclaimed leaving 1066867192 used; max is 1190723584
>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304
>> reclaimed leaving 1067925416 used; max is 1190723584
>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632
>> reclaimed leaving 1068886912 used; max is 1190723584
>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184
>> reclaimed leaving 1069705264 used; max is 1190723584
>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544
>> reclaimed leaving 1070568224 used; max is 1190723584
>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752
>> reclaimed leaving 1071366040 used; max is 1190723584
>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104
>> reclaimed leaving 1072168400 used; max is 1190723584
>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264
>> reclaimed leaving 1072743152 used; max is 1190723584
>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160
>> reclaimed leaving 1073334720 used; max is 1190723584
>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152
>> reclaimed leaving 1073930576 used; max is 1190723584
>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416
>> reclaimed leaving 1074548160 used; max is 1190723584
>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544
>> reclaimed leaving 1075122368 used; max is 1190723584
>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272
>> reclaimed leaving 1075699904 used; max is 1190723584
>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552
>> reclaimed leaving 1076227648 used; max is 1190723584
>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976
>> reclaimed leaving 1076747648 used; max is 1190723584
>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584
>> reclaimed leaving 1077152296 used; max is 1190723584
>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168
>> reclaimed leaving 1077567128 used; max is 1190723584
>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704
>> reclaimed leaving 1077987824 used; max is 1190723584
>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848
>> reclaimed leaving 1078393448 used; max is 1190723584
>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136
>> reclaimed leaving 1078708784 used; max is 1190723584
>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344
>> reclaimed leaving 1079074464 used; max is 1190723584
>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992
>> reclaimed leaving 1079409016 used; max is 1190723584
>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952
>> reclaimed leaving 1079749672 used; max is 1190723584
>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768
>> reclaimed leaving 1080014112 used; max is 1190723584
>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216
>> reclaimed leaving 1080278992 used; max is 1190723584
>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912
>> reclaimed leaving 1080574456 used; max is 1190723584
>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376
>> reclaimed leaving 1080884096 used; max is 1190723584
>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512
>> reclaimed leaving 1081076888 used; max is 1190723584
>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168
>> reclaimed leaving 1081313872 used; max is 1190723584
>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168
>> reclaimed leaving 1081528880 used; max is 1190723584
>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152
>> reclaimed leaving 1081704648 used; max is 1190723584
>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216
>> reclaimed leaving 1081889512 used; max is 1190723584
>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392
>> reclaimed leaving 1082072568 used; max is 1190723584
>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376
>> reclaimed leaving 1082270680 used; max is 1190723584
>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920
>> reclaimed leaving 1082522872 used; max is 1190723584
>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392
>> reclaimed leaving 1082632368 used; max is 1190723584
>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560
>> reclaimed leaving 1082794472 used; max is 1190723584
>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024
>> reclaimed leaving 1082963008 used; max is 1190723584
>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504
>> reclaimed leaving 1083117048 used; max is 1190723584
>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192
>> reclaimed leaving 1083213304 used; max is 1190723584
>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392
>> reclaimed leaving 1083332280 used; max is 1190723584
>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808
>> reclaimed leaving 1083448744 used; max is 1190723584
>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864
>> reclaimed leaving 1083601864 used; max is 1190723584
>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520
>> reclaimed leaving 1083680168 used; max is 1190723584
>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872
>> reclaimed leaving 1083814848 used; max is 1190723584
>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648
>> reclaimed leaving 1083871152 used; max is 1190723584
>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720
>> reclaimed leaving 1083984728 used; max is 1190723584
>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056
>> reclaimed leaving 1084063504 used; max is 1190723584
>>
>
Re: Import using cassandra 0.6.1
Posted by Jonathan Ellis <jb...@gmail.com>.
http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <so...@gmail.com> wrote:
> Currently running on a single node with intensive write operations.
>
>
> After running for a while...
>
> Client starts outputting:
>
> TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670)
> at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643)
>
> Cassandra starts outputting:
>
> INFO 08:08:49,864 Cassandra starting up...
> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving
> 220554976 used; max is 1190723584
> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving
> 230245792 used; max is 1190723584
> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving
> 250127792 used; max is 1190723584
> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving
> 261657720 used; max is 1190723584
> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving
> 273764040 used; max is 1190723584
> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving
> 282912656 used; max is 1190723584
> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving
> 294861824 used; max is 1190723584
> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving
> 304045640 used; max is 1190723584
> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving
> 312964344 used; max is 1190723584
> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving
> 324961320 used; max is 1190723584
> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving
> 345874144 used; max is 1190723584
> INFO 08:20:27,357 Creating new commitlog segment
> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log
> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving
> 366587984 used; max is 1190723584
> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving
> 384059832 used; max is 1190723584
> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving
> 383631432 used; max is 1190723584
> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving
> 399697888 used; max is 1190723584
> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving
> 415428952 used; max is 1190723584
> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving
> 433542656 used; max is 1190723584
> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving
> 452030560 used; max is 1190723584
> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving
> 464013992 used; max is 1190723584
> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving
> 471978840 used; max is 1190723584
> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving
> 475912120 used; max is 1190723584
> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving
> 483742344 used; max is 1190723584
> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving
> 491761672 used; max is 1190723584
> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving
> 507442800 used; max is 1190723584
> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving
> 523028304 used; max is 1190723584
> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving
> 530865568 used; max is 1190723584
> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving
> 542690216 used; max is 1190723584
> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving
> 550728232 used; max is 1190723584
> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving
> 573981584 used; max is 1190723584
> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving
> 563891472 used; max is 1190723584
> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving
> 575756040 used; max is 1190723584
> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving
> 583631432 used; max is 1190723584
> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving
> 595405816 used; max is 1190723584
> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving
> 603389432 used; max is 1190723584
> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving
> 615150584 used; max is 1190723584
> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving
> 623125104 used; max is 1190723584
> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving
> 634661008 used; max is 1190723584
> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving
> 646272840 used; max is 1190723584
> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving
> 653978688 used; max is 1190723584
> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving
> 665462760 used; max is 1190723584
> INFO 08:26:44,895 Creating new commitlog segment
> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log
> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving
> 673423168 used; max is 1190723584
> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving
> 685277616 used; max is 1190723584
> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving
> 692803112 used; max is 1190723584
> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving
> 700213488 used; max is 1190723584
> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving
> 708002984 used; max is 1190723584
> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving
> 719547672 used; max is 1190723584
> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving
> 727261696 used; max is 1190723584
> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving
> 738760480 used; max is 1190723584
> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving
> 746376392 used; max is 1190723584
> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving
> 754102376 used; max is 1190723584
> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving
> 757926896 used; max is 1190723584
> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving
> 763964048 used; max is 1190723584
> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving
> 771496792 used; max is 1190723584
> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving
> 779088496 used; max is 1190723584
> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving
> 786858536 used; max is 1190723584
> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving
> 794395528 used; max is 1190723584
> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving
> 805862000 used; max is 1190723584
> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving
> 813708392 used; max is 1190723584
> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving
> 821245568 used; max is 1190723584
> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving
> 828614768 used; max is 1190723584
> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving
> 835962544 used; max is 1190723584
> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving
> 844183384 used; max is 1190723584
> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving
> 847924056 used; max is 1190723584
> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving
> 841384040 used; max is 1190723584
> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving
> 839554448 used; max is 1190723584
> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving
> 850851600 used; max is 1190723584
> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving
> 869886032 used; max is 1190723584
> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving
> 881346312 used; max is 1190723584
> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving
> 892677840 used; max is 1190723584
> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving
> 911084096 used; max is 1190723584
> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving
> 922370800 used; max is 1190723584
> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving
> 929916416 used; max is 1190723584
> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving
> 941201152 used; max is 1190723584
> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving
> 948631600 used; max is 1190723584
> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving
> 956424384 used; max is 1190723584
> INFO 08:33:14,864 Creating new commitlog segment
> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log
> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving
> 972976288 used; max is 1190723584
> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving
> 981829584 used; max is 1190723584
> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving
> 995907192 used; max is 1190723584
> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving
> 1011914464 used; max is 1190723584
> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408
> reclaimed leaving 1015729984 used; max is 1190723584
> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912
> reclaimed leaving 1019176200 used; max is 1190723584
> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088
> reclaimed leaving 1022673936 used; max is 1190723584
> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904
> reclaimed leaving 1025979696 used; max is 1190723584
> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072
> reclaimed leaving 1028992352 used; max is 1190723584
> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440
> reclaimed leaving 1031961224 used; max is 1190723584
> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568
> reclaimed leaving 1034536576 used; max is 1190723584
> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880
> reclaimed leaving 1037266456 used; max is 1190723584
> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432
> reclaimed leaving 1039709528 used; max is 1190723584
> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800
> reclaimed leaving 1042230248 used; max is 1190723584
> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936
> reclaimed leaving 1044527600 used; max is 1190723584
> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008
> reclaimed leaving 1046621520 used; max is 1190723584
> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336
> reclaimed leaving 1048677040 used; max is 1190723584
> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368
> reclaimed leaving 1050563688 used; max is 1190723584
> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112
> reclaimed leaving 1052423824 used; max is 1190723584
> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048
> reclaimed leaving 1054138992 used; max is 1190723584
> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560
> reclaimed leaving 1055761280 used; max is 1190723584
> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728
> reclaimed leaving 1057201248 used; max is 1190723584
> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792
> reclaimed leaving 1058496000 used; max is 1190723584
> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136
> reclaimed leaving 1060011424 used; max is 1190723584
> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792
> reclaimed leaving 1061388688 used; max is 1190723584
> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096
> reclaimed leaving 1062658344 used; max is 1190723584
> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488
> reclaimed leaving 1063834408 used; max is 1190723584
> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232
> reclaimed leaving 1064800992 used; max is 1190723584
> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288
> reclaimed leaving 1065798256 used; max is 1190723584
> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680
> reclaimed leaving 1066867192 used; max is 1190723584
> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304
> reclaimed leaving 1067925416 used; max is 1190723584
> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632
> reclaimed leaving 1068886912 used; max is 1190723584
> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184
> reclaimed leaving 1069705264 used; max is 1190723584
> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544
> reclaimed leaving 1070568224 used; max is 1190723584
> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752
> reclaimed leaving 1071366040 used; max is 1190723584
> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104
> reclaimed leaving 1072168400 used; max is 1190723584
> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264
> reclaimed leaving 1072743152 used; max is 1190723584
> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160
> reclaimed leaving 1073334720 used; max is 1190723584
> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152
> reclaimed leaving 1073930576 used; max is 1190723584
> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416
> reclaimed leaving 1074548160 used; max is 1190723584
> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544
> reclaimed leaving 1075122368 used; max is 1190723584
> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272
> reclaimed leaving 1075699904 used; max is 1190723584
> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552
> reclaimed leaving 1076227648 used; max is 1190723584
> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976
> reclaimed leaving 1076747648 used; max is 1190723584
> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584
> reclaimed leaving 1077152296 used; max is 1190723584
> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168
> reclaimed leaving 1077567128 used; max is 1190723584
> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704
> reclaimed leaving 1077987824 used; max is 1190723584
> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848
> reclaimed leaving 1078393448 used; max is 1190723584
> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136
> reclaimed leaving 1078708784 used; max is 1190723584
> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344
> reclaimed leaving 1079074464 used; max is 1190723584
> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992
> reclaimed leaving 1079409016 used; max is 1190723584
> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952
> reclaimed leaving 1079749672 used; max is 1190723584
> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768
> reclaimed leaving 1080014112 used; max is 1190723584
> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216
> reclaimed leaving 1080278992 used; max is 1190723584
> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912
> reclaimed leaving 1080574456 used; max is 1190723584
> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376
> reclaimed leaving 1080884096 used; max is 1190723584
> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512
> reclaimed leaving 1081076888 used; max is 1190723584
> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168
> reclaimed leaving 1081313872 used; max is 1190723584
> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168
> reclaimed leaving 1081528880 used; max is 1190723584
> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152
> reclaimed leaving 1081704648 used; max is 1190723584
> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216
> reclaimed leaving 1081889512 used; max is 1190723584
> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392
> reclaimed leaving 1082072568 used; max is 1190723584
> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376
> reclaimed leaving 1082270680 used; max is 1190723584
> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920
> reclaimed leaving 1082522872 used; max is 1190723584
> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392
> reclaimed leaving 1082632368 used; max is 1190723584
> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560
> reclaimed leaving 1082794472 used; max is 1190723584
> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024
> reclaimed leaving 1082963008 used; max is 1190723584
> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504
> reclaimed leaving 1083117048 used; max is 1190723584
> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192
> reclaimed leaving 1083213304 used; max is 1190723584
> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392
> reclaimed leaving 1083332280 used; max is 1190723584
> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808
> reclaimed leaving 1083448744 used; max is 1190723584
> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864
> reclaimed leaving 1083601864 used; max is 1190723584
> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520
> reclaimed leaving 1083680168 used; max is 1190723584
> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872
> reclaimed leaving 1083814848 used; max is 1190723584
> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648
> reclaimed leaving 1083871152 used; max is 1190723584
> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720
> reclaimed leaving 1083984728 used; max is 1190723584
> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056
> reclaimed leaving 1084063504 used; max is 1190723584
>