You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by java8964 <ja...@hotmail.com> on 2015/08/21 23:14:07 UTC
How frequently should full gc we expect
In the test job I am running in Spark 1.3.1 in our stage cluster, I can see following information on the application stage information:
MetricMin25th percentileMedian75th percentileMaxDuration0 ms1.1 min1.5 min1.7 min3.4 minGC Time11 s16 s21 s25 s54 s
>From the GC output log, I can see it is about full GC in the executor every minutes, like below.
My question is that the committed heap is more than 14G, and -XX:MaxPermSize=128m, in this case, the heap usage max is about 10G, why full GC happened every minute?
The job runs fine, just want to know what exception you guys normally have for full GC in the spark jobs?
Thanks
Yong
2015-08-21T16:53:59.561-0400: [Full GC [PSYoungGen: 328038K->0K(3728384K)] [ParOldGen: 10359817K->5856671K(11185152K)] 10687855K->5856671K(14913536K) [PSPermGen: 57214K->57214K(57856K)], 8.6951450 secs] [Times: user=140.72 sys=0.18, real=8.69 secs]
2015-08-21T16:54:09.605-0400: [GC [PSYoungGen: 1864192K->251539K(3728384K)] 7720863K->6108211K(14913536K), 0.1217750 secs] [Times: user=2.12 sys=0.01, real=0.12 secs]
2015-08-21T16:54:11.131-0400: [GC [PSYoungGen: 2115731K->163448K(3728384K)] 7972404K->6197142K(14913536K), 0.1802910 secs] [Times: user=3.19 sys=0.01, real=0.18 secs]
2015-08-21T16:54:12.832-0400: [GC [PSYoungGen: 2027640K->144369K(3728384K)] 8061339K->6314232K(14913536K), 0.1816010 secs] [Times: user=3.03 sys=0.00, real=0.19 secs]
2015-08-21T16:54:14.547-0400: [GC [PSYoungGen: 2008561K->121478K(3728384K)] 8178424K->6435609K(14913536K), 0.1411160 secs] [Times: user=2.50 sys=0.00, real=0.14 secs]
2015-08-21T16:54:15.931-0400: [GC [PSYoungGen: 1985670K->114489K(3728384K)] 8299801K->6550508K(14913536K), 0.1285300 secs] [Times: user=2.13 sys=0.00, real=0.13 secs]
2015-08-21T16:54:17.323-0400: [GC [PSYoungGen: 1978681K->219811K(3792896K)] 8414700K->6769504K(14978048K), 0.1649230 secs] [Times: user=2.89 sys=0.01, real=0.17 secs]
2015-08-21T16:54:18.878-0400: [GC [PSYoungGen: 2148515K->425173K(3728384K)] 8698218K->6974876K(14913536K), 0.3130360 secs] [Times: user=5.56 sys=0.00, real=0.31 secs]
2015-08-21T16:54:20.596-0400: [GC [PSYoungGen: 2353877K->313071K(3985408K)] 8903582K->7071556K(15170560K), 0.2423240 secs] [Times: user=4.30 sys=0.00, real=0.24 secs]
2015-08-21T16:54:22.695-0400: [GC [PSYoungGen: 2608367K->371370K(3902464K)] 9366852K->7338548K(15087616K), 0.2647510 secs] [Times: user=4.48 sys=0.00, real=0.26 secs]
2015-08-21T16:54:24.747-0400: [GC [PSYoungGen: 2666666K->459392K(4174336K)] 9633844K->7528652K(15359488K), 0.3564370 secs] [Times: user=6.36 sys=0.00, real=0.35 secs]
2015-08-21T16:54:26.951-0400: [GC [PSYoungGen: 3116160K->445880K(4075008K)] 10185420K->7746897K(15260160K), 0.2853880 secs] [Times: user=5.07 sys=0.00, real=0.29 secs]
2015-08-21T16:54:29.340-0400: [GC [PSYoungGen: 3102648K->286176K(4314112K)] 10403665K->7809242K(15499264K), 0.2534940 secs] [Times: user=4.48 sys=0.01, real=0.25 secs]
2015-08-21T16:54:31.979-0400: [GC [PSYoungGen: 3269600K->122064K(4261888K)] 10792666K->7863493K(15447040K), 0.2035800 secs] [Times: user=3.41 sys=0.00, real=0.20 secs]
2015-08-21T16:54:34.737-0400: [GC [PSYoungGen: 3105488K->555850K(4373504K)] 10846917K->8297279K(15558656K), 0.2401510 secs] [Times: user=4.14 sys=0.00, real=0.24 secs]
2015-08-21T16:54:38.015-0400: [GC [PSYoungGen: 3675978K->1146062K(4266496K)] 11417409K->8887493K(15451648K), 0.4298600 secs] [Times: user=7.65 sys=0.00, real=0.43 secs]
2015-08-21T16:54:41.492-0400: [GC [PSYoungGen: 4266190K->1326063K(3565056K)] 12007627K->9231644K(14750208K), 0.5542100 secs] [Times: user=9.90 sys=0.01, real=0.55 secs]
2015-08-21T16:54:43.797-0400: [GC [PSYoungGen: 3565039K->1587981K(3827200K)] 11470620K->9612725K(15012352K), 0.5359080 secs] [Times: user=9.57 sys=0.00, real=0.54 secs]
2015-08-21T16:54:45.856-0400: [GC [PSYoungGen: 3826957K->1047737K(3629568K)] 11851701K->9914434K(14814720K), 0.7787060 secs] [Times: user=13.91 sys=0.00, real=0.78 secs]
2015-08-21T16:54:48.174-0400: [GC [PSYoungGen: 2911929K->459808K(3728384K)] 11778626K->10058483K(14913536K), 0.5953360 secs] [Times: user=10.62 sys=0.03, real=0.60 secs]
2015-08-21T16:54:50.217-0400: [GC [PSYoungGen: 2324000K->102928K(3740160K)] 11922675K->10159967K(14925312K), 0.3191560 secs] [Times: user=5.68 sys=0.01, real=0.32 secs]
2015-08-21T16:54:51.951-0400: [GC [PSYoungGen: 1978896K->296227K(3728384K)] 12035935K->10456136K(14913536K), 0.1809970 secs] [Times: user=3.02 sys=0.00, real=0.18 secs]
2015-08-21T16:54:53.550-0400: [GC [PSYoungGen: 2172195K->316636K(3866624K)] 12332104K->10720591K(15051776K), 0.2545970 secs] [Times: user=4.43 sys=0.00, real=0.25 secs]
2015-08-21T16:54:55.340-0400: [GC [PSYoungGen: 2390748K->336907K(3800064K)] 12794703K->11043658K(14985216K), 0.3550330 secs] [Times: user=6.28 sys=0.00, real=0.35 secs]
2015-08-21T16:54:55.695-0400: [Full GC [PSYoungGen: 336907K->0K(3800064K)] [ParOldGen: 10706750K->5725402K(11185152K)] 11043658K->5725402K(14985216K) [PSPermGen: 57214K->57214K(57856K)], 9.5623960 secs] [Times: user=150.15 sys=0.28, real=9.56 secs]
2015-08-21T16:55:06.779-0400: [GC [PSYoungGen: 2074112K->397376K(4020224K)] 7799514K->6122778K(15205376K), 0.2072180 secs] [Times: user=3.65 sys=0.00, real=0.21 secs]
2015-08-21T16:55:09.161-0400: [GC [PSYoungGen: 2763840K->205647K(3938816K)] 8489242K->6268483K(15123968K), 0.2878000 secs] [Times: user=5.11 sys=0.00, real=0.29 secs]
2015-08-21T16:55:11.631-0400: [GC [PSYoungGen: 2572111K->178593K(4110848K)] 8634947K->6446009K(15296000K), 0.1981990 secs] [Times: user=3.51 sys=0.00, real=0.20 secs]
2015-08-21T16:55:14.038-0400: [GC [PSYoungGen: 2762145K->420788K(4065280K)] 9029561K->6688205K(15250432K), 0.1603690 secs] [Times: user=2.84 sys=0.00, real=0.16 secs]
2015-08-21T16:55:16.608-0400: [GC [PSYoungGen: 3004340K->713513K(4204032K)] 9271757K->6980930K(15389184K), 0.2504660 secs] [Times: user=4.45 sys=0.00, real=0.25 secs]
2015-08-21T16:55:19.539-0400: [GC [PSYoungGen: 3526441K->1071565K(3884544K)] 9793858K->7338983K(15069696K), 0.5002750 secs] [Times: user=8.93 sys=0.00, real=0.50 secs]
2015-08-21T16:55:22.942-0400: [GC [PSYoungGen: 3884493K->1386667K(3655168K)] 10151912K->7654086K(14840320K), 0.4753120 secs] [Times: user=8.48 sys=0.00, real=0.48 secs]
2015-08-21T16:55:25.740-0400: [GC [PSYoungGen: 3529387K->1573464K(3716608K)] 9796806K->7840883K(14901760K), 0.5294890 secs] [Times: user=9.46 sys=0.00, real=0.53 secs]
2015-08-21T16:55:28.230-0400: [GC [PSYoungGen: 3716184K->1408775K(3728384K)] 9983603K->8016021K(14913536K), 0.6090130 secs] [Times: user=10.59 sys=0.00, real=0.61 secs]
2015-08-21T16:55:30.115-0400: [GC [PSYoungGen: 3272967K->879217K(3728384K)] 9880213K->8118285K(14913536K), 0.6053810 secs] [Times: user=10.73 sys=0.01, real=0.60 secs]
2015-08-21T16:55:32.029-0400: [GC [PSYoungGen: 2743409K->416860K(3728384K)] 9982477K->8203931K(14913536K), 0.5133040 secs] [Times: user=9.17 sys=0.00, real=0.51 secs]
2015-08-21T16:55:33.871-0400: [GC [PSYoungGen: 2281052K->160473K(3728384K)] 10068123K->8281580K(14913536K), 0.3045040 secs] [Times: user=5.42 sys=0.00, real=0.31 secs]
2015-08-21T16:55:35.575-0400: [GC [PSYoungGen: 2024665K->63592K(3728384K)] 10145772K->8345118K(14913536K), 0.1367630 secs] [Times: user=2.41 sys=0.00, real=0.14 secs]
2015-08-21T16:55:37.049-0400: [GC [PSYoungGen: 1927784K->59174K(3728384K)] 10209310K->8404017K(14913536K), 0.1407080 secs] [Times: user=2.48 sys=0.00, real=0.14 secs]
2015-08-21T16:55:38.600-0400: [GC [PSYoungGen: 1923366K->57465K(3748352K)] 10268209K->8461019K(14933504K), 0.1109560 secs] [Times: user=1.93 sys=0.01, real=0.11 secs]
2015-08-21T16:55:40.411-0400: [GC [PSYoungGen: 1941625K->73353K(3728384K)] 10345179K->8528297K(14913536K), 0.1695660 secs] [Times: user=2.92 sys=0.00, real=0.17 secs]
2015-08-21T16:55:42.128-0400: [GC [PSYoungGen: 1957513K->126635K(3871232K)] 10412457K->8581579K(15056384K), 0.1535930 secs] [Times: user=2.71 sys=0.00, real=0.15 secs]
2015-08-21T16:55:43.705-0400: [GC [PSYoungGen: 2206891K->189292K(3801600K)] 10661835K->8644236K(14986752K), 0.1850870 secs] [Times: user=3.28 sys=0.00, real=0.19 secs]
2015-08-21T16:55:45.390-0400: [GC [PSYoungGen: 2260426K->307881K(4056064K)] 10715370K->8762825K(15241216K), 0.2177180 secs] [Times: user=3.84 sys=0.00, real=0.22 secs]
2015-08-21T16:55:47.530-0400: [GC [PSYoungGen: 2725545K->684173K(3954176K)] 11180489K->9139117K(15139328K), 0.3418670 secs] [Times: user=6.02 sys=0.05, real=0.34 secs]
2015-08-21T16:55:49.983-0400: [GC [PSYoungGen: 3101837K->1162872K(3749376K)] 11556781K->9617816K(14934528K), 0.4373140 secs] [Times: user=7.79 sys=0.00, real=0.44 secs]
2015-08-21T16:55:52.563-0400: [GC [PSYoungGen: 3387512K->1596897K(3821568K)] 11842456K->10121078K(15006720K), 0.5456530 secs] [Times: user=9.73 sys=0.01, real=0.55 secs]
2015-08-21T16:55:55.461-0400: [GC [PSYoungGen: 3821537K->1770995K(3635200K)] 12345718K->10509011K(14820352K), 0.6742280 secs] [Times: user=12.04 sys=0.00, real=0.67 secs]
2015-08-21T16:55:58.088-0400: [GC [PSYoungGen: 3635187K->1560141K(3728384K)] 12373203K->10723772K(14913536K), 0.6732890 secs] [Times: user=11.98 sys=0.01, real=0.68 secs]
2015-08-21T16:56:01.357-0400: [GC [PSYoungGen: 3424333K->815806K(3728384K)] 12587964K->10923015K(14913536K), 0.7893440 secs] [Times: user=14.07 sys=0.01, real=0.79 secs]
2015-08-21T16:56:04.146-0400: [GC [PSYoungGen: 2679998K->364642K(3728384K)] 12787207K->11090639K(14913536K), 0.5218630 secs] [Times: user=9.30 sys=0.01, real=0.52 secs]
2015-08-21T16:56:04.668-0400: [Full GC [PSYoungGen: 364642K->0K(3728384K)] [ParOldGen: 10725997K->6388280K(11185152K)] 11090639K->6388280K(14913536K) [PSPermGen: 57214K->57214K(57856K)], 9.5366890 secs] [Times: user=141.02 sys=0.02, real=9.54 secs]
2015-08-21T16:56:15.906-0400: [GC [PSYoungGen: 1864192K->244290K(3728384K)] 8252472K->6632570K(14913536K), 0.1593280 secs] [Times: user=2.77 sys=0.00, real=0.16 secs]
2015-08-21T16:56:17.697-0400: [GC [PSYoungGen: 2108482K->271776K(3728384K)] 8496762K->6904251K(14913536K), 0.2788640 secs] [Times: user=4.90 sys=0.03, real=0.28 secs]
2015-08-21T16:56:19.662-0400: [GC [PSYoungGen: 2135968K->340850K(3728384K)] 8768443K->7244018K(14913536K), 0.3142780 secs] [Times: user=5.51 sys=0.02, real=0.31 secs]
2015-08-21T16:56:21.646-0400: [GC [PSYoungGen: 2200652K->420754K(3728384K)] 9103820K->7605015K(14913536K), 0.3844990 secs] [Times: user=6.78 sys=0.00, real=0.38 secs]
2015-08-21T16:56:23.995-0400: [GC [PSYoungGen: 2284946K->283289K(3728384K)] 9469207K->7851835K(14913536K), 0.2670850 secs] [Times: user=4.71 sys=0.00, real=0.26 secs]
2015-08-21T16:56:26.054-0400: [GC [PSYoungGen: 2147384K->285799K(3728384K)] 9715930K->8114308K(14913536K), 0.3649000 secs] [Times: user=6.45 sys=0.01, real=0.36 secs]
2015-08-21T16:56:28.386-0400: [GC [PSYoungGen: 2149991K->505992K(3910144K)] 9978500K->8612203K(15095296K), 0.3867880 secs] [Times: user=6.84 sys=0.01, real=0.38 secs]
2015-08-21T16:56:30.799-0400: [GC [PSYoungGen: 2620552K->454232K(3796992K)] 10726763K->9026057K(14982144K), 0.4349370 secs] [Times: user=7.71 sys=0.01, real=0.44 secs]
2015-08-21T16:56:33.271-0400: [GC [PSYoungGen: 2568792K->385079K(4081152K)] 11140619K->9382690K(15266304K), 0.3980990 secs] [Times: user=7.03 sys=0.00, real=0.40 secs]
2015-08-21T16:56:36.305-0400: [GC [PSYoungGen: 2878519K->259616K(4004864K)] 11876136K->9641767K(15190016K), 0.4366160 secs] [Times: user=7.78 sys=0.00, real=0.43 secs]
2015-08-21T16:56:39.004-0400: [GC [PSYoungGen: 2753056K->176304K(4172800K)] 12135207K->9817707K(15357952K), 0.2463590 secs] [Times: user=4.37 sys=0.00, real=0.25 secs]
2015-08-21T16:56:41.525-0400: [GC [PSYoungGen: 2889392K->175585K(4132864K)] 12530795K->9993292K(15318016K), 0.1842920 secs] [Times: user=3.25 sys=0.00, real=0.19 secs]
2015-08-21T16:56:44.508-0400: [GC [PSYoungGen: 2888673K->394614K(4302336K)] 12706380K->10212322K(15487488K), 0.1747460 secs] [Times: user=3.07 sys=0.00, real=0.17 secs]
2015-08-21T16:56:47.239-0400: [GC [PSYoungGen: 3321718K->560043K(4217344K)] 13139426K->10377750K(15402496K), 0.2171460 secs] [Times: user=3.83 sys=0.01, real=0.22 secs]
2015-08-21T16:56:49.944-0400: [GC [PSYoungGen: 3487147K->706210K(4312576K)] 13304854K->10523918K(15497728K), 0.2627380 secs] [Times: user=4.62 sys=0.00, real=0.27 secs]
2015-08-21T16:56:53.378-0400: [GC [PSYoungGen: 3768482K->1088212K(4150784K)] 13586190K->10905920K(15335936K), 0.4293970 secs] [Times: user=7.59 sys=0.00, real=0.43 secs]
2015-08-21T16:56:56.477-0400: [GC [PSYoungGen: 4150484K->1309420K(3783168K)] 13968192K->11127127K(14968320K), 0.4637440 secs] [Times: user=8.23 sys=0.00, real=0.47 secs]
2015-08-21T16:56:58.830-0400: [GC [PSYoungGen: 3679468K->1498621K(3868672K)] 13497176K->11316329K(15053824K), 0.5533130 secs] [Times: user=9.84 sys=0.01, real=0.55 secs]
2015-08-21T16:57:01.296-0400: [GC [PSYoungGen: 3868669K->1474972K(3588096K)] 13686378K->11460991K(14773248K), 0.6170960 secs] [Times: user=10.76 sys=0.02, real=0.62 secs]
2015-08-21T16:57:03.201-0400: [GC [PSYoungGen: 3339164K->1323407K(3728384K)] 13325183K->11631637K(14913536K), 0.6002320 secs] [Times: user=10.66 sys=0.01, real=0.60 secs]
2015-08-21T16:57:05.243-0400: [GC [PSYoungGen: 3187599K->894715K(3728384K)] 13495829K->11696649K(14913536K), 0.5563990 secs] [Times: user=9.85 sys=0.00, real=0.56 secs]
2015-08-21T16:57:05.799-0400: [Full GC [PSYoungGen: 894715K->0K(3728384K)] [ParOldGen: 10801933K->4817827K(11185152K)] 11696649K->4817827K(14913536K) [PSPermGen: 57214K->57214K(57856K)], 9.0502760 secs] [Times: user=141.39 sys=0.16, real=9.04 secs]
2015-08-21T16:57:16.259-0400: [GC [PSYoungGen: 1864192K->61920K(3728384K)] 6682027K->4879755K(14913536K), 0.0411120 secs] [Times: user=0.68 sys=0.00, real=0.04 secs]
2015-08-21T16:57:17.975-0400: [GC [PSYoungGen: 1926112K->115664K(3728384K)] 6743952K->4933504K(14913536K), 0.0570010 secs] [Times: user=0.97 sys=0.00, real=0.06 secs]
2015-08-21T16:57:19.769-0400: [GC [PSYoungGen: 1979856K->154159K(3728384K)] 6797696K->5032420K(14913536K), 0.0877050 secs] [Times: user=1.47 sys=0.00, real=0.09 secs]
2015-08-21T16:57:21.357-0400: [GC [PSYoungGen: 2018351K->209606K(3728384K)] 6896612K->5231255K(14913536K), 0.1261810 secs] [Times: user=2.04 sys=0.01, real=0.13 secs]
2015-08-21T16:57:22.845-0400: [GC [PSYoungGen: 2073798K->134065K(3728384K)] 7095447K->5341302K(14913536K), 0.1327730 secs] [Times: user=2.30 sys=0.00, real=0.13 secs]
2015-08-21T16:57:24.396-0400: [GC [PSYoungGen: 1998257K->439848K(3857408K)] 7205495K->5647085K(15042560K), 0.1896380 secs] [Times: user=3.26 sys=0.01, real=0.19 secs]
2015-08-21T16:57:26.047-0400: [GC [PSYoungGen: 2452008K->805703K(3747328K)] 7659247K->6012942K(14932480K), 0.3043400 secs] [Times: user=5.32 sys=0.01, real=0.30 secs]
2015-08-21T16:57:27.867-0400: [GC [PSYoungGen: 2817863K->1137456K(3686400K)] 8025102K->6344695K(14871552K), 0.4460680 secs] [Times: user=7.69 sys=0.02, real=0.45 secs]
2015-08-21T16:57:30.101-0400: [GC [PSYoungGen: 3110192K->1284856K(3257856K)] 8317431K->6608870K(14443008K), 0.5406400 secs] [Times: user=9.49 sys=0.01, real=0.54 secs]
2015-08-21T16:57:32.584-0400: [GC [PSYoungGen: 3257592K->875015K(3728384K)] 8581606K->6829064K(14913536K), 0.6183670 secs] [Times: user=10.99 sys=0.00, real=0.61 secs]
2015-08-21T16:57:35.140-0400: [GC [PSYoungGen: 2739207K->413106K(3728384K)] 8693256K->7016035K(14913536K), 0.6108230 secs] [Times: user=10.89 sys=0.00, real=0.61 secs]
2015-08-21T16:57:37.420-0400: [GC [PSYoungGen: 2277298K->143538K(3802112K)] 8880227K->7160429K(14987264K), 0.3580630 secs] [Times: user=6.35 sys=0.01, real=0.36 secs]
2015-08-21T16:57:39.480-0400: [GC [PSYoungGen: 2133170K->162775K(3780096K)] 9150061K->7322715K(14965248K), 0.1566920 secs] [Times: user=2.75 sys=0.00, real=0.16 secs]
2015-08-21T16:57:41.162-0400: [GC [PSYoungGen: 2152407K->146880K(3883520K)] 9312347K->7469656K(15068672K), 0.2068290 secs] [Times: user=3.65 sys=0.01, real=0.21 secs]
2015-08-21T16:57:42.985-0400: [GC [PSYoungGen: 2277312K->167519K(3839488K)] 9600088K->7637258K(15024640K), 0.1738150 secs] [Times: user=2.90 sys=0.00, real=0.17 secs]
2015-08-21T16:57:44.896-0400: [GC [PSYoungGen: 2297951K->289764K(4017664K)] 9767690K->7917573K(15202816K), 0.1975750 secs] [Times: user=3.44 sys=0.00, real=0.19 secs]
2015-08-21T16:57:47.206-0400: [GC [PSYoungGen: 2652644K->726301K(3937792K)] 10280453K->8354109K(15122944K), 0.3095890 secs] [Times: user=5.43 sys=0.01, real=0.31 secs]
2015-08-21T16:57:49.515-0400: [GC [PSYoungGen: 3089181K->1279912K(3672064K)] 10716989K->8907720K(14857216K), 0.6163300 secs] [Times: user=10.90 sys=0.01, real=0.62 secs]
2015-08-21T16:57:52.077-0400: [GC [PSYoungGen: 3394984K->1624385K(3739648K)] 11022792K->9464496K(14924800K), 0.7417600 secs] [Times: user=13.21 sys=0.01, real=0.74 secs]
2015-08-21T16:57:54.733-0400: [GC [PSYoungGen: 3739457K->1053135K(3717120K)] 11579568K->9905295K(14902272K), 0.9749830 secs] [Times: user=17.42 sys=0.01, real=0.98 secs]
2015-08-21T16:57:57.333-0400: [GC [PSYoungGen: 2917327K->276431K(3728384K)] 11769487K->10181125K(14913536K), 0.9417270 secs] [Times: user=16.79 sys=0.02, real=0.94 secs]
2015-08-21T16:57:59.836-0400: [GC [PSYoungGen: 2140623K->184074K(3728384K)] 12045405K->10355736K(14913536K), 0.2475730 secs] [Times: user=4.29 sys=0.00, real=0.25 secs]
2015-08-21T16:58:00.084-0400: [Full GC [PSYoungGen: 184074K->0K(3728384K)] [ParOldGen: 10171661K->5615593K(11185152K)] 10355736K->5615593K(14913536K) [PSPermGen: 57214K->57214K(57856K)], 8.3282690 secs] [Times: user=131.68 sys=0.03, real=8.33 secs]
2015-08-21T16:58:09.925-0400: [GC [PSYoungGen: 1864192K->180249K(3728384K)] 7479785K->5795842K(14913536K), 0.1359660 secs] [Times: user=2.20 sys=0.01, real=0.14 secs]
2015-08-21T16:58:11.552-0400: [GC [PSYoungGen: 2044441K->341754K(3816448K)] 7660034K->6134326K(15001600K), 0.2191480 secs] [Times: user=3.84 sys=0.00, real=0.22 secs]
2015-08-21T16:58:13.302-0400: [GC [PSYoungGen: 2295546K->422890K(3729920K)] 8088119K->6514065K(14915072K), 0.3308370 secs] [Times: user=5.83 sys=0.00, real=0.33 secs]
2015-08-21T16:58:15.171-0400: [GC [PSYoungGen: 2376682K->359260K(3996672K)] 8467863K->6838496K(15181824K), 0.4105890 secs] [Times: user=7.31 sys=0.00, real=0.41 secs]
2015-08-21T16:58:17.623-0400: [GC [PSYoungGen: 2678620K->372369K(3915264K)] 9157860K->7204743K(15100416K), 0.4238980 secs] [Times: user=7.53 sys=0.00, real=0.42 secs]
2015-08-21T16:58:20.251-0400: [GC [PSYoungGen: 2691729K->245265K(4132352K)] 9524108K->7450394K(15317504K), 0.3534780 secs] [Times: user=6.26 sys=0.00, real=0.35 secs]
2015-08-21T16:58:23.493-0400: [GC [PSYoungGen: 2862097K->233746K(4077056K)] 10067226K->7683288K(15262208K), 0.2583880 secs] [Times: user=4.57 sys=0.01, real=0.26 secs]
2015-08-21T16:58:26.355-0400: [GC [PSYoungGen: 2850578K->301387K(4254720K)] 10300120K->7979146K(15439872K), 0.3885990 secs] [Times: user=6.78 sys=0.01, real=0.39 secs]
2015-08-21T16:58:29.431-0400: [GC [PSYoungGen: 3150667K->541461K(4187136K)] 10828426K->8474442K(15372288K), 0.4318220 secs] [Times: user=7.52 sys=0.01, real=0.44 secs]
2015-08-21T16:58:32.233-0400: [GC [PSYoungGen: 3390741K->399554K(4381696K)] 11323722K->8841488K(15566848K), 0.4823410 secs] [Times: user=8.37 sys=0.00, real=0.48 secs]
2015-08-21T16:58:35.463-0400: [GC [PSYoungGen: 3496642K->214369K(4307968K)] 11938576K->9056524K(15493120K), 0.3941770 secs] [Times: user=6.75 sys=0.01, real=0.39 secs]
2015-08-21T16:58:39.364-0400: [GC [PSYoungGen: 3311457K->229978K(4460032K)] 12153612K->9284940K(15645184K), 0.2562130 secs] [Times: user=4.49 sys=0.00, real=0.25 secs]
2015-08-21T16:58:43.207-0400: [GC [PSYoungGen: 3515482K->388309K(4418048K)] 12570445K->9443272K(15603200K), 0.1612730 secs] [Times: user=2.81 sys=0.00, real=0.16 secs]
2015-08-21T16:58:46.307-0400: [GC [PSYoungGen: 3673813K->470089K(4571136K)] 12728782K->9525058K(15756288K), 0.1758880 secs] [Times: user=2.92 sys=0.01, real=0.17 secs]
2015-08-21T16:58:49.392-0400: [GC [PSYoungGen: 3978313K->917141K(4425728K)] 13033282K->9972110K(15610880K), 0.3321180 secs] [Times: user=5.87 sys=0.00, real=0.34 secs]
2015-08-21T16:58:53.158-0400: [GC [PSYoungGen: 4425365K->1057340K(4140544K)] 13480334K->10112309K(15325696K), 0.4089550 secs] [Times: user=7.28 sys=0.00, real=0.41 secs]
2015-08-21T16:58:55.894-0400: [GC [PSYoungGen: 4049980K->1299958K(4292608K)] 13104949K->10363575K(15477760K), 0.4509200 secs] [Times: user=8.03 sys=0.00, real=0.45 secs]
2015-08-21T16:58:58.723-0400: [GC [PSYoungGen: 4292598K->1070291K(3699712K)] 13356215K->10474715K(14884864K), 0.5783500 secs] [Times: user=10.28 sys=0.01, real=0.58 secs]
2015-08-21T16:59:01.146-0400: [GC [PSYoungGen: 3470035K->1136931K(3536896K)] 12874459K->10541355K(14722048K), 0.4320870 secs] [Times: user=7.61 sys=0.01, real=0.44 secs]
2015-08-21T16:59:03.597-0400: [GC [PSYoungGen: 3536675K->1093619K(3812352K)] 12941099K->10594202K(14997504K), 0.4521770 secs] [Times: user=7.99 sys=0.00, real=0.45 secs]
2015-08-21T16:59:05.946-0400: [GC [PSYoungGen: 3223027K->699072K(3860992K)] 12723610K->10634428K(15046144K), 0.5153210 secs] [Times: user=9.14 sys=0.00, real=0.52 secs]
2015-08-21T16:59:08.228-0400: [GC [PSYoungGen: 2828480K->315546K(3974656K)] 12763836K->10671299K(15159808K), 0.4134510 secs] [Times: user=7.24 sys=0.00, real=0.41 secs]
2015-08-21T16:59:10.256-0400: [GC [PSYoungGen: 2652314K->169594K(3954688K)] 13008067K->10709291K(15139840K), 0.1790620 secs] [Times: user=3.12 sys=0.00, real=0.18 secs]
2015-08-21T16:59:10.436-0400: [Full GC [PSYoungGen: 169594K->0K(3954688K)] [ParOldGen: 10539697K->5005889K(11185152K)] 10709291K->5005889K(15139840K) [PSPermGen: 57215K->57215K(57344K)], 8.6283890 secs] [Times: user=139.03 sys=0.18, real=8.63 secs]
2015-08-21T16:59:20.551-0400: [GC [PSYoungGen: 2336768K->57691K(3976704K)] 7342657K->5063580K(15161856K), 0.0320200 secs] [Times: user=0.50 sys=0.00, real=0.03 secs]
2015-08-21T16:59:22.141-0400: [GC [PSYoungGen: 2420571K->103430K(3977728K)] 7426460K->5109319K(15162880K), 0.0542670 secs] [Times: user=0.89 sys=0.00, real=0.05 secs]
2015-08-21T16:59:24.166-0400: [GC [PSYoungGen: 2466310K->149395K(4040704K)] 7472199K->5155285K(15225856K), 0.0890410 secs] [Times: user=1.44 sys=0.01, real=0.09 secs]
2015-08-21T16:59:25.878-0400: [GC [PSYoungGen: 2597779K->184081K(4000256K)] 7603669K->5189970K(15185408K), 0.0951090 secs] [Times: user=1.51 sys=0.01, real=0.10 secs]
2015-08-21T16:59:27.641-0400: [GC [PSYoungGen: 2632465K->215665K(4155392K)] 7638354K->5221554K(15340544K), 0.1347880 secs] [Times: user=2.24 sys=0.00, real=0.13 secs]
2015-08-21T16:59:30.014-0400: [GC [PSYoungGen: 2872433K->249168K(4093952K)] 7878322K->5255058K(15279104K), 0.1940370 secs] [Times: user=3.26 sys=0.01, real=0.19 secs]
2015-08-21T16:59:32.061-0400: [GC [PSYoungGen: 2905936K->289950K(4296192K)] 7911834K->5295847K(15481344K), 0.1987530 secs] [Times: user=3.32 sys=0.00, real=0.20 secs]
2015-08-21T16:59:34.387-0400: [GC [PSYoungGen: 3216542K->333450K(4222976K)] 8222445K->5339353K(15408128K), 0.2298800 secs] [Times: user=3.90 sys=0.00, real=0.23 secs]
2015-08-21T16:59:37.098-0400: [GC [PSYoungGen: 3260042K->366870K(4449280K)] 8265945K->5372773K(15634432K), 0.2280860 secs] [Times: user=3.86 sys=0.01, real=0.23 secs]
2015-08-21T16:59:39.947-0400: [GC [PSYoungGen: 3596054K->471644K(4372480K)] 8601957K->5477547K(15557632K), 0.2958990 secs] [Times: user=4.93 sys=0.01, real=0.30 secs]
2015-08-21T16:59:42.747-0400: [GC [PSYoungGen: 3700828K->812416K(4332032K)] 8706737K->5818325K(15517184K), 0.3462400 secs] [Times: user=6.09 sys=0.01, real=0.35 secs]
2015-08-21T16:59:45.565-0400: [GC [PSYoungGen: 4047744K->1178596K(4413952K)] 9053653K->6259289K(15599104K), 0.5043450 secs] [Times: user=8.91 sys=0.01, real=0.50 secs]
2015-08-21T16:59:48.822-0400: [GC [PSYoungGen: 4413924K->1178592K(3244544K)] 9494617K->6813581K(14429696K), 0.9950540 secs] [Times: user=17.66 sys=0.03, real=1.00 secs]
2015-08-21T16:59:51.701-0400: [GC [PSYoungGen: 3244512K->1401555K(3467776K)] 8879501K->7083031K(14652928K), 0.4898300 secs] [Times: user=8.73 sys=0.00, real=0.49 secs]
2015-08-21T16:59:54.086-0400: [GC [PSYoungGen: 3467475K->1517388K(3728384K)] 9148951K->7271321K(14913536K), 0.5393400 secs] [Times: user=9.61 sys=0.00, real=0.54 secs]
2015-08-21T16:59:56.357-0400: [GC [PSYoungGen: 3381580K->1588229K(3728384K)] 9135513K->7409023K(14913536K), 0.5366240 secs] [Times: user=9.56 sys=0.00, real=0.53 secs]
2015-08-21T16:59:58.894-0400: [GC [PSYoungGen: 3452421K->1639924K(3728384K)] 9273215K->7548125K(14913536K), 0.5923160 secs] [Times: user=10.53 sys=0.01, real=0.60 secs]
2015-08-21T17:00:01.290-0400: [GC [PSYoungGen: 3504116K->1679530K(3728384K)] 9412412K->7666735K(14913536K), 0.5578720 secs] [Times: user=9.92 sys=0.00, real=0.56 secs]
2015-08-21T17:00:03.446-0400: [GC [PSYoungGen: 3543722K->1711705K(3728384K)] 9530930K->7867994K(14913536K), 0.5663970 secs] [Times: user=9.98 sys=0.00, real=0.56 secs]
2015-08-21T17:00:06.183-0400: [GC [PSYoungGen: 3575897K->1537411K(3728384K)] 9732186K->8248661K(14913536K), 0.7580860 secs] [Times: user=13.49 sys=0.01, real=0.76 secs]
2015-08-21T17:00:09.207-0400: [GC [PSYoungGen: 3401603K->1294833K(3728384K)] 10112853K->8493004K(14913536K), 0.6302370 secs] [Times: user=11.21 sys=0.04, real=0.63 secs]
2015-08-21T17:00:11.812-0400: [GC [PSYoungGen: 3159025K->1230822K(3728384K)] 10357196K->8696289K(14913536K), 0.5573280 secs] [Times: user=9.50 sys=0.01, real=0.56 secs]
2015-08-21T17:00:14.651-0400: [GC [PSYoungGen: 3095014K->1358347K(3728384K)] 10560481K->9101611K(14913536K), 0.6030070 secs] [Times: user=10.69 sys=0.01, real=0.60 secs]
2015-08-21T17:00:17.101-0400: [GC [PSYoungGen: 3222539K->899612K(3728384K)] 10965809K->9335622K(14913536K), 0.6895790 secs] [Times: user=12.27 sys=0.00, real=0.69 secs]
2015-08-21T17:00:19.795-0400: [GC [PSYoungGen: 2763804K->489281K(3728384K)] 11199816K->9541824K(14913536K), 0.5745470 secs] [Times: user=10.12 sys=0.01, real=0.58 secs]
2015-08-21T17:00:22.501-0400: [GC [PSYoungGen: 2353473K->218299K(3728384K)] 11406016K->9719479K(14913536K), 0.4245700 secs] [Times: user=7.56 sys=0.00, real=0.42 secs]
2015-08-21T17:00:24.838-0400: [GC [PSYoungGen: 2082491K->219232K(3728384K)] 11583671K->9930182K(14913536K), 0.2530360 secs] [Times: user=4.46 sys=0.00, real=0.26 secs]
2015-08-21T17:00:26.859-0400: [GC [PSYoungGen: 2083424K->169198K(3728384K)] 11794375K->10099702K(14913536K), 0.2721990 secs] [Times: user=4.82 sys=0.00, real=0.28 secs]
2015-08-21T17:00:28.939-0400: [GC [PSYoungGen: 2033390K->128783K(3728384K)] 11963894K->10228066K(14913536K), 0.1527300 secs] [Times: user=2.69 sys=0.00, real=0.16 secs]
2015-08-21T17:00:30.706-0400: [GC [PSYoungGen: 1992975K->107792K(3728384K)] 12092258K->10335337K(14913536K), 0.2070530 secs] [Times: user=3.67 sys=0.01, real=0.21 secs]
2015-08-21T17:00:32.644-0400: [GC [PSYoungGen: 1971984K->80581K(3728384K)] 12199529K->10415648K(14913536K), 0.1507680 secs] [Times: user=2.66 sys=0.00, real=0.15 secs]
2015-08-21T17:00:34.445-0400: [GC [PSYoungGen: 1944773K->69161K(3728384K)] 12279840K->10484490K(14913536K), 0.1355530 secs] [Times: user=2.37 sys=0.00, real=0.14 secs]
2015-08-21T17:00:36.051-0400: [GC [PSYoungGen: 1933353K->152599K(3728384K)] 12348682K->10636601K(14913536K), 0.1194720 secs] [Times: user=2.05 sys=0.00, real=0.11 secs]
2015-08-21T17:00:37.650-0400: [GC [PSYoungGen: 2016791K->301922K(3728384K)] 12500793K->10907863K(14913536K), 0.1948420 secs] [Times: user=3.23 sys=0.00, real=0.20 secs]
2015-08-21T17:00:37.845-0400: [Full GC [PSYoungGen: 301922K->0K(3728384K)] [ParOldGen: 10605941K->4977253K(11185152K)] 10907863K->4977253K(14913536K) [PSPermGen: 57215K->57215K(57344K)], 8.7096570 secs] [Times: user=136.17 sys=0.20, real=8.71 secs]
2015-08-21T17:00:47.940-0400: [GC [PSYoungGen: 1864192K->295008K(3740160K)] 6841445K->5272261K(14925312K), 0.2179710 secs] [Times: user=3.78 sys=0.00, real=0.22 secs]
Re: How frequently should full gc we expect
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Used to hit GC: Overhead limit exceeded and sometime GC: Heap error too.
Thanks
Best Regards
On Sat, Aug 22, 2015 at 2:44 AM, java8964 <ja...@hotmail.com> wrote:
> In the test job I am running in Spark 1.3.1 in our stage cluster, I can
> see following information on the application stage information:
>
> MetricMin25th percentileMedian75th percentileMaxDuration0 ms1.1 min1.5 min1.7
> min3.4 minGC Time11 s16 s21 s25 s54 s
>
> From the GC output log, I can see it is about full GC in the executor
> every minutes, like below.
>
> My question is that the committed heap is more than 14G, and
> -XX:MaxPermSize=128m, in this case, the heap usage max is about 10G, why
> full GC happened every minute?
>
> The job runs fine, just want to know what exception you guys normally have
> for full GC in the spark jobs?
>
> Thanks
>
> Yong
>
> 2015-08-21T16:53:59.561-0400: [Full GC [PSYoungGen: 328038K->0K(3728384K)] [ParOldGen: 10359817K->5856671K(11185152K)] 10687855K->5856671K(14913536K) [PSPermGen: 57214K->57214K(57856K)], 8.6951450 secs] [Times: user=140.72 sys=0.18, real=8.69 secs]
> 2015-08-21T16:54:09.605-0400: [GC [PSYoungGen: 1864192K->251539K(3728384K)] 7720863K->6108211K(14913536K), 0.1217750 secs] [Times: user=2.12 sys=0.01, real=0.12 secs]
> 2015-08-21T16:54:11.131-0400: [GC [PSYoungGen: 2115731K->163448K(3728384K)] 7972404K->6197142K(14913536K), 0.1802910 secs] [Times: user=3.19 sys=0.01, real=0.18 secs]
> 2015-08-21T16:54:12.832-0400: [GC [PSYoungGen: 2027640K->144369K(3728384K)] 8061339K->6314232K(14913536K), 0.1816010 secs] [Times: user=3.03 sys=0.00, real=0.19 secs]
> 2015-08-21T16:54:14.547-0400: [GC [PSYoungGen: 2008561K->121478K(3728384K)] 8178424K->6435609K(14913536K), 0.1411160 secs] [Times: user=2.50 sys=0.00, real=0.14 secs]
> 2015-08-21T16:54:15.931-0400: [GC [PSYoungGen: 1985670K->114489K(3728384K)] 8299801K->6550508K(14913536K), 0.1285300 secs] [Times: user=2.13 sys=0.00, real=0.13 secs]
> 2015-08-21T16:54:17.323-0400: [GC [PSYoungGen: 1978681K->219811K(3792896K)] 8414700K->6769504K(14978048K), 0.1649230 secs] [Times: user=2.89 sys=0.01, real=0.17 secs]
> 2015-08-21T16:54:18.878-0400: [GC [PSYoungGen: 2148515K->425173K(3728384K)] 8698218K->6974876K(14913536K), 0.3130360 secs] [Times: user=5.56 sys=0.00, real=0.31 secs]
> 2015-08-21T16:54:20.596-0400: [GC [PSYoungGen: 2353877K->313071K(3985408K)] 8903582K->7071556K(15170560K), 0.2423240 secs] [Times: user=4.30 sys=0.00, real=0.24 secs]
> 2015-08-21T16:54:22.695-0400: [GC [PSYoungGen: 2608367K->371370K(3902464K)] 9366852K->7338548K(15087616K), 0.2647510 secs] [Times: user=4.48 sys=0.00, real=0.26 secs]
> 2015-08-21T16:54:24.747-0400: [GC [PSYoungGen: 2666666K->459392K(4174336K)] 9633844K->7528652K(15359488K), 0.3564370 secs] [Times: user=6.36 sys=0.00, real=0.35 secs]
> 2015-08-21T16:54:26.951-0400: [GC [PSYoungGen: 3116160K->445880K(4075008K)] 10185420K->7746897K(15260160K), 0.2853880 secs] [Times: user=5.07 sys=0.00, real=0.29 secs]
> 2015-08-21T16:54:29.340-0400: [GC [PSYoungGen: 3102648K->286176K(4314112K)] 10403665K->7809242K(15499264K), 0.2534940 secs] [Times: user=4.48 sys=0.01, real=0.25 secs]
> 2015-08-21T16:54:31.979-0400: [GC [PSYoungGen: 3269600K->122064K(4261888K)] 10792666K->7863493K(15447040K), 0.2035800 secs] [Times: user=3.41 sys=0.00, real=0.20 secs]
> 2015-08-21T16:54:34.737-0400: [GC [PSYoungGen: 3105488K->555850K(4373504K)] 10846917K->8297279K(15558656K), 0.2401510 secs] [Times: user=4.14 sys=0.00, real=0.24 secs]
> 2015-08-21T16:54:38.015-0400: [GC [PSYoungGen: 3675978K->1146062K(4266496K)] 11417409K->8887493K(15451648K), 0.4298600 secs] [Times: user=7.65 sys=0.00, real=0.43 secs]
> 2015-08-21T16:54:41.492-0400: [GC [PSYoungGen: 4266190K->1326063K(3565056K)] 12007627K->9231644K(14750208K), 0.5542100 secs] [Times: user=9.90 sys=0.01, real=0.55 secs]
> 2015-08-21T16:54:43.797-0400: [GC [PSYoungGen: 3565039K->1587981K(3827200K)] 11470620K->9612725K(15012352K), 0.5359080 secs] [Times: user=9.57 sys=0.00, real=0.54 secs]
> 2015-08-21T16:54:45.856-0400: [GC [PSYoungGen: 3826957K->1047737K(3629568K)] 11851701K->9914434K(14814720K), 0.7787060 secs] [Times: user=13.91 sys=0.00, real=0.78 secs]
> 2015-08-21T16:54:48.174-0400: [GC [PSYoungGen: 2911929K->459808K(3728384K)] 11778626K->10058483K(14913536K), 0.5953360 secs] [Times: user=10.62 sys=0.03, real=0.60 secs]
> 2015-08-21T16:54:50.217-0400: [GC [PSYoungGen: 2324000K->102928K(3740160K)] 11922675K->10159967K(14925312K), 0.3191560 secs] [Times: user=5.68 sys=0.01, real=0.32 secs]
> 2015-08-21T16:54:51.951-0400: [GC [PSYoungGen: 1978896K->296227K(3728384K)] 12035935K->10456136K(14913536K), 0.1809970 secs] [Times: user=3.02 sys=0.00, real=0.18 secs]
> 2015-08-21T16:54:53.550-0400: [GC [PSYoungGen: 2172195K->316636K(3866624K)] 12332104K->10720591K(15051776K), 0.2545970 secs] [Times: user=4.43 sys=0.00, real=0.25 secs]
> 2015-08-21T16:54:55.340-0400: [GC [PSYoungGen: 2390748K->336907K(3800064K)] 12794703K->11043658K(14985216K), 0.3550330 secs] [Times: user=6.28 sys=0.00, real=0.35 secs]
> 2015-08-21T16:54:55.695-0400: [Full GC [PSYoungGen: 336907K->0K(3800064K)] [ParOldGen: 10706750K->5725402K(11185152K)] 11043658K->5725402K(14985216K) [PSPermGen: 57214K->57214K(57856K)], 9.5623960 secs] [Times: user=150.15 sys=0.28, real=9.56 secs]
> 2015-08-21T16:55:06.779-0400: [GC [PSYoungGen: 2074112K->397376K(4020224K)] 7799514K->6122778K(15205376K), 0.2072180 secs] [Times: user=3.65 sys=0.00, real=0.21 secs]
> 2015-08-21T16:55:09.161-0400: [GC [PSYoungGen: 2763840K->205647K(3938816K)] 8489242K->6268483K(15123968K), 0.2878000 secs] [Times: user=5.11 sys=0.00, real=0.29 secs]
> 2015-08-21T16:55:11.631-0400: [GC [PSYoungGen: 2572111K->178593K(4110848K)] 8634947K->6446009K(15296000K), 0.1981990 secs] [Times: user=3.51 sys=0.00, real=0.20 secs]
> 2015-08-21T16:55:14.038-0400: [GC [PSYoungGen: 2762145K->420788K(4065280K)] 9029561K->6688205K(15250432K), 0.1603690 secs] [Times: user=2.84 sys=0.00, real=0.16 secs]
> 2015-08-21T16:55:16.608-0400: [GC [PSYoungGen: 3004340K->713513K(4204032K)] 9271757K->6980930K(15389184K), 0.2504660 secs] [Times: user=4.45 sys=0.00, real=0.25 secs]
> 2015-08-21T16:55:19.539-0400: [GC [PSYoungGen: 3526441K->1071565K(3884544K)] 9793858K->7338983K(15069696K), 0.5002750 secs] [Times: user=8.93 sys=0.00, real=0.50 secs]
> 2015-08-21T16:55:22.942-0400: [GC [PSYoungGen: 3884493K->1386667K(3655168K)] 10151912K->7654086K(14840320K), 0.4753120 secs] [Times: user=8.48 sys=0.00, real=0.48 secs]
> 2015-08-21T16:55:25.740-0400: [GC [PSYoungGen: 3529387K->1573464K(3716608K)] 9796806K->7840883K(14901760K), 0.5294890 secs] [Times: user=9.46 sys=0.00, real=0.53 secs]
> 2015-08-21T16:55:28.230-0400: [GC [PSYoungGen: 3716184K->1408775K(3728384K)] 9983603K->8016021K(14913536K), 0.6090130 secs] [Times: user=10.59 sys=0.00, real=0.61 secs]
> 2015-08-21T16:55:30.115-0400: [GC [PSYoungGen: 3272967K->879217K(3728384K)] 9880213K->8118285K(14913536K), 0.6053810 secs] [Times: user=10.73 sys=0.01, real=0.60 secs]
> 2015-08-21T16:55:32.029-0400: [GC [PSYoungGen: 2743409K->416860K(3728384K)] 9982477K->8203931K(14913536K), 0.5133040 secs] [Times: user=9.17 sys=0.00, real=0.51 secs]
> 2015-08-21T16:55:33.871-0400: [GC [PSYoungGen: 2281052K->160473K(3728384K)] 10068123K->8281580K(14913536K), 0.3045040 secs] [Times: user=5.42 sys=0.00, real=0.31 secs]
> 2015-08-21T16:55:35.575-0400: [GC [PSYoungGen: 2024665K->63592K(3728384K)] 10145772K->8345118K(14913536K), 0.1367630 secs] [Times: user=2.41 sys=0.00, real=0.14 secs]
> 2015-08-21T16:55:37.049-0400: [GC [PSYoungGen: 1927784K->59174K(3728384K)] 10209310K->8404017K(14913536K), 0.1407080 secs] [Times: user=2.48 sys=0.00, real=0.14 secs]
> 2015-08-21T16:55:38.600-0400: [GC [PSYoungGen: 1923366K->57465K(3748352K)] 10268209K->8461019K(14933504K), 0.1109560 secs] [Times: user=1.93 sys=0.01, real=0.11 secs]
> 2015-08-21T16:55:40.411-0400: [GC [PSYoungGen: 1941625K->73353K(3728384K)] 10345179K->8528297K(14913536K), 0.1695660 secs] [Times: user=2.92 sys=0.00, real=0.17 secs]
> 2015-08-21T16:55:42.128-0400: [GC [PSYoungGen: 1957513K->126635K(3871232K)] 10412457K->8581579K(15056384K), 0.1535930 secs] [Times: user=2.71 sys=0.00, real=0.15 secs]
> 2015-08-21T16:55:43.705-0400: [GC [PSYoungGen: 2206891K->189292K(3801600K)] 10661835K->8644236K(14986752K), 0.1850870 secs] [Times: user=3.28 sys=0.00, real=0.19 secs]
> 2015-08-21T16:55:45.390-0400: [GC [PSYoungGen: 2260426K->307881K(4056064K)] 10715370K->8762825K(15241216K), 0.2177180 secs] [Times: user=3.84 sys=0.00, real=0.22 secs]
> 2015-08-21T16:55:47.530-0400: [GC [PSYoungGen: 2725545K->684173K(3954176K)] 11180489K->9139117K(15139328K), 0.3418670 secs] [Times: user=6.02 sys=0.05, real=0.34 secs]
> 2015-08-21T16:55:49.983-0400: [GC [PSYoungGen: 3101837K->1162872K(3749376K)] 11556781K->9617816K(14934528K), 0.4373140 secs] [Times: user=7.79 sys=0.00, real=0.44 secs]
> 2015-08-21T16:55:52.563-0400: [GC [PSYoungGen: 3387512K->1596897K(3821568K)] 11842456K->10121078K(15006720K), 0.5456530 secs] [Times: user=9.73 sys=0.01, real=0.55 secs]
> 2015-08-21T16:55:55.461-0400: [GC [PSYoungGen: 3821537K->1770995K(3635200K)] 12345718K->10509011K(14820352K), 0.6742280 secs] [Times: user=12.04 sys=0.00, real=0.67 secs]
> 2015-08-21T16:55:58.088-0400: [GC [PSYoungGen: 3635187K->1560141K(3728384K)] 12373203K->10723772K(14913536K), 0.6732890 secs] [Times: user=11.98 sys=0.01, real=0.68 secs]
> 2015-08-21T16:56:01.357-0400: [GC [PSYoungGen: 3424333K->815806K(3728384K)] 12587964K->10923015K(14913536K), 0.7893440 secs] [Times: user=14.07 sys=0.01, real=0.79 secs]
> 2015-08-21T16:56:04.146-0400: [GC [PSYoungGen: 2679998K->364642K(3728384K)] 12787207K->11090639K(14913536K), 0.5218630 secs] [Times: user=9.30 sys=0.01, real=0.52 secs]
> 2015-08-21T16:56:04.668-0400: [Full GC [PSYoungGen: 364642K->0K(3728384K)] [ParOldGen: 10725997K->6388280K(11185152K)] 11090639K->6388280K(14913536K) [PSPermGen: 57214K->57214K(57856K)], 9.5366890 secs] [Times: user=141.02 sys=0.02, real=9.54 secs]
> 2015-08-21T16:56:15.906-0400: [GC [PSYoungGen: 1864192K->244290K(3728384K)] 8252472K->6632570K(14913536K), 0.1593280 secs] [Times: user=2.77 sys=0.00, real=0.16 secs]
> 2015-08-21T16:56:17.697-0400: [GC [PSYoungGen: 2108482K->271776K(3728384K)] 8496762K->6904251K(14913536K), 0.2788640 secs] [Times: user=4.90 sys=0.03, real=0.28 secs]
> 2015-08-21T16:56:19.662-0400: [GC [PSYoungGen: 2135968K->340850K(3728384K)] 8768443K->7244018K(14913536K), 0.3142780 secs] [Times: user=5.51 sys=0.02, real=0.31 secs]
> 2015-08-21T16:56:21.646-0400: [GC [PSYoungGen: 2200652K->420754K(3728384K)] 9103820K->7605015K(14913536K), 0.3844990 secs] [Times: user=6.78 sys=0.00, real=0.38 secs]
> 2015-08-21T16:56:23.995-0400: [GC [PSYoungGen: 2284946K->283289K(3728384K)] 9469207K->7851835K(14913536K), 0.2670850 secs] [Times: user=4.71 sys=0.00, real=0.26 secs]
> 2015-08-21T16:56:26.054-0400: [GC [PSYoungGen: 2147384K->285799K(3728384K)] 9715930K->8114308K(14913536K), 0.3649000 secs] [Times: user=6.45 sys=0.01, real=0.36 secs]
> 2015-08-21T16:56:28.386-0400: [GC [PSYoungGen: 2149991K->505992K(3910144K)] 9978500K->8612203K(15095296K), 0.3867880 secs] [Times: user=6.84 sys=0.01, real=0.38 secs]
> 2015-08-21T16:56:30.799-0400: [GC [PSYoungGen: 2620552K->454232K(3796992K)] 10726763K->9026057K(14982144K), 0.4349370 secs] [Times: user=7.71 sys=0.01, real=0.44 secs]
> 2015-08-21T16:56:33.271-0400: [GC [PSYoungGen: 2568792K->385079K(4081152K)] 11140619K->9382690K(15266304K), 0.3980990 secs] [Times: user=7.03 sys=0.00, real=0.40 secs]
> 2015-08-21T16:56:36.305-0400: [GC [PSYoungGen: 2878519K->259616K(4004864K)] 11876136K->9641767K(15190016K), 0.4366160 secs] [Times: user=7.78 sys=0.00, real=0.43 secs]
> 2015-08-21T16:56:39.004-0400: [GC [PSYoungGen: 2753056K->176304K(4172800K)] 12135207K->9817707K(15357952K), 0.2463590 secs] [Times: user=4.37 sys=0.00, real=0.25 secs]
> 2015-08-21T16:56:41.525-0400: [GC [PSYoungGen: 2889392K->175585K(4132864K)] 12530795K->9993292K(15318016K), 0.1842920 secs] [Times: user=3.25 sys=0.00, real=0.19 secs]
> 2015-08-21T16:56:44.508-0400: [GC [PSYoungGen: 2888673K->394614K(4302336K)] 12706380K->10212322K(15487488K), 0.1747460 secs] [Times: user=3.07 sys=0.00, real=0.17 secs]
> 2015-08-21T16:56:47.239-0400: [GC [PSYoungGen: 3321718K->560043K(4217344K)] 13139426K->10377750K(15402496K), 0.2171460 secs] [Times: user=3.83 sys=0.01, real=0.22 secs]
> 2015-08-21T16:56:49.944-0400: [GC [PSYoungGen: 3487147K->706210K(4312576K)] 13304854K->10523918K(15497728K), 0.2627380 secs] [Times: user=4.62 sys=0.00, real=0.27 secs]
> 2015-08-21T16:56:53.378-0400: [GC [PSYoungGen: 3768482K->1088212K(4150784K)] 13586190K->10905920K(15335936K), 0.4293970 secs] [Times: user=7.59 sys=0.00, real=0.43 secs]
> 2015-08-21T16:56:56.477-0400: [GC [PSYoungGen: 4150484K->1309420K(3783168K)] 13968192K->11127127K(14968320K), 0.4637440 secs] [Times: user=8.23 sys=0.00, real=0.47 secs]
> 2015-08-21T16:56:58.830-0400: [GC [PSYoungGen: 3679468K->1498621K(3868672K)] 13497176K->11316329K(15053824K), 0.5533130 secs] [Times: user=9.84 sys=0.01, real=0.55 secs]
> 2015-08-21T16:57:01.296-0400: [GC [PSYoungGen: 3868669K->1474972K(3588096K)] 13686378K->11460991K(14773248K), 0.6170960 secs] [Times: user=10.76 sys=0.02, real=0.62 secs]
> 2015-08-21T16:57:03.201-0400: [GC [PSYoungGen: 3339164K->1323407K(3728384K)] 13325183K->11631637K(14913536K), 0.6002320 secs] [Times: user=10.66 sys=0.01, real=0.60 secs]
> 2015-08-21T16:57:05.243-0400: [GC [PSYoungGen: 3187599K->894715K(3728384K)] 13495829K->11696649K(14913536K), 0.5563990 secs] [Times: user=9.85 sys=0.00, real=0.56 secs]
> 2015-08-21T16:57:05.799-0400: [Full GC [PSYoungGen: 894715K->0K(3728384K)] [ParOldGen: 10801933K->4817827K(11185152K)] 11696649K->4817827K(14913536K) [PSPermGen: 57214K->57214K(57856K)], 9.0502760 secs] [Times: user=141.39 sys=0.16, real=9.04 secs]
> 2015-08-21T16:57:16.259-0400: [GC [PSYoungGen: 1864192K->61920K(3728384K)] 6682027K->4879755K(14913536K), 0.0411120 secs] [Times: user=0.68 sys=0.00, real=0.04 secs]
> 2015-08-21T16:57:17.975-0400: [GC [PSYoungGen: 1926112K->115664K(3728384K)] 6743952K->4933504K(14913536K), 0.0570010 secs] [Times: user=0.97 sys=0.00, real=0.06 secs]
> 2015-08-21T16:57:19.769-0400: [GC [PSYoungGen: 1979856K->154159K(3728384K)] 6797696K->5032420K(14913536K), 0.0877050 secs] [Times: user=1.47 sys=0.00, real=0.09 secs]
> 2015-08-21T16:57:21.357-0400: [GC [PSYoungGen: 2018351K->209606K(3728384K)] 6896612K->5231255K(14913536K), 0.1261810 secs] [Times: user=2.04 sys=0.01, real=0.13 secs]
> 2015-08-21T16:57:22.845-0400: [GC [PSYoungGen: 2073798K->134065K(3728384K)] 7095447K->5341302K(14913536K), 0.1327730 secs] [Times: user=2.30 sys=0.00, real=0.13 secs]
> 2015-08-21T16:57:24.396-0400: [GC [PSYoungGen: 1998257K->439848K(3857408K)] 7205495K->5647085K(15042560K), 0.1896380 secs] [Times: user=3.26 sys=0.01, real=0.19 secs]
> 2015-08-21T16:57:26.047-0400: [GC [PSYoungGen: 2452008K->805703K(3747328K)] 7659247K->6012942K(14932480K), 0.3043400 secs] [Times: user=5.32 sys=0.01, real=0.30 secs]
> 2015-08-21T16:57:27.867-0400: [GC [PSYoungGen: 2817863K->1137456K(3686400K)] 8025102K->6344695K(14871552K), 0.4460680 secs] [Times: user=7.69 sys=0.02, real=0.45 secs]
> 2015-08-21T16:57:30.101-0400: [GC [PSYoungGen: 3110192K->1284856K(3257856K)] 8317431K->6608870K(14443008K), 0.5406400 secs] [Times: user=9.49 sys=0.01, real=0.54 secs]
> 2015-08-21T16:57:32.584-0400: [GC [PSYoungGen: 3257592K->875015K(3728384K)] 8581606K->6829064K(14913536K), 0.6183670 secs] [Times: user=10.99 sys=0.00, real=0.61 secs]
> 2015-08-21T16:57:35.140-0400: [GC [PSYoungGen: 2739207K->413106K(3728384K)] 8693256K->7016035K(14913536K), 0.6108230 secs] [Times: user=10.89 sys=0.00, real=0.61 secs]
> 2015-08-21T16:57:37.420-0400: [GC [PSYoungGen: 2277298K->143538K(3802112K)] 8880227K->7160429K(14987264K), 0.3580630 secs] [Times: user=6.35 sys=0.01, real=0.36 secs]
> 2015-08-21T16:57:39.480-0400: [GC [PSYoungGen: 2133170K->162775K(3780096K)] 9150061K->7322715K(14965248K), 0.1566920 secs] [Times: user=2.75 sys=0.00, real=0.16 secs]
> 2015-08-21T16:57:41.162-0400: [GC [PSYoungGen: 2152407K->146880K(3883520K)] 9312347K->7469656K(15068672K), 0.2068290 secs] [Times: user=3.65 sys=0.01, real=0.21 secs]
> 2015-08-21T16:57:42.985-0400: [GC [PSYoungGen: 2277312K->167519K(3839488K)] 9600088K->7637258K(15024640K), 0.1738150 secs] [Times: user=2.90 sys=0.00, real=0.17 secs]
> 2015-08-21T16:57:44.896-0400: [GC [PSYoungGen: 2297951K->289764K(4017664K)] 9767690K->7917573K(15202816K), 0.1975750 secs] [Times: user=3.44 sys=0.00, real=0.19 secs]
> 2015-08-21T16:57:47.206-0400: [GC [PSYoungGen: 2652644K->726301K(3937792K)] 10280453K->8354109K(15122944K), 0.3095890 secs] [Times: user=5.43 sys=0.01, real=0.31 secs]
> 2015-08-21T16:57:49.515-0400: [GC [PSYoungGen: 3089181K->1279912K(3672064K)] 10716989K->8907720K(14857216K), 0.6163300 secs] [Times: user=10.90 sys=0.01, real=0.62 secs]
> 2015-08-21T16:57:52.077-0400: [GC [PSYoungGen: 3394984K->1624385K(3739648K)] 11022792K->9464496K(14924800K), 0.7417600 secs] [Times: user=13.21 sys=0.01, real=0.74 secs]
> 2015-08-21T16:57:54.733-0400: [GC [PSYoungGen: 3739457K->1053135K(3717120K)] 11579568K->9905295K(14902272K), 0.9749830 secs] [Times: user=17.42 sys=0.01, real=0.98 secs]
> 2015-08-21T16:57:57.333-0400: [GC [PSYoungGen: 2917327K->276431K(3728384K)] 11769487K->10181125K(14913536K), 0.9417270 secs] [Times: user=16.79 sys=0.02, real=0.94 secs]
> 2015-08-21T16:57:59.836-0400: [GC [PSYoungGen: 2140623K->184074K(3728384K)] 12045405K->10355736K(14913536K), 0.2475730 secs] [Times: user=4.29 sys=0.00, real=0.25 secs]
> 2015-08-21T16:58:00.084-0400: [Full GC [PSYoungGen: 184074K->0K(3728384K)] [ParOldGen: 10171661K->5615593K(11185152K)] 10355736K->5615593K(14913536K) [PSPermGen: 57214K->57214K(57856K)], 8.3282690 secs] [Times: user=131.68 sys=0.03, real=8.33 secs]
> 2015-08-21T16:58:09.925-0400: [GC [PSYoungGen: 1864192K->180249K(3728384K)] 7479785K->5795842K(14913536K), 0.1359660 secs] [Times: user=2.20 sys=0.01, real=0.14 secs]
> 2015-08-21T16:58:11.552-0400: [GC [PSYoungGen: 2044441K->341754K(3816448K)] 7660034K->6134326K(15001600K), 0.2191480 secs] [Times: user=3.84 sys=0.00, real=0.22 secs]
> 2015-08-21T16:58:13.302-0400: [GC [PSYoungGen: 2295546K->422890K(3729920K)] 8088119K->6514065K(14915072K), 0.3308370 secs] [Times: user=5.83 sys=0.00, real=0.33 secs]
> 2015-08-21T16:58:15.171-0400: [GC [PSYoungGen: 2376682K->359260K(3996672K)] 8467863K->6838496K(15181824K), 0.4105890 secs] [Times: user=7.31 sys=0.00, real=0.41 secs]
> 2015-08-21T16:58:17.623-0400: [GC [PSYoungGen: 2678620K->372369K(3915264K)] 9157860K->7204743K(15100416K), 0.4238980 secs] [Times: user=7.53 sys=0.00, real=0.42 secs]
> 2015-08-21T16:58:20.251-0400: [GC [PSYoungGen: 2691729K->245265K(4132352K)] 9524108K->7450394K(15317504K), 0.3534780 secs] [Times: user=6.26 sys=0.00, real=0.35 secs]
> 2015-08-21T16:58:23.493-0400: [GC [PSYoungGen: 2862097K->233746K(4077056K)] 10067226K->7683288K(15262208K), 0.2583880 secs] [Times: user=4.57 sys=0.01, real=0.26 secs]
> 2015-08-21T16:58:26.355-0400: [GC [PSYoungGen: 2850578K->301387K(4254720K)] 10300120K->7979146K(15439872K), 0.3885990 secs] [Times: user=6.78 sys=0.01, real=0.39 secs]
> 2015-08-21T16:58:29.431-0400: [GC [PSYoungGen: 3150667K->541461K(4187136K)] 10828426K->8474442K(15372288K), 0.4318220 secs] [Times: user=7.52 sys=0.01, real=0.44 secs]
> 2015-08-21T16:58:32.233-0400: [GC [PSYoungGen: 3390741K->399554K(4381696K)] 11323722K->8841488K(15566848K), 0.4823410 secs] [Times: user=8.37 sys=0.00, real=0.48 secs]
> 2015-08-21T16:58:35.463-0400: [GC [PSYoungGen: 3496642K->214369K(4307968K)] 11938576K->9056524K(15493120K), 0.3941770 secs] [Times: user=6.75 sys=0.01, real=0.39 secs]
> 2015-08-21T16:58:39.364-0400: [GC [PSYoungGen: 3311457K->229978K(4460032K)] 12153612K->9284940K(15645184K), 0.2562130 secs] [Times: user=4.49 sys=0.00, real=0.25 secs]
> 2015-08-21T16:58:43.207-0400: [GC [PSYoungGen: 3515482K->388309K(4418048K)] 12570445K->9443272K(15603200K), 0.1612730 secs] [Times: user=2.81 sys=0.00, real=0.16 secs]
> 2015-08-21T16:58:46.307-0400: [GC [PSYoungGen: 3673813K->470089K(4571136K)] 12728782K->9525058K(15756288K), 0.1758880 secs] [Times: user=2.92 sys=0.01, real=0.17 secs]
> 2015-08-21T16:58:49.392-0400: [GC [PSYoungGen: 3978313K->917141K(4425728K)] 13033282K->9972110K(15610880K), 0.3321180 secs] [Times: user=5.87 sys=0.00, real=0.34 secs]
> 2015-08-21T16:58:53.158-0400: [GC [PSYoungGen: 4425365K->1057340K(4140544K)] 13480334K->10112309K(15325696K), 0.4089550 secs] [Times: user=7.28 sys=0.00, real=0.41 secs]
> 2015-08-21T16:58:55.894-0400: [GC [PSYoungGen: 4049980K->1299958K(4292608K)] 13104949K->10363575K(15477760K), 0.4509200 secs] [Times: user=8.03 sys=0.00, real=0.45 secs]
> 2015-08-21T16:58:58.723-0400: [GC [PSYoungGen: 4292598K->1070291K(3699712K)] 13356215K->10474715K(14884864K), 0.5783500 secs] [Times: user=10.28 sys=0.01, real=0.58 secs]
> 2015-08-21T16:59:01.146-0400: [GC [PSYoungGen: 3470035K->1136931K(3536896K)] 12874459K->10541355K(14722048K), 0.4320870 secs] [Times: user=7.61 sys=0.01, real=0.44 secs]
> 2015-08-21T16:59:03.597-0400: [GC [PSYoungGen: 3536675K->1093619K(3812352K)] 12941099K->10594202K(14997504K), 0.4521770 secs] [Times: user=7.99 sys=0.00, real=0.45 secs]
> 2015-08-21T16:59:05.946-0400: [GC [PSYoungGen: 3223027K->699072K(3860992K)] 12723610K->10634428K(15046144K), 0.5153210 secs] [Times: user=9.14 sys=0.00, real=0.52 secs]
> 2015-08-21T16:59:08.228-0400: [GC [PSYoungGen: 2828480K->315546K(3974656K)] 12763836K->10671299K(15159808K), 0.4134510 secs] [Times: user=7.24 sys=0.00, real=0.41 secs]
> 2015-08-21T16:59:10.256-0400: [GC [PSYoungGen: 2652314K->169594K(3954688K)] 13008067K->10709291K(15139840K), 0.1790620 secs] [Times: user=3.12 sys=0.00, real=0.18 secs]
> 2015-08-21T16:59:10.436-0400: [Full GC [PSYoungGen: 169594K->0K(3954688K)] [ParOldGen: 10539697K->5005889K(11185152K)] 10709291K->5005889K(15139840K) [PSPermGen: 57215K->57215K(57344K)], 8.6283890 secs] [Times: user=139.03 sys=0.18, real=8.63 secs]
> 2015-08-21T16:59:20.551-0400: [GC [PSYoungGen: 2336768K->57691K(3976704K)] 7342657K->5063580K(15161856K), 0.0320200 secs] [Times: user=0.50 sys=0.00, real=0.03 secs]
> 2015-08-21T16:59:22.141-0400: [GC [PSYoungGen: 2420571K->103430K(3977728K)] 7426460K->5109319K(15162880K), 0.0542670 secs] [Times: user=0.89 sys=0.00, real=0.05 secs]
> 2015-08-21T16:59:24.166-0400: [GC [PSYoungGen: 2466310K->149395K(4040704K)] 7472199K->5155285K(15225856K), 0.0890410 secs] [Times: user=1.44 sys=0.01, real=0.09 secs]
> 2015-08-21T16:59:25.878-0400: [GC [PSYoungGen: 2597779K->184081K(4000256K)] 7603669K->5189970K(15185408K), 0.0951090 secs] [Times: user=1.51 sys=0.01, real=0.10 secs]
> 2015-08-21T16:59:27.641-0400: [GC [PSYoungGen: 2632465K->215665K(4155392K)] 7638354K->5221554K(15340544K), 0.1347880 secs] [Times: user=2.24 sys=0.00, real=0.13 secs]
> 2015-08-21T16:59:30.014-0400: [GC [PSYoungGen: 2872433K->249168K(4093952K)] 7878322K->5255058K(15279104K), 0.1940370 secs] [Times: user=3.26 sys=0.01, real=0.19 secs]
> 2015-08-21T16:59:32.061-0400: [GC [PSYoungGen: 2905936K->289950K(4296192K)] 7911834K->5295847K(15481344K), 0.1987530 secs] [Times: user=3.32 sys=0.00, real=0.20 secs]
> 2015-08-21T16:59:34.387-0400: [GC [PSYoungGen: 3216542K->333450K(4222976K)] 8222445K->5339353K(15408128K), 0.2298800 secs] [Times: user=3.90 sys=0.00, real=0.23 secs]
> 2015-08-21T16:59:37.098-0400: [GC [PSYoungGen: 3260042K->366870K(4449280K)] 8265945K->5372773K(15634432K), 0.2280860 secs] [Times: user=3.86 sys=0.01, real=0.23 secs]
> 2015-08-21T16:59:39.947-0400: [GC [PSYoungGen: 3596054K->471644K(4372480K)] 8601957K->5477547K(15557632K), 0.2958990 secs] [Times: user=4.93 sys=0.01, real=0.30 secs]
> 2015-08-21T16:59:42.747-0400: [GC [PSYoungGen: 3700828K->812416K(4332032K)] 8706737K->5818325K(15517184K), 0.3462400 secs] [Times: user=6.09 sys=0.01, real=0.35 secs]
> 2015-08-21T16:59:45.565-0400: [GC [PSYoungGen: 4047744K->1178596K(4413952K)] 9053653K->6259289K(15599104K), 0.5043450 secs] [Times: user=8.91 sys=0.01, real=0.50 secs]
> 2015-08-21T16:59:48.822-0400: [GC [PSYoungGen: 4413924K->1178592K(3244544K)] 9494617K->6813581K(14429696K), 0.9950540 secs] [Times: user=17.66 sys=0.03, real=1.00 secs]
> 2015-08-21T16:59:51.701-0400: [GC [PSYoungGen: 3244512K->1401555K(3467776K)] 8879501K->7083031K(14652928K), 0.4898300 secs] [Times: user=8.73 sys=0.00, real=0.49 secs]
> 2015-08-21T16:59:54.086-0400: [GC [PSYoungGen: 3467475K->1517388K(3728384K)] 9148951K->7271321K(14913536K), 0.5393400 secs] [Times: user=9.61 sys=0.00, real=0.54 secs]
> 2015-08-21T16:59:56.357-0400: [GC [PSYoungGen: 3381580K->1588229K(3728384K)] 9135513K->7409023K(14913536K), 0.5366240 secs] [Times: user=9.56 sys=0.00, real=0.53 secs]
> 2015-08-21T16:59:58.894-0400: [GC [PSYoungGen: 3452421K->1639924K(3728384K)] 9273215K->7548125K(14913536K), 0.5923160 secs] [Times: user=10.53 sys=0.01, real=0.60 secs]
> 2015-08-21T17:00:01.290-0400: [GC [PSYoungGen: 3504116K->1679530K(3728384K)] 9412412K->7666735K(14913536K), 0.5578720 secs] [Times: user=9.92 sys=0.00, real=0.56 secs]
> 2015-08-21T17:00:03.446-0400: [GC [PSYoungGen: 3543722K->1711705K(3728384K)] 9530930K->7867994K(14913536K), 0.5663970 secs] [Times: user=9.98 sys=0.00, real=0.56 secs]
> 2015-08-21T17:00:06.183-0400: [GC [PSYoungGen: 3575897K->1537411K(3728384K)] 9732186K->8248661K(14913536K), 0.7580860 secs] [Times: user=13.49 sys=0.01, real=0.76 secs]
> 2015-08-21T17:00:09.207-0400: [GC [PSYoungGen: 3401603K->1294833K(3728384K)] 10112853K->8493004K(14913536K), 0.6302370 secs] [Times: user=11.21 sys=0.04, real=0.63 secs]
> 2015-08-21T17:00:11.812-0400: [GC [PSYoungGen: 3159025K->1230822K(3728384K)] 10357196K->8696289K(14913536K), 0.5573280 secs] [Times: user=9.50 sys=0.01, real=0.56 secs]
> 2015-08-21T17:00:14.651-0400: [GC [PSYoungGen: 3095014K->1358347K(3728384K)] 10560481K->9101611K(14913536K), 0.6030070 secs] [Times: user=10.69 sys=0.01, real=0.60 secs]
> 2015-08-21T17:00:17.101-0400: [GC [PSYoungGen: 3222539K->899612K(3728384K)] 10965809K->9335622K(14913536K), 0.6895790 secs] [Times: user=12.27 sys=0.00, real=0.69 secs]
> 2015-08-21T17:00:19.795-0400: [GC [PSYoungGen: 2763804K->489281K(3728384K)] 11199816K->9541824K(14913536K), 0.5745470 secs] [Times: user=10.12 sys=0.01, real=0.58 secs]
> 2015-08-21T17:00:22.501-0400: [GC [PSYoungGen: 2353473K->218299K(3728384K)] 11406016K->9719479K(14913536K), 0.4245700 secs] [Times: user=7.56 sys=0.00, real=0.42 secs]
> 2015-08-21T17:00:24.838-0400: [GC [PSYoungGen: 2082491K->219232K(3728384K)] 11583671K->9930182K(14913536K), 0.2530360 secs] [Times: user=4.46 sys=0.00, real=0.26 secs]
> 2015-08-21T17:00:26.859-0400: [GC [PSYoungGen: 2083424K->169198K(3728384K)] 11794375K->10099702K(14913536K), 0.2721990 secs] [Times: user=4.82 sys=0.00, real=0.28 secs]
> 2015-08-21T17:00:28.939-0400: [GC [PSYoungGen: 2033390K->128783K(3728384K)] 11963894K->10228066K(14913536K), 0.1527300 secs] [Times: user=2.69 sys=0.00, real=0.16 secs]
> 2015-08-21T17:00:30.706-0400: [GC [PSYoungGen: 1992975K->107792K(3728384K)] 12092258K->10335337K(14913536K), 0.2070530 secs] [Times: user=3.67 sys=0.01, real=0.21 secs]
> 2015-08-21T17:00:32.644-0400: [GC [PSYoungGen: 1971984K->80581K(3728384K)] 12199529K->10415648K(14913536K), 0.1507680 secs] [Times: user=2.66 sys=0.00, real=0.15 secs]
> 2015-08-21T17:00:34.445-0400: [GC [PSYoungGen: 1944773K->69161K(3728384K)] 12279840K->10484490K(14913536K), 0.1355530 secs] [Times: user=2.37 sys=0.00, real=0.14 secs]
> 2015-08-21T17:00:36.051-0400: [GC [PSYoungGen: 1933353K->152599K(3728384K)] 12348682K->10636601K(14913536K), 0.1194720 secs] [Times: user=2.05 sys=0.00, real=0.11 secs]
> 2015-08-21T17:00:37.650-0400: [GC [PSYoungGen: 2016791K->301922K(3728384K)] 12500793K->10907863K(14913536K), 0.1948420 secs] [Times: user=3.23 sys=0.00, real=0.20 secs]
> 2015-08-21T17:00:37.845-0400: [Full GC [PSYoungGen: 301922K->0K(3728384K)] [ParOldGen: 10605941K->4977253K(11185152K)] 10907863K->4977253K(14913536K) [PSPermGen: 57215K->57215K(57344K)], 8.7096570 secs] [Times: user=136.17 sys=0.20, real=8.71 secs]
> 2015-08-21T17:00:47.940-0400: [GC [PSYoungGen: 1864192K->295008K(3740160K)] 6841445K->5272261K(14925312K), 0.2179710 secs] [Times: user=3.78 sys=0.00, real=0.22 secs]
>
>