You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Durity, Sean R" <SE...@homedepot.com> on 2018/01/03 20:59:59 UTC
RE: [EXTERNAL] Cassandra cluster add new node slowly
You don't mention the version, but here are some general suggestions
- 2 GB heap is very small for a node, especially with 1 TB+ of data. What is the physical RAM on the host? In general, you want ½ of physical RAM for the JVM. (Look in jvm.options or cassandra-env.sh)
- You can change the streaming throughput from the existing nodes, if it looks like the new node can handle it. Look at nodetool setstreamthroughput. Default is 200 (MB/sec).
- You might want to check for a streaming_socket_timeout_in_ms. This has changed over the versions. Some details are at: https://issues.apache.org/jira/browse/CASSANDRA-11839. 24 hours is good recommendation.
- If your new node can't compact fast enough to keep disk usage down, look at compactionthroughput on that node
- nodetool netstats | grep -v "100%" is a good way to see what is happening/if anything is stuck. Newer versions give a bit more info on progress.
- Don't forget to run cleanup on existing nodes after the new nodes are added.
Sean Durity
From: qf zhou [mailto:zhouqf2013@gmail.com]
Sent: Tuesday, January 02, 2018 10:30 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra cluster add new node slowly
The cluster has 3 nodes, and the data in each node is about 1.2 T. I want to add two new nodes to expand the cluster.
Following the instructions from the datastax website, ie, (http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/operations/opsAddNodeToCluster.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_archived_cassandra_3.x_cassandra_operations_opsAddNodeToCluster.html&d=DwMFoQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=AfP0BxfTG1o_-2nRtGSjngs45YndH060h39eVlypVso&s=AaiOypurMtWA470038fHMbaXJPIZeYtLzv3Yt9vf1Sc&e=>),
I try to add one node to the cluster. However, it is too slow and time cost too much. After about 24 hours, it still didn't success.
I run the command: nodetool netstats on the new node, it shows that:
(tb1fullwithstate2 is a big table and 90% of the cluster data is in it. Here I use CompactionStrategy: TimeWindowCompactionStrategy).
/*.*.*.3
Receiving 136 files, 328573794609 bytes total. Already received 8 files, 5774621188 bytes total
tb1/tb1fullwithneweststatetest 3758271/3758271 bytes(100%) received from idx:0/*.*.*.3
system_distributed/repair_history 57534/57534 bytes(100%) received from idx:0/*.*.*.3
system_distributed/parent_repair_history 507660/507660 bytes(100%) received from idx:0/*.*.*.3
tb1/tb1_device_last_state_eachday 15754096/15754096 bytes(100%) received from idx:0/*.*.*.3
mytest1/tb1_test1 8143775/8143775 bytes(100%) received from idx:0/*.*.*.3
tb1/tb1fullwithstate 2251191007/2251191007 bytes(100%) received from idx:0/*.*.*.3
applocationinfo/weiyirong_app 2760/2760 bytes(100%) received from idx:0/*.*.*.3
tb1/tb1fullwithstate2 3490748006/4909554503 bytes(71%) received from idx:0/*.*.*.3
tb1/tb1fullwithneweststate 4458079/4458079 bytes(100%) received from idx:0/*.*.*.3
/*.*.*.2
Receiving 136 files, 336762487360 bytes total. Already received 3 files, 5695770181 bytes total
system_distributed/repair_history 31684/31684 bytes(100%) received from idx:0/*.*.*.2
tb1/tb1fullwithstate 908260516/908260516 bytes(100%) received from idx:0/*.*.*.2
tb1/tb1fullwithstate2 4783622958/4990450588 bytes(95%) received from idx:0/*.*.*.2
tb1/tb1fullwithneweststate 3855023/3855023 bytes(100%) received from idx:0/*.*.*.2
/*.*.*.4
Receiving 132 files, 236250553620 bytes total. Already received 10 files, 3117465128 bytes total
mytest1/wordstest2 46/46 bytes(100%) received from idx:0/*.*.*.4
tb1/tb1fullwithneweststatetest 3416891/3416891 bytes(100%) received from idx:0/*.*.*.4
system_distributed/repair_history 39720/39720 bytes(100%) received from idx:0/*.*.*.4
system_distributed/parent_repair_history 452250/452250 bytes(100%) received from idx:0/*.*.*.4
mytest1/weblogs 104/104 bytes(100%) received from idx:0/*.*.*.4
tb1/tb1_device_last_state_eachday 12670998/12670998 bytes(100%) received from idx:0/*.*.*.4
mytest1/tb1_test1 3257952/3257952 bytes(100%) received from idx:0/*.*.*.4
tb1/tb1fullwithstate 647702056/647702056 bytes(100%) received from idx:0/*.*.*.4
applocationinfo/weiyirong_app 3509/3509 bytes(100%) received from idx:0/*.*.*.4
tb1/tb1fullwithstate2 2446436305/3566562762 bytes(68%) received from idx:0/*.*.*.4
tb1/tb1fullwithneweststate 3485297/3485297 bytes(100%) received from idx:0/*.*.*.4
check the log in the logs/system.log, it shows that:
INFO 06:09:33 Updating topology for /*.*.*.2
INFO 06:09:33 Updating topology for /*.*.*.2
INFO 06:09:33 Node /*.*.*.3 has restarted, now UP
INFO 06:09:33 InetAddress /*.*.*.3 is now UP
INFO 06:09:33 Node /*.*.*.3 state jump to NORMAL
INFO 06:09:33 Updating topology for /*.*.*.3
INFO 06:09:33 Updating topology for /*.*.*.3
INFO 06:09:33 Handshaking version with /*.*.*.2
INFO 06:09:33 Handshaking version with /*.*.*.3
INFO 06:09:36 Handshaking version with /*.*.*.4
INFO 06:10:03 JOINING: Starting to bootstrap...
INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Executing streaming plan for Bootstrap
INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.4
INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.4
INFO 06:10:04 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints
INFO 06:10:04 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO 06:10:11 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints
INFO 06:10:11 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO 06:10:12 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints
INFO 06:10:12 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO 06:10:20 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints
INFO 06:10:20 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO 06:10:21 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints
INFO 06:10:21 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO 06:10:28 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints
INFO 06:10:28 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 132 files(220.025GiB), sending 0 files(0.000KiB)
INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.2
INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.2
INFO 06:10:34 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints
INFO 06:10:34 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO 06:10:42 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints
INFO 06:10:42 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO 06:10:58 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints
INFO 06:10:58 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 136 files(313.635GiB), sending 0 files(0.000KiB)
INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.3
INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.3
INFO 06:11:06 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints
INFO 06:11:06 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO 06:11:29 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints
INFO 06:11:29 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO 06:11:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 136 files(306.008GiB), sending 0 files(0.000KiB)
INFO 06:11:36 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints
INFO 06:11:36 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO 06:11:49 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints
INFO 06:11:49 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO 06:11:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints
INFO 06:11:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO 06:12:26 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints
INFO 06:12:26 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
INFO 06:12:45 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints
INFO 06:12:45 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
INFO 06:12:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints
INFO 06:12:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
INFO 06:24:03 G1 Young Generation GC in 242ms. G1 Eden Space: 18924699648 -> 0; G1 Old Gen: 4360876536 -> 5095617016; G1 Survivor Space: 1677721600 -
INFO 06:26:06 G1 Young Generation GC in 215ms. G1 Eden Space: 18220056576 -> 0; G1 Old Gen: 5981077496 -> 6597610488; G1 Survivor Space: 1493172224 -
INFO 06:29:04 G1 Young Generation GC in 225ms. G1 Eden Space: 18538823680 -> 0; G1 Old Gen: 7910457840 -> 8860639728; G1 Survivor Space: 1895825408 -
INFO 06:33:22 G1 Young Generation GC in 214ms. G1 Eden Space: 16559112192 -> 0; G1 Old Gen: 11868321784 -> 11941600752; G1 Survivor Space: 1241513984
INFO 06:33:33 G1 Young Generation GC in 205ms. G1 Eden Space: 8069840896 -> 0; G1 Old Gen: 11941600752 -> 13683606520; G1 Survivor Space: 1895825408
INFO 06:41:01 G1 Young Generation GC in 383ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 4770399736 -> 5551887344; G1 Survivor Space: 1728053248 -
INFO 06:45:07 G1 Young Generation GC in 226ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 8036286448 -> 8739698672; G1 Survivor Space: 1577058304 -
INFO 06:47:11 G1 Young Generation GC in 203ms. G1 Eden Space: 18538823680 -> 0; G1 Survivor Space: 1191182336 -> 1778384896;
INFO 06:47:22 G1 Young Generation GC in 241ms. G1 Eden Space: 8086618112 -> 0; G1 Old Gen: 9999220720 -> 11591876088; G1 Survivor Space: 1778384896 -
INFO 06:49:43 G1 Young Generation GC in 201ms. G1 Eden Space: 16022241280 -> 0; G1 Old Gen: 11591876088 -> 12084766712; G1 Survivor Space: 1644167168
INFO 06:53:21 G1 Young Generation GC in 322ms. G1 Eden Space: 13237223424 -> 0; G1 Old Gen: 14755561464 -> 15445886448; G1 Survivor Space: 1459617792
INFO 06:55:34 G1 Young Generation GC in 343ms. G1 Eden Space: 16525557760 -> 0; G1 Old Gen: 2939907056 -> 3636770792; G1 Survivor Space: 1610612736 -
INFO 07:09:30 Redistributing index summaries
INFO 07:14:25 G1 Young Generation GC in 270ms. G1 Eden Space: 15602810880 -> 0; G1 Old Gen: 4360518136 -> 5243274224; G1 Survivor Space: 1610612736 -
INFO 07:16:51 G1 Young Generation GC in 229ms. G1 Eden Space: 19008585728 -> 0; G1 Old Gen: 6132072440 -> 6815145976; G1 Survivor Space: 1593835520 -
INFO 07:19:34 G1 Young Generation GC in 215ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 8689337336 -> 9071161328; G1 Survivor Space: 1610612736 -
INFO 07:23:35 G1 Young Generation GC in 293ms. G1 Eden Space: 15753805824 -> 0; G1 Old Gen: 11777605616 -> 12289649640; G1 Survivor Space: 1694498816
INFO 07:29:42 G1 Young Generation GC in 362ms. G1 Eden Space: 15955132416 -> 0; G1 Old Gen: 3742637560 -> 4410361824; G1 Survivor Space: 1644167168 -
INFO 07:35:12 G1 Young Generation GC in 208ms. G1 Eden Space: 19478347776 -> 0; G1 Old Gen: 7199737832 -> 7208126432; G1 Survivor Space: 1124073472 -
INFO 07:35:33 G1 Young Generation GC in 219ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 7208126432 -> 8072478184; G1 Survivor Space: 1862270976 -
INFO 07:45:19 G1 Young Generation GC in 213ms. G1 Eden Space: 13153337344 -> 0; G1 Old Gen: 15418261472 -> 16123805152; G1 Survivor Space: 1174405120
INFO 07:48:08 G1 Young Generation GC in 335ms. G1 Eden Space: 18387828736 -> 0; G1 Old Gen: 3852453368 -> 4464797168; G1 Survivor Space: 1644167168 -
INFO 07:48:08 G1 Young Generation GC in 209ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 4464797168 -> 5989466096; G1 Survivor Space: 1476395008 ->
INFO 07:50:25 G1 Young Generation GC in 221ms. G1 Eden Space: 17498636288 -> 0; G1 Old Gen: 5981077496 -> 6488668656; G1 Survivor Space: 1593835520 -
INFO 08:03:31 G1 Young Generation GC in 748ms. G1 Eden Space: 8673820672 -> 0; G1 Old Gen: 4386367472 -> 5874747352; G1 Survivor Space: 1627389952 ->
INFO 08:09:30 Redistributing index summaries
INFO 08:10:05 G1 Young Generation GC in 213ms. G1 Eden Space: 18220056576 -> 0; G1 Old Gen: 9276024272 -> 9969348048; G1 Survivor Space: 1711276032 -
INFO 08:18:59 G1 Young Generation GC in 228ms. G1 Eden Space: 17347641344 -> 0; G1 Old Gen: 4546626040 -> 5180788216; G1 Survivor Space: 1593835520 -
INFO 08:33:12 G1 Young Generation GC in 207ms. G1 Eden Space: 19411238912 -> 0; G1 Survivor Space: 1191182336 -> 1828716544;
INFO 08:33:13 G1 Young Generation GC in 273ms. G1 Eden Space: 452984832 -> 0; G1 Old Gen: 4668340728 -> 6320276984; G1 Survivor Space: 1828716544 ->
INFO 08:41:59 G1 Young Generation GC in 202ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 8701765104 -> 8927144944; G1 Survivor Space: 1577058304 -
INFO 08:56:31 G1 Young Generation GC in 205ms. G1 Eden Space: 11072962560 -> 0; G1 Old Gen: 9563013104 -> 11068430328; G1 Survivor Space: 1660944384
INFO 09:05:49 G1 Young Generation GC in 367ms. G1 Eden Space: 18522046464 -> 0; G1 Old Gen: 4009754608 -> 4732347872; G1 Survivor Space: 1778384896 -
INFO 09:09:30 Redistributing index summaries
INFO 09:14:52 G1 Young Generation GC in 213ms. G1 Eden Space: 15267266560 -> 0; G1 Old Gen: 12155092968 -> 12906235360; G1 Survivor Space: 1744830464
INFO 09:17:23 G1 Young Generation GC in 203ms. G1 Eden Space: 14159970304 -> 0; G1 Old Gen: 14779568608 -> 14771180008; G1 Survivor Space: 989855744
INFO 09:17:45 G1 Young Generation GC in 204ms. G1 Eden Space: 12616466432 -> 0; G1 Old Gen: 14771180008 -> 15500684768; G1 Survivor Space: 1761607680
INFO 09:20:52 G1 Young Generation GC in 361ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 4351990768 -> 5086824424; G1 Survivor Space: 1577058304 -
INFO 09:23:51 G1 Young Generation GC in 207ms. G1 Eden Space: 18891145216 -> 0; G1 Old Gen: 7001879024 -> 7716560880; G1 Survivor Space: 1711276032 -
INFO 09:29:20 G1 Young Generation GC in 202ms. G1 Eden Space: 17028874240 -> 0; G1 Survivor Space: 1040187392 -> 1828716544;
INFO 09:29:40 G1 Young Generation GC in 219ms. G1 Eden Space: 15451815936 -> 0; G1 Old Gen: 11802771944 -> 12574988776; G1 Survivor Space: 1828716544
INFO 09:44:51 G1 Young Generation GC in 202ms. G1 Eden Space: 18001952768 -> 0; G1 Old Gen: 9325481456 -> 10085596664; G1 Survivor Space: 1795162112
INFO 09:45:23 G1 Young Generation GC in 209ms. G1 Eden Space: 9177137152 -> 0; G1 Old Gen: 10093985264 -> 12309118960; G1 Survivor Space: 1677721600
INFO 09:50:33 G1 Young Generation GC in 201ms. G1 Eden Space: 12901679104 -> 0; G1 Old Gen: 14487498224 -> 15268560880; G1 Survivor Space: 1761607680
INFO 10:08:09 Saved KeyCache (2989 items) in 92 ms
INFO 10:09:30 Redistributing index summaries
INFO 10:22:17 G1 Young Generation GC in 216ms. G1 Eden Space: 19495124992 -> 0; G1 Old Gen: 7742571504 -> 7734182904; G1 Survivor Space: 1107296256 -
INFO 10:25:31 G1 Young Generation GC in 206ms. G1 Eden Space: 18622709760 -> 0; G1 Survivor Space: 1040187392 -> 1711276032;
INFO 10:34:54 G1 Young Generation GC in 251ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 5066719736 -> 5691775992; G1 Survivor Space: 1610612736 -
INFO 10:40:08 G1 Young Generation GC in 209ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 9000976880 -> 9090653168; G1 Survivor Space: 1442840576 -
INFO 10:40:08 G1 Young Generation GC in 223ms. G1 Eden Space: 469762048 -> 0; G1 Old Gen: 9090653168 -> 11140124152; G1 Survivor Space: 1862270976 ->
INFO 10:52:15 G1 Young Generation GC in 219ms. G1 Eden Space: 15820914688 -> 0; G1 Old Gen: 5331537912 -> 5828939768; G1 Survivor Space: 1275068416 -
INFO 10:55:30 G1 Young Generation GC in 207ms. G1 Eden Space: 15804137472 -> 0; G1 Old Gen: 7501364216 -> 8001282032; G1 Survivor Space: 1459617792 -
INFO 11:07:57 G1 Young Generation GC in 295ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4301379064 -> 4919709168; G1 Survivor Space: 1442840576 -
INFO 11:09:30 Redistributing index summaries
INFO 11:13:06 G1 Young Generation GC in 204ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 8044675064 -> 8659737080; G1 Survivor Space: 1476395008 -
INFO 11:27:07 G1 Young Generation GC in 202ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4328521712 -> 4905718776; G1 Survivor Space: 1442840576 -
INFO 11:30:08 G1 Young Generation GC in 209ms. G1 Eden Space: 19243466752 -> 0; G1 Survivor Space: 1358954496 -> 1828716544;
INFO 11:30:25 G1 Young Generation GC in 226ms. G1 Eden Space: 14277410816 -> 0; G1 Old Gen: 6349296624 -> 7552639472; G1 Survivor Space: 1828716544 -
INFO 11:32:30 G1 Young Generation GC in 215ms. G1 Eden Space: 19142803456 -> 0; G1 Survivor Space: 1459617792 -> 2046820352;
INFO 11:33:01 G1 Young Generation GC in 247ms. G1 Eden Space: 17263755264 -> 0; G1 Old Gen: 7544250872 -> 8698295280; G1 Survivor Space: 2046820352 -
INFO 11:46:29 G1 Young Generation GC in 355ms. G1 Eden Space: 17230200832 -> 0; G1 Old Gen: 4769358328 -> 5362416624; G1 Survivor Space: 1761607680 -
INFO 11:49:18 G1 Young Generation GC in 233ms. G1 Eden Space: 17867735040 -> 0; G1 Old Gen: 6752829432 -> 6944805360; G1 Survivor Space: 1509949440 -
INFO 11:52:24 G1 Young Generation GC in 209ms. G1 Eden Space: 19193135104 -> 0; G1 Survivor Space: 1409286144 -> 1946157056;
INFO 11:52:47 G1 Young Generation GC in 338ms. G1 Eden Space: 18287165440 -> 0; G1 Old Gen: 8740881904 -> 9340426216; G1 Survivor Space: 1946157056 -
INFO 12:04:08 G1 Young Generation GC in 810ms. G1 Eden Space: 18119393280 -> 0; G1 Old Gen: 3471267832 -> 5003518440; G1 Survivor Space: 1325400064 -
INFO 12:06:28 G1 Young Generation GC in 345ms. G1 Eden Space: 8673820672 -> 0; G1 Old Gen: 5318377952 -> 6577762784; G1 Survivor Space: 1409286144 ->
INFO 12:09:31 G1 Young Generation GC in 204ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 6577762784 -> 6586151384; G1 Survivor Space: 1442840576 -
INFO 12:09:31 Redistributing index summaries
INFO 12:10:01 G1 Young Generation GC in 298ms. G1 Eden Space: 17230200832 -> 0; G1 Old Gen: 6586151384 -> 7710758368; G1 Survivor Space: 1946157056 -
INFO 12:12:37 G1 Young Generation GC in 214ms. G1 Eden Space: 18773704704 -> 0; G1 Old Gen: 8380219864 -> 8919123416; G1 Survivor Space: 1728053248 -
INFO 12:24:54 G1 Young Generation GC in 299ms. G1 Eden Space: 16055795712 -> 0; G1 Old Gen: 4375908856 -> 5009812976; G1 Survivor Space: 1392508928 -
INFO 12:28:34 G1 Young Generation GC in 485ms. G1 Eden Space: 13304332288 -> 0; G1 Old Gen: 6481607672 -> 7506585072; G1 Survivor Space: 1979711488 -
INFO 12:44:52 G1 Young Generation GC in 261ms. G1 Eden Space: 18773704704 -> 0; G1 Old Gen: 4239590896 -> 4692442608; G1 Survivor Space: 1828716544 -
INFO 12:48:37 G1 Young Generation GC in 251ms. G1 Eden Space: 13656653824 -> 0; G1 Old Gen: 6719275504 -> 7776254960; G1 Survivor Space: 1862270976 -
INFO 12:53:47 G1 Young Generation GC in 217ms. G1 Eden Space: 17548967936 -> 0; G1 Old Gen: 9667142640 -> 10202159608; G1 Survivor Space: 1845493760
INFO 13:05:40 G1 Young Generation GC in 287ms. G1 Eden Space: 18320719872 -> 0; G1 Old Gen: 5298861040 -> 5826686456; G1 Survivor Space: 1862270976 -
INFO 13:09:31 Redistributing index summaries
INFO 13:11:44 G1 Young Generation GC in 268ms. G1 Eden Space: 10871635968 -> 0; G1 Old Gen: 8799649784 -> 9797878264; G1 Survivor Space: 1811939328 -
INFO 13:14:22 G1 Young Generation GC in 202ms. G1 Eden Space: 17750294528 -> 0; G1 Survivor Space: 1325400064 -> 1795162112;
INFO 13:14:40 G1 Young Generation GC in 254ms. G1 Eden Space: 14092861440 -> 0; G1 Old Gen: 10509551608 -> 11563311096; G1 Survivor Space: 1795162112
INFO 13:16:33 G1 Young Generation GC in 213ms. G1 Eden Space: 16039018496 -> 0; G1 Old Gen: 11619767792 -> 12115354104; G1 Survivor Space: 1627389952
INFO 13:25:40 G1 Young Generation GC in 245ms. G1 Eden Space: 18790481920 -> 0; G1 Old Gen: 4253024752 -> 4761733616; G1 Survivor Space: 1811939328 -
INFO 13:27:38 G1 Young Generation GC in 208ms. G1 Eden Space: 13958643712 -> 0; G1 Old Gen: 5377097720 -> 6427987960; G1 Survivor Space: 1912602624 -
INFO 13:29:30 G1 Young Generation GC in 207ms. G1 Eden Space: 12264144896 -> 0; G1 Old Gen: 6427987960 -> 7337681912; G1 Survivor Space: 1795162112 -
INFO 13:33:32 G1 Young Generation GC in 202ms. G1 Eden Space: 18589155328 -> 0; G1 Old Gen: 8506316280 -> 8514704880; G1 Survivor Space: 1308622848 -
INFO 13:35:09 G1 Young Generation GC in 203ms. G1 Eden Space: 17532190720 -> 0; G1 Survivor Space: 1258291200 -> 1761607680;
INFO 13:38:33 G1 Young Generation GC in 216ms. G1 Eden Space: 11106516992 -> 0; G1 Old Gen: 11571635704 -> 12698973168; G1 Survivor Space: 1795162112
INFO 13:47:36 G1 Young Generation GC in 229ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 3825205752 -> 4303660528; G1 Survivor Space: 1526726656 -
INFO 13:53:43 G1 Young Generation GC in 207ms. G1 Eden Space: 14042529792 -> 0; G1 Old Gen: 7324985848 -> 8363582960; G1 Survivor Space: 1660944384 -
INFO 13:55:40 G1 Young Generation GC in 201ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 8355194360 -> 8820888560; G1 Survivor Space: 1728053248 -
INFO 13:56:22 G1 Young Generation GC in 206ms. G1 Eden Space: 19394461696 -> 0; G1 Survivor Space: 1207959552 -> 1711276032;
INFO 14:08:09 Saved KeyCache (4563 items) in 189 ms
INFO 14:09:31 Redistributing index summaries
INFO 14:12:47 G1 Young Generation GC in 271ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 4345299448 -> 4814148600; G1 Survivor Space: 1728053248 -
INFO 14:18:05 G1 Young Generation GC in 224ms. G1 Eden Space: 18907922432 -> 0; G1 Old Gen: 7021265392 -> 7483297784; G1 Survivor Space: 1627389952 -
INFO 14:36:05 G1 Young Generation GC in 304ms. G1 Eden Space: 18958254080 -> 0; G1 Old Gen: 3976200696 -> 4437455344; G1 Survivor Space: 1644167168 -
INFO 14:36:07 G1 Young Generation GC in 212ms. G1 Eden Space: 1711276032 -> 0; G1 Old Gen: 4437455344 -> 5830082552; G1 Survivor Space: 1459617792 ->
INFO 15:00:30 G1 Young Generation GC in 233ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4040399864 -> 4343952368; G1 Survivor Space: 1442840576 -
INFO 15:09:31 Redistributing index summaries
INFO 15:26:28 G1 Young Generation GC in 234ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 4604513264 -> 5064485872; G1 Survivor Space: 1526726656 -
INFO 15:52:08 G1 Young Generation GC in 222ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 4118806520 -> 4391313912; G1 Survivor Space: 1476395008 -
INFO 15:55:06 G1 Young Generation GC in 202ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 5267292152 -> 5559551472;
INFO 16:09:31 Redistributing index summaries
INFO 16:20:48 G1 Young Generation GC in 241ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 3854587896 -> 4287600112; G1 Survivor Space: 1476395008 -
INFO 16:31:07 G1 Young Generation GC in 403ms. G1 Eden Space: 822083584 -> 0; G1 Old Gen: 9219080688 -> 9994162168; G1 Survivor Space: 889192448 -> 2
INFO 16:33:41 G1 Young Generation GC in 426ms. G1 Eden Space: 620756992 -> 0; G1 Old Gen: 10147301880 -> 11065889792; G1 Survivor Space: 1090519040 -
INFO 16:38:17 G1 Young Generation GC in 835ms. G1 Eden Space: 251658240 -> 0; G1 Old Gen: 12062818288 -> 13404581312; G1 Survivor Space: 1459617792 -
INFO 16:43:08 G1 Young Generation GC in 569ms. G1 Eden Space: 3154116608 -> 0; G1 Old Gen: 14361297352 -> 15461740984; G1 Survivor Space: 1258291200
INFO 16:46:08 G1 Young Generation GC in 432ms. G1 Eden Space: 18706595840 -> 0; G1 Old Gen: 3256274416 -> 4355653104; G1 Survivor Space: 1023410176 -
INFO 17:09:31 Redistributing index summaries
INFO 17:26:10 G1 Young Generation GC in 744ms. G1 Eden Space: 201326592 -> 0; G1 Old Gen: 6969713640 -> 8462801376; G1 Survivor Space: 1577058304 ->
INFO 17:29:04 G1 Young Generation GC in 204ms. G1 Eden Space: 603979776 -> 0; G1 Old Gen: 8454412776 -> 9439609320; G1 Survivor Space: 1107296256 ->
INFO 17:41:09 G1 Young Generation GC in 279ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 13615267816 -> 14860613592; G1 Survivor Space: 1358954496 -
INFO 17:46:51 G1 Young Generation GC in 493ms. G1 Eden Space: 754974720 -> 0; G1 Old Gen: 3560839664 -> 4350490576; G1 Survivor Space: 956301312 -> 2
INFO 17:49:09 G1 Young Generation GC in 290ms. G1 Eden Space: 335544320 -> 0; G1 Old Gen: 4350490576 -> 5603827664; G1 Survivor Space: 1375731712 ->
INFO 17:55:17 G1 Young Generation GC in 803ms. G1 Eden Space: 201326592 -> 0; G1 Old Gen: 6543114704 -> 8036691400; G1 Survivor Space: 1577058304 ->
INFO 17:58:07 G1 Young Generation GC in 206ms. G1 Eden Space: 4764729344 -> 0; G1 Old Gen: 8036691400 -> 8934491600; G1 Survivor Space: 973078528 ->
INFO 18:07:29 G1 Young Generation GC in 306ms. G1 Eden Space: 704643072 -> 0; G1 Old Gen: 12163481536 -> 13045510592; G1 Survivor Space: 1006632960 -
INFO 18:08:09 Saved KeyCache (5307 items) in 85 ms
INFO 18:09:31 Redistributing index summaries
INFO 18:10:11 G1 Young Generation GC in 277ms. G1 Eden Space: 402653184 -> 0; G1 Old Gen: 13045510592 -> 14252493256; G1 Survivor Space: 1308622848 -
INFO 18:15:39 G1 Young Generation GC in 238ms. G1 Eden Space: 13052674048 -> 0; G1 Old Gen: 15141437888 -> 15520873416; G1 Survivor Space: 1358954496
INFO 18:20:16 G1 Young Generation GC in 540ms. G1 Eden Space: 7616856064 -> 0; G1 Old Gen: 3909694456 -> 4886048752; G1 Survivor Space: 1509949440 ->
INFO 18:23:04 G1 Young Generation GC in 221ms. G1 Eden Space: 16894656512 -> 0; G1 Old Gen: 5420089848 -> 6187322368; G1 Survivor Space: 939524096 ->
INFO 18:42:08 G1 Young Generation GC in 708ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 12840221688 -> 14085112816; G1 Survivor Space: 1358954496 -
INFO 18:50:13 G1 Young Generation GC in 318ms. G1 Eden Space: 218103808 -> 0; G1 Old Gen: 3631963120 -> 5033289720; G1 Survivor Space: 1493172224 ->
INFO 19:09:31 Redistributing index summaries
INFO 19:27:02 G1 Young Generation GC in 384ms. G1 Eden Space: 12901679104 -> 0; G1 Old Gen: 7034014192 -> 7759489528; G1 Survivor Space: 889192448 ->
INFO 19:31:25 G1 Young Generation GC in 770ms. G1 Eden Space: 218103808 -> 0; G1 Old Gen: 8506048488 -> 9921951720; G1 Survivor Space: 1493172224 ->
INFO 20:09:31 Redistributing index summaries
INFO 21:09:31 Redistributing index summaries
INFO 21:19:39 G1 Young Generation GC in 201ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 3917480432 -> 4124623344; G1 Survivor Space: 1459617792 -
INFO 22:04:05 G1 Young Generation GC in 209ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 4420796912 -> 4618141680; G1 Survivor Space: 1459617792 -
INFO 22:07:49 G1 Young Generation GC in 215ms. G1 Eden Space: 14713618432 -> 0; G1 Old Gen: 5970255352 -> 6555802608; G1 Survivor Space: 1493172224 -
INFO 22:08:09 Saved KeyCache (5845 items) in 87 ms
INFO 22:09:31 Redistributing index summaries
INFO 22:38:04 G1 Young Generation GC in 201ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 4248291312 -> 4459621368; G1 Survivor Space: 1526726656 -
INFO 22:44:12 G1 Young Generation GC in 203ms. G1 Eden Space: 10770972672 -> 0; G1 Old Gen: 6703617016 -> 7549747184; G1 Survivor Space: 1325400064 -
INFO 22:47:05 G1 Young Generation GC in 207ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 7549747184 -> 7771343352; G1 Survivor Space: 1577058304 -
INFO 23:08:59 G1 Young Generation GC in 369ms. G1 Eden Space: 15820914688 -> 0; G1 Old Gen: 4823449592 -> 5478502888; G1 Survivor Space: 1660944384 -
INFO 23:09:31 Redistributing index summaries
INFO 23:45:21 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO 23:58:40 G1 Young Generation GC in 283ms. G1 Eden Space: 18941476864 -> 0; G1 Old Gen: 4831838192 -> 5297926640; G1 Survivor Space: 1660944384 -
INFO 00:01:54 G1 Young Generation GC in 211ms. G1 Eden Space: 18891145216 -> 0; G1 Old Gen: 6425673712 -> 6912948208; G1 Survivor Space: 1711276032 -
INFO 00:05:37 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO 00:09:31 Redistributing index summaries
INFO 00:14:33 G1 Young Generation GC in 282ms. G1 Eden Space: 14344519680 -> 0; G1 Old Gen: 14159246832 -> 14677156848; G1 Survivor Space: 1207959552
INFO 00:21:32 G1 Young Generation GC in 225ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 4346926576 -> 4865982456; G1 Survivor Space: 1560281088 -
INFO 00:22:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO 00:24:31 G1 Young Generation GC in 266ms. G1 Eden Space: 19008585728 -> 0; G1 Old Gen: 6106907128 -> 6613907952; G1 Survivor Space: 1593835520 -
INFO 00:29:46 G1 Young Generation GC in 233ms. G1 Eden Space: 18236833792 -> 0; G1 Old Gen: 9076474360 -> 9498544624; G1 Survivor Space: 1795162112 -
INFO 00:32:35 G1 Young Generation GC in 226ms. G1 Eden Space: 16492003328 -> 0; G1 Old Gen: 10804527088 -> 11351712752; G1 Survivor Space: 1811939328
INFO 00:37:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO 00:41:24 G1 Young Generation GC in 330ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 3426117112 -> 3961507312; G1 Survivor Space: 1459617792 -
INFO 00:44:26 G1 Young Generation GC in 325ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 5268046320 -> 5788210120; G1 Survivor Space: 1560281088 -
INFO 00:47:13 G1 Young Generation GC in 209ms. G1 Eden Space: 19243466752 -> 0; G1 Old Gen: 7147093960 -> 7138705360; G1 Survivor Space: 1358954496 -
INFO 00:47:28 G1 Young Generation GC in 312ms. G1 Eden Space: 18605932544 -> 0; G1 Old Gen: 7138705360 -> 7710094280; G1 Survivor Space: 1996488704 -
INFO 00:50:35 G1 Young Generation GC in 330ms. G1 Eden Space: 17934843904 -> 0; G1 Old Gen: 9118416840 -> 9688820680; G1 Survivor Space: 1929379840 -
INFO 00:52:52 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO 00:56:29 G1 Young Generation GC in 251ms. G1 Eden Space: 14797504512 -> 0; G1 Old Gen: 13404995520 -> 13787049928; G1 Survivor Space: 1358954496
INFO 00:58:14 G1 Young Generation GC in 662ms. G1 Eden Space: 369098752 -> 0; G1 Old Gen: 14730395584 -> 15984921520; G1 Survivor Space: 1342177280 -
INFO 01:01:25 G1 Young Generation GC in 288ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 2618580968 -> 3127466464; G1 Survivor Space: 1560281088 -
INFO 01:04:17 G1 Young Generation GC in 201ms. G1 Eden Space: 19327352832 -> 0; G1 Old Gen: 4521459688 -> 4529848288; G1 Survivor Space: 1275068416 -
INFO 01:04:39 G1 Young Generation GC in 283ms. G1 Eden Space: 18723373056 -> 0; G1 Old Gen: 4529848288 -> 5024210408; G1 Survivor Space: 1879048192 -
INFO 01:07:23 G1 Young Generation GC in 294ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 6308233704 -> 6875745256; G1 Survivor Space: 1862270976 -
INFO 01:07:53 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO 01:09:31 Redistributing index summaries
INFO 01:10:12 G1 Young Generation GC in 213ms. G1 Eden Space: 19243466752 -> 0; G1 Old Gen: 8262779360 -> 8254390760; G1 Survivor Space: 1358954496 -
INFO 01:10:33 G1 Young Generation GC in 319ms. G1 Eden Space: 18639486976 -> 0; G1 Old Gen: 8254390760 -> 8821123552; G1 Survivor Space: 1962934272 -
INFO 01:22:55 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO 01:24:25 G1 Young Generation GC in 227ms. G1 Eden Space: 19226689536 -> 0; G1 Old Gen: 4245307376 -> 4236918776; G1 Survivor Space: 1375731712 -
INFO 01:24:48 G1 Young Generation GC in 219ms. G1 Eden Space: 15401484288 -> 0; G1 Old Gen: 4236918776 -> 5231503864; G1 Survivor Space: 2097152000 -
INFO 01:42:01 G1 Young Generation GC in 212ms. G1 Eden Space: 16005464064 -> 0; G1 Old Gen: 4320133112 -> 5100782576; G1 Survivor Space: 1627389952 -
INFO 01:48:28 G1 Young Generation GC in 209ms. G1 Eden Space: 14730395648 -> 0; G1 Old Gen: 8901994480 -> 9402393592; G1 Survivor Space: 1291845632 -
INFO 02:04:18 G1 Young Generation GC in 241ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 7825270264 -> 8327173624; G1 Survivor Space: 1560281088 -
INFO 02:08:09 Saved KeyCache (8634 items) in 94 ms
INFO 02:09:31 Redistributing index summaries
INFO 02:14:50 G1 Young Generation GC in 314ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 3505807352 -> 4140375528; G1 Survivor Space: 1526726656 -
INFO 02:48:35 G1 Young Generation GC in 321ms. G1 Eden Space: 11576279040 -> 0; G1 Old Gen: 3865890800 -> 4482588664; G1 Survivor Space: 1543503872 -
INFO 02:53:54 G1 Young Generation GC in 203ms. G1 Eden Space: 17901289472 -> 0; G1 Old Gen: 7927235056 -> 8552357368; G1 Survivor Space: 1593835520 -
INFO 03:04:33 G1 Young Generation GC in 345ms. G1 Eden Space: 13505658880 -> 0; G1 Old Gen: 3765058032 -> 4443454952; G1 Survivor Space: 1442840576 -
INFO 03:09:31 Redistributing index summaries
INFO 03:22:56 G1 Young Generation GC in 250ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 4806672880 -> 5340134904; G1 Survivor Space: 1610612736 -
INFO 03:35:13 G1 Young Generation GC in 210ms. G1 Eden Space: 14445182976 -> 0; G1 Survivor Space: 1073741824 -> 1744830464;
INFO 03:38:42 G1 Young Generation GC in 214ms. G1 Eden Space: 18303942656 -> 0; G1 Old Gen: 4786540536 -> 5432116720; G1 Survivor Space: 1375731712 -
INFO 03:52:11 G1 Young Generation GC in 207ms. G1 Eden Space: 117440512 -> 0; G1 Old Gen: 12054831088 -> 10154180600; G1 Survivor Space: 1593835520 -
INFO 03:54:48 G1 Young Generation GC in 224ms. G1 Eden Space: 16156459008 -> 0; G1 Old Gen: 4303355896 -> 4801558520; G1 Survivor Space: 1560281088 -
INFO 04:09:31 Redistributing index summaries
INFO 04:16:09 G1 Young Generation GC in 216ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 5853311984 -> 6366188536; G1 Survivor Space: 1459617792 -
INFO 04:19:13 G1 Young Generation GC in 229ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 7826571256 -> 8457920504; G1 Survivor Space: 1560281088 -
INFO 04:27:41 G1 Young Generation GC in 311ms. G1 Eden Space: 13723762688 -> 0; G1 Old Gen: 14176747504 -> 14808230392; G1 Survivor Space: 1509949440
INFO 04:29:56 G1 Young Generation GC in 274ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 3376452080 -> 3818638328; G1 Survivor Space: 1459617792 -
INFO 04:33:40 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
INFO 04:49:07 G1 Young Generation GC in 319ms. G1 Eden Space: 16005464064 -> 0; G1 Old Gen: 4435714040 -> 5023449072; G1 Survivor Space: 1442840576 -
INFO 04:54:06 G1 Young Generation GC in 208ms. G1 Eden Space: 18706595840 -> 0; G1 Old Gen: 7843348472 -> 8466889208; G1 Survivor Space: 1476395008 -
INFO 05:09:31 Redistributing index summaries
I found INFO Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB, does it mean the memory is not enough ?
I want to know why adding a new node spent so much time and what need I do to complete a new node joining the cluster ?
I need some help and really need some suggestions.
Thanks a lot!
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
Re: [EXTERNAL] Cassandra cluster add new node slowly
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
>
> I suspect the compactionthroughput has an influence on the new node
> joining. The command nodetool | getcompactionthroughput says 'Current
> compaction throughput: 32 MB/s’.
I would say this guess is true, but maybe not the way you think: the more
disk IO you use for compactions, the slower the stream will be (if joining
machine is CPU / Disk IO bounded). When using vnodes, all the node will be
sending data to this new node. Often joining node are sticking at 100 % CPU
or disk usage (or both). This node is considered as an extra replica until
it is up and is not read from, so it is not a big deal.
Thus it is possible to tune compaction to be faster without risk while the
node is joining, but it might have the opposite effect and slow down the
streaming process as it will use more resources that are already a
bottleneck during bootstrap.
Be aware that when the node join you could have performance issues if
compactions are taking too much resources or if you have too many sstables
(the opposite situation were compaction was running too slow). It's good to
find a balance between the streaming speed and what the node can cope with
so when the node joins the ring it does it in a healthy state (acceptable
state at least).
Often I observed that the streaming speed is substantially slower at the
end of the bootstrap as only a few (and eventually just one) nodes are
sending the data while other nodes are done with the streaming. In this
phase, compactions can catch up, the disk space used and the number of
SSTable is greatly reduced, allowing the node to join in good conditions.
C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France / Spain
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2018-01-04 6:00 GMT+00:00 Anthony Grasso <an...@gmail.com>:
> The speed at which compactions operate is also physically restricted by
> the speed of the disk. If the disks used on the new node are HDDs, then
> increasing the compaction throughput will be of little help. However, if
> the disks on the new node are SSDs then increasing the compaction
> throughput to at least 64MB/s should help speed up compactions.
>
> Regards,
> Anthony
>
> On 4 January 2018 at 14:13, qf zhou <zh...@gmail.com> wrote:
>
>> The cassandra version is 3.0.9.
>>
>> I have changed the heap size (about 32G). Also, the streaming
>> throughput is set 800MB/sec, and the streaming_socket_timeout_in_ms is
>> default 86400000.
>> I suspect the compactionthroughput has an influence on the new node
>> joining. The command nodetool | getcompactionthroughput says
>> 'Current compaction throughput: 32 MB/s’.
>>
>>
>>
>>
>> 在 2018年1月4日，上午4:59，Durity, Sean R <SE...@homedepot.com> 写道：
>>
>> You don’t mention the version, but here are some general suggestions
>>
>> - 2 GB heap is very small for a node, especially with 1 TB+ of
>> data. What is the physical RAM on the host? In general, you want ½ of
>> physical RAM for the JVM. (Look in jvm.options or cassandra-env.sh)
>> - You can change the streaming throughput from the existing
>> nodes, if it looks like the new node can handle it. Look at nodetool
>> setstreamthroughput. Default is 200 (MB/sec).
>> - You might want to check for a streaming_socket_timeout_in_ms.
>> This has changed over the versions. Some details are at:
>> https://issues.apache.org/jira/browse/CASSANDRA-11839. 24 hours is good
>> recommendation.
>> - If your new node can’t compact fast enough to keep disk usage
>> down, look at compactionthroughput on that node
>> - nodetool netstats | grep –v “100%” is a good way to see what
>> is happening/if anything is stuck. Newer versions give a bit more info on
>> progress.
>> - Don’t forget to run cleanup on existing nodes after the new
>> nodes are added.
>>
>>
>>
>> Sean Durity
>> *From:* qf zhou [mailto:zhouqf2013@gmail.com <zh...@gmail.com>]
>> *Sent:* Tuesday, January 02, 2018 10:30 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] Cassandra cluster add new node slowly
>>
>> The cluster has 3 nodes, and the data in each node is about 1.2 T. I
>> want to add two new nodes to expand the cluster.
>>
>> Following the instructions from the datastax website, ie, (
>> http://docs.datastax.com/en/archived/cassandra/3.x/cassand
>> ra/operations/opsAddNodeToCluster.html
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_archived_cassandra_3.x_cassandra_operations_opsAddNodeToCluster.html&d=DwMFoQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=AfP0BxfTG1o_-2nRtGSjngs45YndH060h39eVlypVso&s=AaiOypurMtWA470038fHMbaXJPIZeYtLzv3Yt9vf1Sc&e=>),
>>
>>
>> I try to add one node to the cluster. However, it is too slow and
>> time cost too much. After about 24 hours, it still didn’t success.
>>
>> I run the command: nodetool netstats on the new node, it shows that:
>>
>> (tb1fullwithstate2 is a big table and 90% of the cluster data is in
>> it. Here I use CompactionStrategy: TimeWindowCompactionStrategy).
>>
>> /*.*.*.3
>> Receiving 136 files, 328573794609 bytes total. Already received 8
>> files, 5774621188 <(577)%20462-1188> bytes total
>> tb1/tb1fullwithneweststatetest 3758271/3758271 bytes(100%)
>> received from idx:0/*.*.*.3
>> system_distributed/repair_history 57534/57534 bytes(100%)
>> received from idx:0/*.*.*.3
>> system_distributed/parent_repair_history 507660/507660
>> bytes(100%) received from idx:0/*.*.*.3
>> tb1/tb1_device_last_state_eachday 15754096/15754096
>> bytes(100%) received from idx:0/*.*.*.3
>> mytest1/tb1_test1 8143775/8143775 bytes(100%) received from
>> idx:0/*.*.*.3
>> tb1/tb1fullwithstate 2251191007/2251191007 bytes(100%)
>> received from idx:0/*.*.*.3
>> applocationinfo/weiyirong_app 2760/2760 bytes(100%) received
>> from idx:0/*.*.*.3
>> tb1/tb1fullwithstate2 3490748006/4909554503 bytes(71%)
>> received from idx:0/*.*.*.3
>> tb1/tb1fullwithneweststate 4458079/4458079 bytes(100%)
>> received from idx:0/*.*.*.3
>> /*.*.*.2
>> Receiving 136 files, 336762487360 bytes total. Already received 3
>> files, 5695770181 bytes total
>> system_distributed/repair_history 31684/31684 bytes(100%)
>> received from idx:0/*.*.*.2
>> tb1/tb1fullwithstate 908260516/908260516 bytes(100%) received
>> from idx:0/*.*.*.2
>> tb1/tb1fullwithstate2 4783622958/4990450588 bytes(95%)
>> received from idx:0/*.*.*.2
>> tb1/tb1fullwithneweststate 3855023/3855023 bytes(100%)
>> received from idx:0/*.*.*.2
>> /*.*.*.4
>> Receiving 132 files, 236250553620 bytes total. Already received
>> 10 files, 3117465128 bytes total
>> mytest1/wordstest2 46/46 bytes(100%) received from
>> idx:0/*.*.*.4
>> tb1/tb1fullwithneweststatetest 3416891/3416891 bytes(100%)
>> received from idx:0/*.*.*.4
>> system_distributed/repair_history 39720/39720 bytes(100%)
>> received from idx:0/*.*.*.4
>> system_distributed/parent_repair_history 452250/452250
>> bytes(100%) received from idx:0/*.*.*.4
>> mytest1/weblogs 104/104 bytes(100%) received from
>> idx:0/*.*.*.4
>> tb1/tb1_device_last_state_eachday 12670998/12670998
>> bytes(100%) received from idx:0/*.*.*.4
>> mytest1/tb1_test1 3257952/3257952 bytes(100%) received from
>> idx:0/*.*.*.4
>> tb1/tb1fullwithstate 647702056/647702056 bytes(100%) received
>> from idx:0/*.*.*.4
>> applocationinfo/weiyirong_app 3509/3509 bytes(100%) received
>> from idx:0/*.*.*.4
>> tb1/tb1fullwithstate2 2446436305/3566562762 bytes(68%)
>> received from idx:0/*.*.*.4
>> tb1/tb1fullwithneweststate 3485297/3485297 bytes(100%)
>> received from idx:0/*.*.*.4
>>
>>
>>
>> check the log in the logs/system.log, it shows that:
>>
>>
>> INFO 06:09:33 Updating topology for /*.*.*.2
>> INFO 06:09:33 Updating topology for /*.*.*.2
>> INFO 06:09:33 Node /*.*.*.3 has restarted, now UP
>> INFO 06:09:33 InetAddress /*.*.*.3 is now UP
>> INFO 06:09:33 Node /*.*.*.3 state jump to NORMAL
>> INFO 06:09:33 Updating topology for /*.*.*.3
>> INFO 06:09:33 Updating topology for /*.*.*.3
>> INFO 06:09:33 Handshaking version with /*.*.*.2
>> INFO 06:09:33 Handshaking version with /*.*.*.3
>> INFO 06:09:36 Handshaking version with /*.*.*.4
>> INFO 06:10:03 JOINING: Starting to bootstrap...
>> INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Executing
>> streaming plan for Bootstrap
>> INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting
>> streaming to /*.*.*.4
>> INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0]
>> Beginning stream session with /*.*.*.4
>> INFO 06:10:04 Deleted hint file c5271e74-19a1-4cee-98d7-dc169c
>> f87e95-1514527790874-1.hints
>> INFO 06:10:04 Finished hinted handoff of file
>> c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints to endpoint
>> c5271e74-19a1-4cee-98d7-dc169cf87
>> INFO 06:10:11 Deleted hint file c5271e74-19a1-4cee-98d7-dc169c
>> f87e95-1514527800873-1.hints
>> INFO 06:10:11 Finished hinted handoff of file
>> c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints to endpoint
>> c5271e74-19a1-4cee-98d7-dc169cf87
>> INFO 06:10:12 Deleted hint file c555e44c-9590-4f45-aea4-f5eca6
>> 8180b2-1514527790881-1.hints
>> INFO 06:10:12 Finished hinted handoff of file
>> c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints to endpoint
>> c555e44c-9590-4f45-aea4-f5eca6818
>> INFO 06:10:20 Deleted hint file c555e44c-9590-4f45-aea4-f5eca6
>> 8180b2-1514527800880-1.hints
>> INFO 06:10:20 Finished hinted handoff of file
>> c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints to endpoint
>> c555e44c-9590-4f45-aea4-f5eca6818
>> INFO 06:10:21 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115
>> 719d59-1514527790884-1.hints
>> INFO 06:10:21 Finished hinted handoff of file
>> c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints to endpoint
>> c623bbc0-9839-4d2d-8ff3-db7115719
>> INFO 06:10:28 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115
>> 719d59-1514527800887-1.hints
>> INFO 06:10:28 Finished hinted handoff of file
>> c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints to endpoint
>> c623bbc0-9839-4d2d-8ff3-db7115719
>> INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0]
>> Prepare completed. Receiving 132 files(220.025GiB), sending 0
>> files(0.000KiB)
>> INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting
>> streaming to /*.*.*.2
>> INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0]
>> Beginning stream session with /*.*.*.2
>> INFO 06:10:34 Deleted hint file c5271e74-19a1-4cee-98d7-dc169c
>> f87e95-1514527810873-1.hints
>> INFO 06:10:34 Finished hinted handoff of file
>> c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints to endpoint
>> c5271e74-19a1-4cee-98d7-dc169cf87
>> INFO 06:10:42 Deleted hint file c5271e74-19a1-4cee-98d7-dc169c
>> f87e95-1514527820873-1.hints
>> INFO 06:10:42 Finished hinted handoff of file
>> c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints to endpoint
>> c5271e74-19a1-4cee-98d7-dc169cf87
>> INFO 06:10:58 Deleted hint file c555e44c-9590-4f45-aea4-f5eca6
>> 8180b2-1514527810881-1.hints
>> INFO 06:10:58 Finished hinted handoff of file
>> c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints to endpoint
>> c555e44c-9590-4f45-aea4-f5eca6818
>> INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0]
>> Prepare completed. Receiving 136 files(313.635GiB), sending 0
>> files(0.000KiB)
>> INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting
>> streaming to /*.*.*.3
>> INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0]
>> Beginning stream session with /*.*.*.3
>> INFO 06:11:06 Deleted hint file c555e44c-9590-4f45-aea4-f5eca6
>> 8180b2-1514527830880-1.hints
>> INFO 06:11:06 Finished hinted handoff of file
>> c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints to endpoint
>> c555e44c-9590-4f45-aea4-f5eca6818
>> INFO 06:11:29 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115
>> 719d59-1514527810889-1.hints
>> INFO 06:11:29 Finished hinted handoff of file
>> c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints to endpoint
>> c623bbc0-9839-4d2d-8ff3-db7115719
>> INFO 06:11:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0]
>> Prepare completed. Receiving 136 files(306.008GiB), sending 0
>> files(0.000KiB)
>> INFO 06:11:36 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115
>> 719d59-1514527840887-1.hints
>> INFO 06:11:36 Finished hinted handoff of file
>> c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints to endpoint
>> c623bbc0-9839-4d2d-8ff3-db7115719
>> INFO 06:11:49 Deleted hint file c5271e74-19a1-4cee-98d7-dc169c
>> f87e95-1514527830873-1.hints
>> INFO 06:11:49 Finished hinted handoff of file
>> c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints to endpoint
>> c5271e74-19a1-4cee-98d7-dc169cf87
>> INFO 06:11:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169c
>> f87e95-1514527850874-1.hints
>> INFO 06:11:56 Finished hinted handoff of file
>> c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints to endpoint
>> c5271e74-19a1-4cee-98d7-dc169cf87
>> INFO 06:12:26 Deleted hint file c555e44c-9590-4f45-aea4-f5eca6
>> 8180b2-1514527840880-1.hints
>> INFO 06:12:26 Finished hinted handoff of file
>> c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints to endpoint
>> c555e44c-9590-4f45-aea4-f5eca6818
>> INFO 06:12:45 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115
>> 719d59-1514527850883-1.hints
>> INFO 06:12:45 Finished hinted handoff of file
>> c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints to endpoint
>> c623bbc0-9839-4d2d-8ff3-db7115719
>> INFO 06:12:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169c
>> f87e95-1514527860874-1.hints
>> INFO 06:12:56 Finished hinted handoff of file
>> c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints to endpoint
>> c5271e74-19a1-4cee-98d7-dc169cf87
>> INFO 06:24:03 G1 Young Generation GC in 242ms. G1 Eden Space:
>> 18924699648 -> 0; G1 Old Gen: 4360876536 -> 5095617016; G1 Survivor Space:
>> 1677721600 -
>> INFO 06:26:06 G1 Young Generation GC in 215ms. G1 Eden Space:
>> 18220056576 -> 0; G1 Old Gen: 5981077496 -> 6597610488; G1 Survivor Space:
>> 1493172224 -
>> INFO 06:29:04 G1 Young Generation GC in 225ms. G1 Eden Space:
>> 18538823680 -> 0; G1 Old Gen: 7910457840 -> 8860639728; G1 Survivor Space:
>> 1895825408 -
>> INFO 06:33:22 G1 Young Generation GC in 214ms. G1 Eden Space:
>> 16559112192 -> 0; G1 Old Gen: 11868321784 -> 11941600752; G1 Survivor
>> Space: 1241513984
>> INFO 06:33:33 G1 Young Generation GC in 205ms. G1 Eden Space:
>> 8069840896 <(806)%20984-0896> -> 0; G1 Old Gen: 11941600752 ->
>> 13683606520; G1 Survivor Space: 1895825408
>> INFO 06:41:01 G1 Young Generation GC in 383ms. G1 Eden Space:
>> 18874368000 -> 0; G1 Old Gen: 4770399736 -> 5551887344; G1 Survivor Space:
>> 1728053248 -
>> INFO 06:45:07 G1 Young Generation GC in 226ms. G1 Eden Space:
>> 19025362944 <(902)%20536-2944> -> 0; G1 Old Gen: 8036286448
>> <(803)%20628-6448> -> 8739698672 <(873)%20969-8672>; G1 Survivor Space:
>> 1577058304 -
>> INFO 06:47:11 G1 Young Generation GC in 203ms. G1 Eden Space:
>> 18538823680 -> 0; G1 Survivor Space: 1191182336 -> 1778384896;
>> INFO 06:47:22 G1 Young Generation GC in 241ms. G1 Eden Space:
>> 8086618112 <(808)%20661-8112> -> 0; G1 Old Gen: 9999220720 ->
>> 11591876088; G1 Survivor Space: 1778384896 -
>> INFO 06:49:43 G1 Young Generation GC in 201ms. G1 Eden Space:
>> 16022241280 <(602)%20224-1280> -> 0; G1 Old Gen: 11591876088 ->
>> 12084766712 <(208)%20476-6712>; G1 Survivor Space: 1644167168
>> INFO 06:53:21 G1 Young Generation GC in 322ms. G1 Eden Space:
>> 13237223424 <(323)%20722-3424> -> 0; G1 Old Gen: 14755561464
>> <(475)%20556-1464> -> 15445886448 <(544)%20588-6448>; G1 Survivor Space:
>> 1459617792
>> INFO 06:55:34 G1 Young Generation GC in 343ms. G1 Eden Space:
>> 16525557760 -> 0; G1 Old Gen: 2939907056 -> 3636770792; G1 Survivor Space:
>> 1610612736 -
>> INFO 07:09:30 Redistributing index summaries
>> INFO 07:14:25 G1 Young Generation GC in 270ms. G1 Eden Space:
>> 15602810880 -> 0; G1 Old Gen: 4360518136 -> 5243274224; G1 Survivor Space:
>> 1610612736 -
>> INFO 07:16:51 G1 Young Generation GC in 229ms. G1 Eden Space:
>> 19008585728 <(900)%20858-5728> -> 0; G1 Old Gen: 6132072440 ->
>> 6815145976; G1 Survivor Space: 1593835520 -
>> INFO 07:19:34 G1 Young Generation GC in 215ms. G1 Eden Space:
>> 18991808512 -> 0; G1 Old Gen: 8689337336 -> 9071161328; G1 Survivor Space:
>> 1610612736 -
>> INFO 07:23:35 G1 Young Generation GC in 293ms. G1 Eden Space:
>> 15753805824 <(575)%20380-5824> -> 0; G1 Old Gen: 11777605616 ->
>> 12289649640; G1 Survivor Space: 1694498816
>> INFO 07:29:42 G1 Young Generation GC in 362ms. G1 Eden Space:
>> 15955132416 -> 0; G1 Old Gen: 3742637560 -> 4410361824; G1 Survivor Space:
>> 1644167168 -
>> INFO 07:35:12 G1 Young Generation GC in 208ms. G1 Eden Space:
>> 19478347776 <(947)%20834-7776> -> 0; G1 Old Gen: 7199737832 ->
>> 7208126432; G1 Survivor Space: 1124073472 -
>> INFO 07:35:33 G1 Young Generation GC in 219ms. G1 Eden Space:
>> 18740150272 -> 0; G1 Old Gen: 7208126432 <(720)%20812-6432> ->
>> 8072478184; G1 Survivor Space: 1862270976 -
>> INFO 07:45:19 G1 Young Generation GC in 213ms. G1 Eden Space:
>> 13153337344 <(315)%20333-7344> -> 0; G1 Old Gen: 15418261472 ->
>> 16123805152; G1 Survivor Space: 1174405120
>> INFO 07:48:08 G1 Young Generation GC in 335ms. G1 Eden Space:
>> 18387828736 <(838)%20782-8736> -> 0; G1 Old Gen: 3852453368 ->
>> 4464797168; G1 Survivor Space: 1644167168 -
>> INFO 07:48:08 G1 Young Generation GC in 209ms. G1 Eden Space: 352321536
>> -> 0; G1 Old Gen: 4464797168 -> 5989466096; G1 Survivor Space: 1476395008
>> ->
>> INFO 07:50:25 G1 Young Generation GC in 221ms. G1 Eden Space:
>> 17498636288 -> 0; G1 Old Gen: 5981077496 -> 6488668656; G1 Survivor Space:
>> 1593835520 -
>> INFO 08:03:31 G1 Young Generation GC in 748ms. G1 Eden Space:
>> 8673820672 <(867)%20382-0672> -> 0; G1 Old Gen: 4386367472 ->
>> 5874747352; G1 Survivor Space: 1627389952 ->
>> INFO 08:09:30 Redistributing index summaries
>> INFO 08:10:05 G1 Young Generation GC in 213ms. G1 Eden Space:
>> 18220056576 -> 0; G1 Old Gen: 9276024272 -> 9969348048; G1 Survivor Space:
>> 1711276032 -
>> INFO 08:18:59 G1 Young Generation GC in 228ms. G1 Eden Space:
>> 17347641344 <(734)%20764-1344> -> 0; G1 Old Gen: 4546626040 ->
>> 5180788216; G1 Survivor Space: 1593835520 -
>> INFO 08:33:12 G1 Young Generation GC in 207ms. G1 Eden Space:
>> 19411238912 -> 0; G1 Survivor Space: 1191182336 -> 1828716544;
>> INFO 08:33:13 G1 Young Generation GC in 273ms. G1 Eden Space: 452984832
>> -> 0; G1 Old Gen: 4668340728 -> 6320276984; G1 Survivor Space: 1828716544
>> ->
>> INFO 08:41:59 G1 Young Generation GC in 202ms. G1 Eden Space:
>> 19025362944 <(902)%20536-2944> -> 0; G1 Old Gen: 8701765104 ->
>> 8927144944; G1 Survivor Space: 1577058304 -
>> INFO 08:56:31 G1 Young Generation GC in 205ms. G1 Eden Space:
>> 11072962560 -> 0; G1 Old Gen: 9563013104 <(956)%20301-3104> ->
>> 11068430328; G1 Survivor Space: 1660944384
>> INFO 09:05:49 G1 Young Generation GC in 367ms. G1 Eden Space:
>> 18522046464 -> 0; G1 Old Gen: 4009754608 -> 4732347872; G1 Survivor Space:
>> 1778384896 -
>> INFO 09:09:30 Redistributing index summaries
>> INFO 09:14:52 G1 Young Generation GC in 213ms. G1 Eden Space:
>> 15267266560 -> 0; G1 Old Gen: 12155092968 <(215)%20509-2968> ->
>> 12906235360; G1 Survivor Space: 1744830464
>> INFO 09:17:23 G1 Young Generation GC in 203ms. G1 Eden Space:
>> 14159970304 <(415)%20997-0304> -> 0; G1 Old Gen: 14779568608 ->
>> 14771180008; G1 Survivor Space: 989855744
>> INFO 09:17:45 G1 Young Generation GC in 204ms. G1 Eden Space:
>> 12616466432 -> 0; G1 Old Gen: 14771180008 -> 15500684768; G1 Survivor
>> Space: 1761607680
>> INFO 09:20:52 G1 Young Generation GC in 361ms. G1 Eden Space:
>> 19025362944 <(902)%20536-2944> -> 0; G1 Old Gen: 4351990768 ->
>> 5086824424; G1 Survivor Space: 1577058304 -
>> INFO 09:23:51 G1 Young Generation GC in 207ms. G1 Eden Space:
>> 18891145216 -> 0; G1 Old Gen: 7001879024 -> 7716560880; G1 Survivor Space:
>> 1711276032 -
>> INFO 09:29:20 G1 Young Generation GC in 202ms. G1 Eden Space:
>> 17028874240 <(702)%20887-4240> -> 0; G1 Survivor Space: 1040187392 ->
>> 1828716544;
>> INFO 09:29:40 G1 Young Generation GC in 219ms. G1 Eden Space:
>> 15451815936 -> 0; G1 Old Gen: 11802771944 -> 12574988776; G1 Survivor
>> Space: 1828716544
>> INFO 09:44:51 G1 Young Generation GC in 202ms. G1 Eden Space:
>> 18001952768 -> 0; G1 Old Gen: 9325481456 -> 10085596664; G1 Survivor Space:
>> 1795162112
>> INFO 09:45:23 G1 Young Generation GC in 209ms. G1 Eden Space:
>> 9177137152 <(917)%20713-7152> -> 0; G1 Old Gen: 10093985264 ->
>> 12309118960; G1 Survivor Space: 1677721600
>> INFO 09:50:33 G1 Young Generation GC in 201ms. G1 Eden Space:
>> 12901679104 -> 0; G1 Old Gen: 14487498224 -> 15268560880; G1 Survivor
>> Space: 1761607680
>> INFO 10:08:09 Saved KeyCache (2989 items) in 92 ms
>> INFO 10:09:30 Redistributing index summaries
>> INFO 10:22:17 G1 Young Generation GC in 216ms. G1 Eden Space:
>> 19495124992 <(949)%20512-4992> -> 0; G1 Old Gen: 7742571504 ->
>> 7734182904; G1 Survivor Space: 1107296256 -
>> INFO 10:25:31 G1 Young Generation GC in 206ms. G1 Eden Space:
>> 18622709760 <(862)%20270-9760> -> 0; G1 Survivor Space: 1040187392 ->
>> 1711276032;
>> INFO 10:34:54 G1 Young Generation GC in 251ms. G1 Eden Space:
>> 18991808512 -> 0; G1 Old Gen: 5066719736 <(506)%20671-9736> ->
>> 5691775992; G1 Survivor Space: 1610612736 -
>> INFO 10:40:08 G1 Young Generation GC in 209ms. G1 Eden Space:
>> 19025362944 <(902)%20536-2944> -> 0; G1 Old Gen: 9000976880 ->
>> 9090653168; G1 Survivor Space: 1442840576 -
>> INFO 10:40:08 G1 Young Generation GC in 223ms. G1 Eden Space: 469762048
>> -> 0; G1 Old Gen: 9090653168 -> 11140124152; G1 Survivor Space: 1862270976
>> ->
>> INFO 10:52:15 G1 Young Generation GC in 219ms. G1 Eden Space:
>> 15820914688 -> 0; G1 Old Gen: 5331537912 -> 5828939768; G1 Survivor Space:
>> 1275068416 -
>> INFO 10:55:30 G1 Young Generation GC in 207ms. G1 Eden Space:
>> 15804137472 <(580)%20413-7472> -> 0; G1 Old Gen: 7501364216 ->
>> 8001282032; G1 Survivor Space: 1459617792 -
>> INFO 11:07:57 G1 Young Generation GC in 295ms. G1 Eden Space:
>> 19159580672 <(915)%20958-0672> -> 0; G1 Old Gen: 4301379064 ->
>> 4919709168; G1 Survivor Space: 1442840576 -
>> INFO 11:09:30 Redistributing index summaries
>> INFO 11:13:06 G1 Young Generation GC in 204ms. G1 Eden Space:
>> 19126026240 <(912)%20602-6240> -> 0; G1 Old Gen: 8044675064 ->
>> 8659737080; G1 Survivor Space: 1476395008 -
>> INFO 11:27:07 G1 Young Generation GC in 202ms. G1 Eden Space:
>> 19159580672 <(915)%20958-0672> -> 0; G1 Old Gen: 4328521712 ->
>> 4905718776; G1 Survivor Space: 1442840576 -
>> INFO 11:30:08 G1 Young Generation GC in 209ms. G1 Eden Space:
>> 19243466752 -> 0; G1 Survivor Space: 1358954496 -> 1828716544;
>> INFO 11:30:25 G1 Young Generation GC in 226ms. G1 Eden Space:
>> 14277410816 -> 0; G1 Old Gen: 6349296624 -> 7552639472; G1 Survivor Space:
>> 1828716544 -
>> INFO 11:32:30 G1 Young Generation GC in 215ms. G1 Eden Space:
>> 19142803456 <(914)%20280-3456> -> 0; G1 Survivor Space: 1459617792 ->
>> 2046820352;
>> INFO 11:33:01 G1 Young Generation GC in 247ms. G1 Eden Space:
>> 17263755264 <(726)%20375-5264> -> 0; G1 Old Gen: 7544250872 ->
>> 8698295280; G1 Survivor Space: 2046820352 -
>> INFO 11:46:29 G1 Young Generation GC in 355ms. G1 Eden Space:
>> 17230200832 -> 0; G1 Old Gen: 4769358328 -> 5362416624; G1 Survivor Space:
>> 1761607680 -
>> INFO 11:49:18 G1 Young Generation GC in 233ms. G1 Eden Space:
>> 17867735040 <(786)%20773-5040> -> 0; G1 Old Gen: 6752829432 ->
>> 6944805360; G1 Survivor Space: 1509949440 -
>> INFO 11:52:24 G1 Young Generation GC in 209ms. G1 Eden Space:
>> 19193135104 <(919)%20313-5104> -> 0; G1 Survivor Space: 1409286144 ->
>> 1946157056;
>> INFO 11:52:47 G1 Young Generation GC in 338ms. G1 Eden Space:
>> 18287165440 <(828)%20716-5440> -> 0; G1 Old Gen: 8740881904 ->
>> 9340426216; G1 Survivor Space: 1946157056 -
>> INFO 12:04:08 G1 Young Generation GC in 810ms. G1 Eden Space:
>> 18119393280 -> 0; G1 Old Gen: 3471267832 -> 5003518440; G1 Survivor Space:
>> 1325400064 -
>> INFO 12:06:28 G1 Young Generation GC in 345ms. G1 Eden Space:
>> 8673820672 <(867)%20382-0672> -> 0; G1 Old Gen: 5318377952 ->
>> 6577762784; G1 Survivor Space: 1409286144 ->
>> INFO 12:09:31 G1 Young Generation GC in 204ms. G1 Eden Space:
>> 19159580672 <(915)%20958-0672> -> 0; G1 Old Gen: 6577762784 ->
>> 6586151384; G1 Survivor Space: 1442840576 -
>> INFO 12:09:31 Redistributing index summaries
>> INFO 12:10:01 G1 Young Generation GC in 298ms. G1 Eden Space:
>> 17230200832 -> 0; G1 Old Gen: 6586151384 -> 7710758368; G1 Survivor Space:
>> 1946157056 -
>> INFO 12:12:37 G1 Young Generation GC in 214ms. G1 Eden Space:
>> 18773704704 <(877)%20370-4704> -> 0; G1 Old Gen: 8380219864 ->
>> 8919123416; G1 Survivor Space: 1728053248 -
>> INFO 12:24:54 G1 Young Generation GC in 299ms. G1 Eden Space:
>> 16055795712 <(605)%20579-5712> -> 0; G1 Old Gen: 4375908856 ->
>> 5009812976; G1 Survivor Space: 1392508928 -
>> INFO 12:28:34 G1 Young Generation GC in 485ms. G1 Eden Space:
>> 13304332288 <(330)%20433-2288> -> 0; G1 Old Gen: 6481607672 ->
>> 7506585072; G1 Survivor Space: 1979711488 -
>> INFO 12:44:52 G1 Young Generation GC in 261ms. G1 Eden Space:
>> 18773704704 <(877)%20370-4704> -> 0; G1 Old Gen: 4239590896 ->
>> 4692442608; G1 Survivor Space: 1828716544 -
>> INFO 12:48:37 G1 Young Generation GC in 251ms. G1 Eden Space:
>> 13656653824 <(365)%20665-3824> -> 0; G1 Old Gen: 6719275504 ->
>> 7776254960; G1 Survivor Space: 1862270976 -
>> INFO 12:53:47 G1 Young Generation GC in 217ms. G1 Eden Space:
>> 17548967936 <(754)%20896-7936> -> 0; G1 Old Gen: 9667142640 ->
>> 10202159608; G1 Survivor Space: 1845493760
>> INFO 13:05:40 G1 Young Generation GC in 287ms. G1 Eden Space:
>> 18320719872 -> 0; G1 Old Gen: 5298861040 -> 5826686456; G1 Survivor Space:
>> 1862270976 -
>> INFO 13:09:31 Redistributing index summaries
>> INFO 13:11:44 G1 Young Generation GC in 268ms. G1 Eden Space:
>> 10871635968 -> 0; G1 Old Gen: 8799649784 -> 9797878264; G1 Survivor Space:
>> 1811939328 -
>> INFO 13:14:22 G1 Young Generation GC in 202ms. G1 Eden Space:
>> 17750294528 -> 0; G1 Survivor Space: 1325400064 -> 1795162112;
>> INFO 13:14:40 G1 Young Generation GC in 254ms. G1 Eden Space:
>> 14092861440 <(409)%20286-1440> -> 0; G1 Old Gen: 10509551608 ->
>> 11563311096; G1 Survivor Space: 1795162112
>> INFO 13:16:33 G1 Young Generation GC in 213ms. G1 Eden Space:
>> 16039018496 <(603)%20901-8496> -> 0; G1 Old Gen: 11619767792 ->
>> 12115354104; G1 Survivor Space: 1627389952
>> INFO 13:25:40 G1 Young Generation GC in 245ms. G1 Eden Space:
>> 18790481920 -> 0; G1 Old Gen: 4253024752 <(425)%20302-4752> ->
>> 4761733616; G1 Survivor Space: 1811939328 -
>> INFO 13:27:38 G1 Young Generation GC in 208ms. G1 Eden Space:
>> 13958643712 -> 0; G1 Old Gen: 5377097720 -> 6427987960; G1 Survivor Space:
>> 1912602624 -
>> INFO 13:29:30 G1 Young Generation GC in 207ms. G1 Eden Space:
>> 12264144896 <(226)%20414-4896> -> 0; G1 Old Gen: 6427987960 ->
>> 7337681912; G1 Survivor Space: 1795162112 -
>> INFO 13:33:32 G1 Young Generation GC in 202ms. G1 Eden Space:
>> 18589155328 <(858)%20915-5328> -> 0; G1 Old Gen: 8506316280 ->
>> 8514704880; G1 Survivor Space: 1308622848 -
>> INFO 13:35:09 G1 Young Generation GC in 203ms. G1 Eden Space:
>> 17532190720 -> 0; G1 Survivor Space: 1258291200 -> 1761607680;
>> INFO 13:38:33 G1 Young Generation GC in 216ms. G1 Eden Space:
>> 11106516992 -> 0; G1 Old Gen: 11571635704 -> 12698973168; G1 Survivor
>> Space: 1795162112
>> INFO 13:47:36 G1 Young Generation GC in 229ms. G1 Eden Space:
>> 19075694592 <(907)%20569-4592> -> 0; G1 Old Gen: 3825205752 ->
>> 4303660528; G1 Survivor Space: 1526726656 -
>> INFO 13:53:43 G1 Young Generation GC in 207ms. G1 Eden Space:
>> 14042529792 <(404)%20252-9792> -> 0; G1 Old Gen: 7324985848 ->
>> 8363582960; G1 Survivor Space: 1660944384 -
>> INFO 13:55:40 G1 Young Generation GC in 201ms. G1 Eden Space:
>> 18874368000 -> 0; G1 Old Gen: 8355194360 <(835)%20519-4360> ->
>> 8820888560; G1 Survivor Space: 1728053248 -
>> INFO 13:56:22 G1 Young Generation GC in 206ms. G1 Eden Space:
>> 19394461696 <(939)%20446-1696> -> 0; G1 Survivor Space: 1207959552 ->
>> 1711276032;
>> INFO 14:08:09 Saved KeyCache (4563 items) in 189 ms
>> INFO 14:09:31 Redistributing index summaries
>> INFO 14:12:47 G1 Young Generation GC in 271ms. G1 Eden Space:
>> 18874368000 -> 0; G1 Old Gen: 4345299448 <(434)%20529-9448> ->
>> 4814148600; G1 Survivor Space: 1728053248 -
>> INFO 14:18:05 G1 Young Generation GC in 224ms. G1 Eden Space:
>> 18907922432 -> 0; G1 Old Gen: 7021265392 -> 7483297784; G1 Survivor Space:
>> 1627389952 -
>> INFO 14:36:05 G1 Young Generation GC in 304ms. G1 Eden Space:
>> 18958254080 -> 0; G1 Old Gen: 3976200696 -> 4437455344; G1 Survivor Space:
>> 1644167168 -
>> INFO 14:36:07 G1 Young Generation GC in 212ms. G1 Eden Space:
>> 1711276032 -> 0; G1 Old Gen: 4437455344 <(443)%20745-5344> ->
>> 5830082552; G1 Survivor Space: 1459617792 ->
>> INFO 15:00:30 G1 Young Generation GC in 233ms. G1 Eden Space:
>> 19159580672 <(915)%20958-0672> -> 0; G1 Old Gen: 4040399864 ->
>> 4343952368; G1 Survivor Space: 1442840576 -
>> INFO 15:09:31 Redistributing index summaries
>> INFO 15:26:28 G1 Young Generation GC in 234ms. G1 Eden Space:
>> 19075694592 <(907)%20569-4592> -> 0; G1 Old Gen: 4604513264 ->
>> 5064485872; G1 Survivor Space: 1526726656 -
>> INFO 15:52:08 G1 Young Generation GC in 222ms. G1 Eden Space:
>> 19126026240 <(912)%20602-6240> -> 0; G1 Old Gen: 4118806520 ->
>> 4391313912; G1 Survivor Space: 1476395008 -
>> INFO 15:55:06 G1 Young Generation GC in 202ms. G1 Eden Space:
>> 19142803456 <(914)%20280-3456> -> 0; G1 Old Gen: 5267292152 ->
>> 5559551472;
>> INFO 16:09:31 Redistributing index summaries
>> INFO 16:20:48 G1 Young Generation GC in 241ms. G1 Eden Space:
>> 19126026240 <(912)%20602-6240> -> 0; G1 Old Gen: 3854587896 ->
>> 4287600112; G1 Survivor Space: 1476395008 -
>> INFO 16:31:07 G1 Young Generation GC in 403ms. G1 Eden Space: 822083584
>> -> 0; G1 Old Gen: 9219080688 -> 9994162168; G1 Survivor Space: 889192448 ->
>> 2
>> INFO 16:33:41 G1 Young Generation GC in 426ms. G1 Eden Space: 620756992
>> -> 0; G1 Old Gen: 10147301880 -> 11065889792; G1 Survivor Space: 1090519040
>> -
>> INFO 16:38:17 G1 Young Generation GC in 835ms. G1 Eden Space: 251658240
>> -> 0; G1 Old Gen: 12062818288 <(206)%20281-8288> -> 13404581312; G1
>> Survivor Space: 1459617792 -
>> INFO 16:43:08 G1 Young Generation GC in 569ms. G1 Eden Space:
>> 3154116608 <(315)%20411-6608> -> 0; G1 Old Gen: 14361297352 ->
>> 15461740984; G1 Survivor Space: 1258291200
>> INFO 16:46:08 G1 Young Generation GC in 432ms. G1 Eden Space:
>> 18706595840 <(870)%20659-5840> -> 0; G1 Old Gen: 3256274416 ->
>> 4355653104; G1 Survivor Space: 1023410176 -
>> INFO 17:09:31 Redistributing index summaries
>> INFO 17:26:10 G1 Young Generation GC in 744ms. G1 Eden Space: 201326592
>> -> 0; G1 Old Gen: 6969713640 -> 8462801376; G1 Survivor Space: 1577058304
>> ->
>> INFO 17:29:04 G1 Young Generation GC in 204ms. G1 Eden Space: 603979776
>> -> 0; G1 Old Gen: 8454412776 <(845)%20441-2776> -> 9439609320; G1
>> Survivor Space: 1107296256 ->
>> INFO 17:41:09 G1 Young Generation GC in 279ms. G1 Eden Space: 352321536
>> -> 0; G1 Old Gen: 13615267816 <(361)%20526-7816> -> 14860613592; G1
>> Survivor Space: 1358954496 -
>> INFO 17:46:51 G1 Young Generation GC in 493ms. G1 Eden Space: 754974720
>> -> 0; G1 Old Gen: 3560839664 -> 4350490576; G1 Survivor Space: 956301312 ->
>> 2
>> INFO 17:49:09 G1 Young Generation GC in 290ms. G1 Eden Space: 335544320
>> -> 0; G1 Old Gen: 4350490576 -> 5603827664; G1 Survivor Space: 1375731712
>> ->
>> INFO 17:55:17 G1 Young Generation GC in 803ms. G1 Eden Space: 201326592
>> -> 0; G1 Old Gen: 6543114704 -> 8036691400; G1 Survivor Space: 1577058304
>> ->
>> INFO 17:58:07 G1 Young Generation GC in 206ms. G1 Eden Space:
>> 4764729344 -> 0; G1 Old Gen: 8036691400 <(803)%20669-1400> ->
>> 8934491600; G1 Survivor Space: 973078528 ->
>> INFO 18:07:29 G1 Young Generation GC in 306ms. G1 Eden Space: 704643072
>> -> 0; G1 Old Gen: 12163481536 <(216)%20348-1536> -> 13045510592; G1
>> Survivor Space: 1006632960 -
>> INFO 18:08:09 Saved KeyCache (5307 items) in 85 ms
>> INFO 18:09:31 Redistributing index summaries
>> INFO 18:10:11 G1 Young Generation GC in 277ms. G1 Eden Space: 402653184
>> -> 0; G1 Old Gen: 13045510592 <(304)%20551-0592> -> 14252493256; G1
>> Survivor Space: 1308622848 -
>> INFO 18:15:39 G1 Young Generation GC in 238ms. G1 Eden Space:
>> 13052674048 <(305)%20267-4048> -> 0; G1 Old Gen: 15141437888 ->
>> 15520873416; G1 Survivor Space: 1358954496
>> INFO 18:20:16 G1 Young Generation GC in 540ms. G1 Eden Space:
>> 7616856064 -> 0; G1 Old Gen: 3909694456 -> 4886048752; G1 Survivor Space:
>> 1509949440 ->
>> INFO 18:23:04 G1 Young Generation GC in 221ms. G1 Eden Space:
>> 16894656512 -> 0; G1 Old Gen: 5420089848 -> 6187322368; G1 Survivor Space:
>> 939524096 ->
>> INFO 18:42:08 G1 Young Generation GC in 708ms. G1 Eden Space: 352321536
>> -> 0; G1 Old Gen: 12840221688 -> 14085112816; G1 Survivor Space: 1358954496
>> -
>> INFO 18:50:13 G1 Young Generation GC in 318ms. G1 Eden Space: 218103808
>> -> 0; G1 Old Gen: 3631963120 -> 5033289720; G1 Survivor Space: 1493172224
>> ->
>> INFO 19:09:31 Redistributing index summaries
>> INFO 19:27:02 G1 Young Generation GC in 384ms. G1 Eden Space:
>> 12901679104 -> 0; G1 Old Gen: 7034014192 <(703)%20401-4192> ->
>> 7759489528; G1 Survivor Space: 889192448 ->
>> INFO 19:31:25 G1 Young Generation GC in 770ms. G1 Eden Space: 218103808
>> -> 0; G1 Old Gen: 8506048488 <(850)%20604-8488> -> 9921951720; G1
>> Survivor Space: 1493172224 ->
>> INFO 20:09:31 Redistributing index summaries
>> INFO 21:09:31 Redistributing index summaries
>> INFO 21:19:39 G1 Young Generation GC in 201ms. G1 Eden Space:
>> 19142803456 <(914)%20280-3456> -> 0; G1 Old Gen: 3917480432 ->
>> 4124623344; G1 Survivor Space: 1459617792 -
>> INFO 22:04:05 G1 Young Generation GC in 209ms. G1 Eden Space:
>> 19142803456 <(914)%20280-3456> -> 0; G1 Old Gen: 4420796912 ->
>> 4618141680; G1 Survivor Space: 1459617792 -
>> INFO 22:07:49 G1 Young Generation GC in 215ms. G1 Eden Space:
>> 14713618432 -> 0; G1 Old Gen: 5970255352 -> 6555802608; G1 Survivor Space:
>> 1493172224 -
>> INFO 22:08:09 Saved KeyCache (5845 items) in 87 ms
>> INFO 22:09:31 Redistributing index summaries
>> INFO 22:38:04 G1 Young Generation GC in 201ms. G1 Eden Space:
>> 19075694592 <(907)%20569-4592> -> 0; G1 Old Gen: 4248291312 ->
>> 4459621368; G1 Survivor Space: 1526726656 -
>> INFO 22:44:12 G1 Young Generation GC in 203ms. G1 Eden Space:
>> 10770972672 -> 0; G1 Old Gen: 6703617016 -> 7549747184; G1 Survivor Space:
>> 1325400064 -
>> INFO 22:47:05 G1 Young Generation GC in 207ms. G1 Eden Space:
>> 19025362944 <(902)%20536-2944> -> 0; G1 Old Gen: 7549747184 ->
>> 7771343352; G1 Survivor Space: 1577058304 -
>> INFO 23:08:59 G1 Young Generation GC in 369ms. G1 Eden Space:
>> 15820914688 -> 0; G1 Old Gen: 4823449592 -> 5478502888; G1 Survivor Space:
>> 1660944384 -
>> INFO 23:09:31 Redistributing index summaries
>> INFO 23:45:21 Maximum memory usage reached (2.000GiB), cannot allocate
>> chunk of 1.000MiB
>> INFO 23:58:40 G1 Young Generation GC in 283ms. G1 Eden Space:
>> 18941476864 -> 0; G1 Old Gen: 4831838192 -> 5297926640; G1 Survivor Space:
>> 1660944384 -
>> INFO 00:01:54 G1 Young Generation GC in 211ms. G1 Eden Space:
>> 18891145216 -> 0; G1 Old Gen: 6425673712 -> 6912948208; G1 Survivor Space:
>> 1711276032 -
>> INFO 00:05:37 Maximum memory usage reached (2.000GiB), cannot allocate
>> chunk of 1.000MiB
>> INFO 00:09:31 Redistributing index summaries
>> INFO 00:14:33 G1 Young Generation GC in 282ms. G1 Eden Space:
>> 14344519680 <(434)%20451-9680> -> 0; G1 Old Gen: 14159246832 ->
>> 14677156848; G1 Survivor Space: 1207959552
>> INFO 00:21:32 G1 Young Generation GC in 225ms. G1 Eden Space:
>> 19042140160 <(904)%20214-0160> -> 0; G1 Old Gen: 4346926576 ->
>> 4865982456; G1 Survivor Space: 1560281088 -
>> INFO 00:22:09 Maximum memory usage reached (2.000GiB), cannot allocate
>> chunk of 1.000MiB
>> INFO 00:24:31 G1 Young Generation GC in 266ms. G1 Eden Space:
>> 19008585728 <(900)%20858-5728> -> 0; G1 Old Gen: 6106907128 ->
>> 6613907952; G1 Survivor Space: 1593835520 -
>> INFO 00:29:46 G1 Young Generation GC in 233ms. G1 Eden Space:
>> 18236833792 -> 0; G1 Old Gen: 9076474360 <(907)%20647-4360> ->
>> 9498544624; G1 Survivor Space: 1795162112 -
>> INFO 00:32:35 G1 Young Generation GC in 226ms. G1 Eden Space:
>> 16492003328 -> 0; G1 Old Gen: 10804527088 -> 11351712752; G1 Survivor
>> Space: 1811939328
>> INFO 00:37:09 Maximum memory usage reached (2.000GiB), cannot allocate
>> chunk of 1.000MiB
>> INFO 00:41:24 G1 Young Generation GC in 330ms. G1 Eden Space:
>> 19142803456 <(914)%20280-3456> -> 0; G1 Old Gen: 3426117112 ->
>> 3961507312; G1 Survivor Space: 1459617792 -
>> INFO 00:44:26 G1 Young Generation GC in 325ms. G1 Eden Space:
>> 19042140160 <(904)%20214-0160> -> 0; G1 Old Gen: 5268046320 ->
>> 5788210120; G1 Survivor Space: 1560281088 -
>> INFO 00:47:13 G1 Young Generation GC in 209ms. G1 Eden Space:
>> 19243466752 -> 0; G1 Old Gen: 7147093960 <(714)%20709-3960> ->
>> 7138705360; G1 Survivor Space: 1358954496 -
>> INFO 00:47:28 G1 Young Generation GC in 312ms. G1 Eden Space:
>> 18605932544 <(860)%20593-2544> -> 0; G1 Old Gen: 7138705360 ->
>> 7710094280; G1 Survivor Space: 1996488704 -
>> INFO 00:50:35 G1 Young Generation GC in 330ms. G1 Eden Space:
>> 17934843904 -> 0; G1 Old Gen: 9118416840 -> 9688820680; G1 Survivor Space:
>> 1929379840 -
>> INFO 00:52:52 Maximum memory usage reached (2.000GiB), cannot allocate
>> chunk of 1.000MiB
>> INFO 00:56:29 G1 Young Generation GC in 251ms. G1 Eden Space:
>> 14797504512 <(479)%20750-4512> -> 0; G1 Old Gen: 13404995520 ->
>> 13787049928; G1 Survivor Space: 1358954496
>> INFO 00:58:14 G1 Young Generation GC in 662ms. G1 Eden Space: 369098752
>> -> 0; G1 Old Gen: 14730395584 -> 15984921520; G1 Survivor Space: 1342177280
>> -
>> INFO 01:01:25 G1 Young Generation GC in 288ms. G1 Eden Space:
>> 19042140160 <(904)%20214-0160> -> 0; G1 Old Gen: 2618580968 ->
>> 3127466464; G1 Survivor Space: 1560281088 -
>> INFO 01:04:17 G1 Young Generation GC in 201ms. G1 Eden Space:
>> 19327352832 -> 0; G1 Old Gen: 4521459688 -> 4529848288; G1 Survivor Space:
>> 1275068416 -
>> INFO 01:04:39 G1 Young Generation GC in 283ms. G1 Eden Space:
>> 18723373056 <(872)%20337-3056> -> 0; G1 Old Gen: 4529848288 ->
>> 5024210408; G1 Survivor Space: 1879048192 -
>> INFO 01:07:23 G1 Young Generation GC in 294ms. G1 Eden Space:
>> 18740150272 -> 0; G1 Old Gen: 6308233704 <(630)%20823-3704> ->
>> 6875745256; G1 Survivor Space: 1862270976 -
>> INFO 01:07:53 Maximum memory usage reached (2.000GiB), cannot allocate
>> chunk of 1.000MiB
>> INFO 01:09:31 Redistributing index summaries
>> INFO 01:10:12 G1 Young Generation GC in 213ms. G1 Eden Space:
>> 19243466752 -> 0; G1 Old Gen: 8262779360 -> 8254390760; G1 Survivor Space:
>> 1358954496 -
>> INFO 01:10:33 G1 Young Generation GC in 319ms. G1 Eden Space:
>> 18639486976 <(863)%20948-6976> -> 0; G1 Old Gen: 8254390760 ->
>> 8821123552; G1 Survivor Space: 1962934272 -
>> INFO 01:22:55 Maximum memory usage reached (2.000GiB), cannot allocate
>> chunk of 1.000MiB
>> INFO 01:24:25 G1 Young Generation GC in 227ms. G1 Eden Space:
>> 19226689536 -> 0; G1 Old Gen: 4245307376 <(424)%20530-7376> ->
>> 4236918776; G1 Survivor Space: 1375731712 -
>> INFO 01:24:48 G1 Young Generation GC in 219ms. G1 Eden Space:
>> 15401484288 -> 0; G1 Old Gen: 4236918776 <(423)%20691-8776> ->
>> 5231503864; G1 Survivor Space: 2097152000 -
>> INFO 01:42:01 G1 Young Generation GC in 212ms. G1 Eden Space:
>> 16005464064 <(600)%20546-4064> -> 0; G1 Old Gen: 4320133112 ->
>> 5100782576; G1 Survivor Space: 1627389952 -
>> INFO 01:48:28 G1 Young Generation GC in 209ms. G1 Eden Space:
>> 14730395648 -> 0; G1 Old Gen: 8901994480 -> 9402393592; G1 Survivor Space:
>> 1291845632 -
>> INFO 02:04:18 G1 Young Generation GC in 241ms. G1 Eden Space:
>> 19042140160 <(904)%20214-0160> -> 0; G1 Old Gen: 7825270264 ->
>> 8327173624; G1 Survivor Space: 1560281088 -
>> INFO 02:08:09 Saved KeyCache (8634 items) in 94 ms
>> INFO 02:09:31 Redistributing index summaries
>> INFO 02:14:50 G1 Young Generation GC in 314ms. G1 Eden Space:
>> 19075694592 <(907)%20569-4592> -> 0; G1 Old Gen: 3505807352 ->
>> 4140375528; G1 Survivor Space: 1526726656 -
>> INFO 02:48:35 G1 Young Generation GC in 321ms. G1 Eden Space:
>> 11576279040 -> 0; G1 Old Gen: 3865890800 <(386)%20589-0800> ->
>> 4482588664; G1 Survivor Space: 1543503872 -
>> INFO 02:53:54 G1 Young Generation GC in 203ms. G1 Eden Space:
>> 17901289472 -> 0; G1 Old Gen: 7927235056 -> 8552357368; G1 Survivor Space:
>> 1593835520 -
>> INFO 03:04:33 G1 Young Generation GC in 345ms. G1 Eden Space:
>> 13505658880 -> 0; G1 Old Gen: 3765058032 -> 4443454952; G1 Survivor Space:
>> 1442840576 -
>> INFO 03:09:31 Redistributing index summaries
>> INFO 03:22:56 G1 Young Generation GC in 250ms. G1 Eden Space:
>> 18991808512 -> 0; G1 Old Gen: 4806672880 <(480)%20667-2880> ->
>> 5340134904; G1 Survivor Space: 1610612736 -
>> INFO 03:35:13 G1 Young Generation GC in 210ms. G1 Eden Space:
>> 14445182976 -> 0; G1 Survivor Space: 1073741824 -> 1744830464;
>> INFO 03:38:42 G1 Young Generation GC in 214ms. G1 Eden Space:
>> 18303942656 <(830)%20394-2656> -> 0; G1 Old Gen: 4786540536 ->
>> 5432116720; G1 Survivor Space: 1375731712 -
>> INFO 03:52:11 G1 Young Generation GC in 207ms. G1 Eden Space: 117440512
>> -> 0; G1 Old Gen: 12054831088 <(205)%20483-1088> -> 10154180600; G1
>> Survivor Space: 1593835520 -
>> INFO 03:54:48 G1 Young Generation GC in 224ms. G1 Eden Space:
>> 16156459008 <(615)%20645-9008> -> 0; G1 Old Gen: 4303355896 ->
>> 4801558520; G1 Survivor Space: 1560281088 -
>> INFO 04:09:31 Redistributing index summaries
>> INFO 04:16:09 G1 Young Generation GC in 216ms. G1 Eden Space:
>> 19142803456 <(914)%20280-3456> -> 0; G1 Old Gen: 5853311984 ->
>> 6366188536; G1 Survivor Space: 1459617792 -
>> INFO 04:19:13 G1 Young Generation GC in 229ms. G1 Eden Space:
>> 19042140160 <(904)%20214-0160> -> 0; G1 Old Gen: 7826571256 ->
>> 8457920504; G1 Survivor Space: 1560281088 -
>> INFO 04:27:41 G1 Young Generation GC in 311ms. G1 Eden Space:
>> 13723762688 -> 0; G1 Old Gen: 14176747504 <(417)%20674-7504> ->
>> 14808230392; G1 Survivor Space: 1509949440
>> INFO 04:29:56 G1 Young Generation GC in 274ms. G1 Eden Space:
>> 18740150272 -> 0; G1 Old Gen: 3376452080 <(337)%20645-2080> ->
>> 3818638328; G1 Survivor Space: 1459617792 -
>> INFO 04:33:40 Maximum memory usage reached (2.000GiB), cannot allocate
>> chunk of 1.000MiB
>> INFO 04:49:07 G1 Young Generation GC in 319ms. G1 Eden Space:
>> 16005464064 <(600)%20546-4064> -> 0; G1 Old Gen: 4435714040 ->
>> 5023449072; G1 Survivor Space: 1442840576 -
>> INFO 04:54:06 G1 Young Generation GC in 208ms. G1 Eden Space:
>> 18706595840 <(870)%20659-5840> -> 0; G1 Old Gen: 7843348472 ->
>> 8466889208; G1 Survivor Space: 1476395008 -
>> INFO 05:09:31 Redistributing index summaries
>>
>>
>> I found INFO Maximum memory usage reached (2.000GiB), cannot allocate
>> chunk of 1.000MiB, does it mean the memory is not enough ?
>>
>>
>> I want to know why adding a new node spent so much time and what
>> need I do to complete a new node joining the cluster ?
>>
>>
>> I need some help and really need some suggestions.
>>
>>
>>
>> Thanks a lot!
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> The information in this Internet Email is confidential and may be legally
>> privileged. It is intended solely for the addressee. Access to this Email
>> by anyone else is unauthorized. If you are not the intended recipient, any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it, is prohibited and may be unlawful. When addressed
>> to our clients any opinions or advice contained in this Email are subject
>> to the terms and conditions expressed in any applicable governing The Home
>> Depot terms of business or client engagement letter. The Home Depot
>> disclaims all responsibility and liability for the accuracy and content of
>> this attachment and for any damages or losses arising from any
>> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
>> items of a destructive nature, which may be contained in this attachment
>> and shall not be liable for direct, indirect, consequential or special
>> damages in connection with this e-mail message or its attachment.
>>
>>
>>
>
Re: [EXTERNAL] Cassandra cluster add new node slowly
Posted by Anthony Grasso <an...@gmail.com>.
The speed at which compactions operate is also physically restricted by the
speed of the disk. If the disks used on the new node are HDDs, then
increasing the compaction throughput will be of little help. However, if
the disks on the new node are SSDs then increasing the compaction
throughput to at least 64MB/s should help speed up compactions.
Regards,
Anthony
On 4 January 2018 at 14:13, qf zhou <zh...@gmail.com> wrote:
> The cassandra version is 3.0.9.
>
> I have changed the heap size (about 32G). Also, the streaming
> throughput is set 800MB/sec, and the streaming_socket_timeout_in_ms is
> default 86400000.
> I suspect the compactionthroughput has an influence on the new node
> joining. The command nodetool | getcompactionthroughput says 'Current
> compaction throughput: 32 MB/s’.
>
>
>
>
> 在 2018年1月4日，上午4:59，Durity, Sean R <SE...@homedepot.com> 写道：
>
> You don’t mention the version, but here are some general suggestions
>
> - 2 GB heap is very small for a node, especially with 1 TB+ of
> data. What is the physical RAM on the host? In general, you want ½ of
> physical RAM for the JVM. (Look in jvm.options or cassandra-env.sh)
> - You can change the streaming throughput from the existing
> nodes, if it looks like the new node can handle it. Look at nodetool
> setstreamthroughput. Default is 200 (MB/sec).
> - You might want to check for a streaming_socket_timeout_in_ms.
> This has changed over the versions. Some details are at:
> https://issues.apache.org/jira/browse/CASSANDRA-11839. 24 hours is good
> recommendation.
> - If your new node can’t compact fast enough to keep disk usage
> down, look at compactionthroughput on that node
> - nodetool netstats | grep –v “100%” is a good way to see what
> is happening/if anything is stuck. Newer versions give a bit more info on
> progress.
> - Don’t forget to run cleanup on existing nodes after the new
> nodes are added.
>
>
>
> Sean Durity
> *From:* qf zhou [mailto:zhouqf2013@gmail.com <zh...@gmail.com>]
> *Sent:* Tuesday, January 02, 2018 10:30 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Cassandra cluster add new node slowly
>
> The cluster has 3 nodes, and the data in each node is about 1.2 T. I
> want to add two new nodes to expand the cluster.
>
> Following the instructions from the datastax website, ie, (
> http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/operations/
> opsAddNodeToCluster.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_archived_cassandra_3.x_cassandra_operations_opsAddNodeToCluster.html&d=DwMFoQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=AfP0BxfTG1o_-2nRtGSjngs45YndH060h39eVlypVso&s=AaiOypurMtWA470038fHMbaXJPIZeYtLzv3Yt9vf1Sc&e=>),
>
>
> I try to add one node to the cluster. However, it is too slow and
> time cost too much. After about 24 hours, it still didn’t success.
>
> I run the command: nodetool netstats on the new node, it shows that:
>
> (tb1fullwithstate2 is a big table and 90% of the cluster data is in
> it. Here I use CompactionStrategy: TimeWindowCompactionStrategy).
>
> /*.*.*.3
> Receiving 136 files, 328573794609 bytes total. Already received 8
> files, 5774621188 bytes total
> tb1/tb1fullwithneweststatetest 3758271/3758271 bytes(100%)
> received from idx:0/*.*.*.3
> system_distributed/repair_history 57534/57534 bytes(100%)
> received from idx:0/*.*.*.3
> system_distributed/parent_repair_history 507660/507660
> bytes(100%) received from idx:0/*.*.*.3
> tb1/tb1_device_last_state_eachday 15754096/15754096
> bytes(100%) received from idx:0/*.*.*.3
> mytest1/tb1_test1 8143775/8143775 bytes(100%) received from
> idx:0/*.*.*.3
> tb1/tb1fullwithstate 2251191007/2251191007 bytes(100%)
> received from idx:0/*.*.*.3
> applocationinfo/weiyirong_app 2760/2760 bytes(100%) received
> from idx:0/*.*.*.3
> tb1/tb1fullwithstate2 3490748006/4909554503 bytes(71%)
> received from idx:0/*.*.*.3
> tb1/tb1fullwithneweststate 4458079/4458079 bytes(100%)
> received from idx:0/*.*.*.3
> /*.*.*.2
> Receiving 136 files, 336762487360 bytes total. Already received 3
> files, 5695770181 bytes total
> system_distributed/repair_history 31684/31684 bytes(100%)
> received from idx:0/*.*.*.2
> tb1/tb1fullwithstate 908260516/908260516 bytes(100%) received
> from idx:0/*.*.*.2
> tb1/tb1fullwithstate2 4783622958/4990450588 bytes(95%)
> received from idx:0/*.*.*.2
> tb1/tb1fullwithneweststate 3855023/3855023 bytes(100%)
> received from idx:0/*.*.*.2
> /*.*.*.4
> Receiving 132 files, 236250553620 bytes total. Already received 10
> files, 3117465128 bytes total
> mytest1/wordstest2 46/46 bytes(100%) received from
> idx:0/*.*.*.4
> tb1/tb1fullwithneweststatetest 3416891/3416891 bytes(100%)
> received from idx:0/*.*.*.4
> system_distributed/repair_history 39720/39720 bytes(100%)
> received from idx:0/*.*.*.4
> system_distributed/parent_repair_history 452250/452250
> bytes(100%) received from idx:0/*.*.*.4
> mytest1/weblogs 104/104 bytes(100%) received from idx:0/*.*.*.4
> tb1/tb1_device_last_state_eachday 12670998/12670998
> bytes(100%) received from idx:0/*.*.*.4
> mytest1/tb1_test1 3257952/3257952 bytes(100%) received from
> idx:0/*.*.*.4
> tb1/tb1fullwithstate 647702056/647702056 bytes(100%) received
> from idx:0/*.*.*.4
> applocationinfo/weiyirong_app 3509/3509 bytes(100%) received
> from idx:0/*.*.*.4
> tb1/tb1fullwithstate2 2446436305/3566562762 bytes(68%)
> received from idx:0/*.*.*.4
> tb1/tb1fullwithneweststate 3485297/3485297 bytes(100%)
> received from idx:0/*.*.*.4
>
>
>
> check the log in the logs/system.log, it shows that:
>
>
> INFO 06:09:33 Updating topology for /*.*.*.2
> INFO 06:09:33 Updating topology for /*.*.*.2
> INFO 06:09:33 Node /*.*.*.3 has restarted, now UP
> INFO 06:09:33 InetAddress /*.*.*.3 is now UP
> INFO 06:09:33 Node /*.*.*.3 state jump to NORMAL
> INFO 06:09:33 Updating topology for /*.*.*.3
> INFO 06:09:33 Updating topology for /*.*.*.3
> INFO 06:09:33 Handshaking version with /*.*.*.2
> INFO 06:09:33 Handshaking version with /*.*.*.3
> INFO 06:09:36 Handshaking version with /*.*.*.4
> INFO 06:10:03 JOINING: Starting to bootstrap...
> INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Executing
> streaming plan for Bootstrap
> INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting
> streaming to /*.*.*.4
> INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0]
> Beginning stream session with /*.*.*.4
> INFO 06:10:04 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527790874-1.hints
> INFO 06:10:04 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527790874-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO 06:10:11 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527800873-1.hints
> INFO 06:10:11 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527800873-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO 06:10:12 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527790881-1.hints
> INFO 06:10:12 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527790881-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO 06:10:20 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527800880-1.hints
> INFO 06:10:20 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527800880-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO 06:10:21 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527790884-1.hints
> INFO 06:10:21 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527790884-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO 06:10:28 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527800887-1.hints
> INFO 06:10:28 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527800887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0]
> Prepare completed. Receiving 132 files(220.025GiB), sending 0
> files(0.000KiB)
> INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting
> streaming to /*.*.*.2
> INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0]
> Beginning stream session with /*.*.*.2
> INFO 06:10:34 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527810873-1.hints
> INFO 06:10:34 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527810873-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO 06:10:42 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527820873-1.hints
> INFO 06:10:42 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527820873-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO 06:10:58 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527810881-1.hints
> INFO 06:10:58 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527810881-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0]
> Prepare completed. Receiving 136 files(313.635GiB), sending 0
> files(0.000KiB)
> INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting
> streaming to /*.*.*.3
> INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0]
> Beginning stream session with /*.*.*.3
> INFO 06:11:06 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527830880-1.hints
> INFO 06:11:06 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527830880-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO 06:11:29 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527810889-1.hints
> INFO 06:11:29 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527810889-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO 06:11:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0]
> Prepare completed. Receiving 136 files(306.008GiB), sending 0
> files(0.000KiB)
> INFO 06:11:36 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527840887-1.hints
> INFO 06:11:36 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527840887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO 06:11:49 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527830873-1.hints
> INFO 06:11:49 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527830873-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO 06:11:56 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527850874-1.hints
> INFO 06:11:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527850874-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO 06:12:26 Deleted hint file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527840880-1.hints
> INFO 06:12:26 Finished hinted handoff of file c555e44c-9590-4f45-aea4-
> f5eca68180b2-1514527840880-1.hints to endpoint c555e44c-9590-4f45-aea4-
> f5eca6818
> INFO 06:12:45 Deleted hint file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527850883-1.hints
> INFO 06:12:45 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-
> db7115719d59-1514527850883-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-
> db7115719
> INFO 06:12:56 Deleted hint file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527860874-1.hints
> INFO 06:12:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-
> dc169cf87e95-1514527860874-1.hints to endpoint c5271e74-19a1-4cee-98d7-
> dc169cf87
> INFO 06:24:03 G1 Young Generation GC in 242ms. G1 Eden Space:
> 18924699648 -> 0; G1 Old Gen: 4360876536 -> 5095617016; G1 Survivor Space:
> 1677721600 -
> INFO 06:26:06 G1 Young Generation GC in 215ms. G1 Eden Space:
> 18220056576 -> 0; G1 Old Gen: 5981077496 -> 6597610488; G1 Survivor Space:
> 1493172224 -
> INFO 06:29:04 G1 Young Generation GC in 225ms. G1 Eden Space:
> 18538823680 -> 0; G1 Old Gen: 7910457840 -> 8860639728; G1 Survivor Space:
> 1895825408 -
> INFO 06:33:22 G1 Young Generation GC in 214ms. G1 Eden Space:
> 16559112192 -> 0; G1 Old Gen: 11868321784 -> 11941600752; G1 Survivor
> Space: 1241513984
> INFO 06:33:33 G1 Young Generation GC in 205ms. G1 Eden Space: 8069840896
> -> 0; G1 Old Gen: 11941600752 -> 13683606520; G1 Survivor Space: 1895825408
> INFO 06:41:01 G1 Young Generation GC in 383ms. G1 Eden Space:
> 18874368000 -> 0; G1 Old Gen: 4770399736 -> 5551887344; G1 Survivor Space:
> 1728053248 -
> INFO 06:45:07 G1 Young Generation GC in 226ms. G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 8036286448 -> 8739698672; G1 Survivor Space:
> 1577058304 -
> INFO 06:47:11 G1 Young Generation GC in 203ms. G1 Eden Space:
> 18538823680 -> 0; G1 Survivor Space: 1191182336 -> 1778384896;
> INFO 06:47:22 G1 Young Generation GC in 241ms. G1 Eden Space: 8086618112
> -> 0; G1 Old Gen: 9999220720 -> 11591876088; G1 Survivor Space: 1778384896 -
> INFO 06:49:43 G1 Young Generation GC in 201ms. G1 Eden Space:
> 16022241280 -> 0; G1 Old Gen: 11591876088 -> 12084766712; G1 Survivor
> Space: 1644167168
> INFO 06:53:21 G1 Young Generation GC in 322ms. G1 Eden Space:
> 13237223424 -> 0; G1 Old Gen: 14755561464 -> 15445886448; G1 Survivor
> Space: 1459617792
> INFO 06:55:34 G1 Young Generation GC in 343ms. G1 Eden Space:
> 16525557760 -> 0; G1 Old Gen: 2939907056 -> 3636770792; G1 Survivor Space:
> 1610612736 -
> INFO 07:09:30 Redistributing index summaries
> INFO 07:14:25 G1 Young Generation GC in 270ms. G1 Eden Space:
> 15602810880 -> 0; G1 Old Gen: 4360518136 -> 5243274224; G1 Survivor Space:
> 1610612736 -
> INFO 07:16:51 G1 Young Generation GC in 229ms. G1 Eden Space:
> 19008585728 -> 0; G1 Old Gen: 6132072440 -> 6815145976; G1 Survivor Space:
> 1593835520 -
> INFO 07:19:34 G1 Young Generation GC in 215ms. G1 Eden Space:
> 18991808512 -> 0; G1 Old Gen: 8689337336 -> 9071161328; G1 Survivor Space:
> 1610612736 -
> INFO 07:23:35 G1 Young Generation GC in 293ms. G1 Eden Space:
> 15753805824 -> 0; G1 Old Gen: 11777605616 -> 12289649640; G1 Survivor
> Space: 1694498816
> INFO 07:29:42 G1 Young Generation GC in 362ms. G1 Eden Space:
> 15955132416 -> 0; G1 Old Gen: 3742637560 -> 4410361824; G1 Survivor Space:
> 1644167168 -
> INFO 07:35:12 G1 Young Generation GC in 208ms. G1 Eden Space:
> 19478347776 -> 0; G1 Old Gen: 7199737832 -> 7208126432; G1 Survivor Space:
> 1124073472 -
> INFO 07:35:33 G1 Young Generation GC in 219ms. G1 Eden Space:
> 18740150272 -> 0; G1 Old Gen: 7208126432 -> 8072478184; G1 Survivor Space:
> 1862270976 -
> INFO 07:45:19 G1 Young Generation GC in 213ms. G1 Eden Space:
> 13153337344 -> 0; G1 Old Gen: 15418261472 -> 16123805152; G1 Survivor
> Space: 1174405120
> INFO 07:48:08 G1 Young Generation GC in 335ms. G1 Eden Space:
> 18387828736 -> 0; G1 Old Gen: 3852453368 -> 4464797168; G1 Survivor Space:
> 1644167168 -
> INFO 07:48:08 G1 Young Generation GC in 209ms. G1 Eden Space: 352321536
> -> 0; G1 Old Gen: 4464797168 -> 5989466096; G1 Survivor Space: 1476395008
> ->
> INFO 07:50:25 G1 Young Generation GC in 221ms. G1 Eden Space:
> 17498636288 -> 0; G1 Old Gen: 5981077496 -> 6488668656; G1 Survivor Space:
> 1593835520 -
> INFO 08:03:31 G1 Young Generation GC in 748ms. G1 Eden Space: 8673820672
> -> 0; G1 Old Gen: 4386367472 -> 5874747352; G1 Survivor Space: 1627389952 ->
> INFO 08:09:30 Redistributing index summaries
> INFO 08:10:05 G1 Young Generation GC in 213ms. G1 Eden Space:
> 18220056576 -> 0; G1 Old Gen: 9276024272 -> 9969348048; G1 Survivor Space:
> 1711276032 -
> INFO 08:18:59 G1 Young Generation GC in 228ms. G1 Eden Space:
> 17347641344 -> 0; G1 Old Gen: 4546626040 -> 5180788216; G1 Survivor Space:
> 1593835520 -
> INFO 08:33:12 G1 Young Generation GC in 207ms. G1 Eden Space:
> 19411238912 -> 0; G1 Survivor Space: 1191182336 -> 1828716544;
> INFO 08:33:13 G1 Young Generation GC in 273ms. G1 Eden Space: 452984832
> -> 0; G1 Old Gen: 4668340728 -> 6320276984; G1 Survivor Space: 1828716544
> ->
> INFO 08:41:59 G1 Young Generation GC in 202ms. G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 8701765104 -> 8927144944; G1 Survivor Space:
> 1577058304 -
> INFO 08:56:31 G1 Young Generation GC in 205ms. G1 Eden Space:
> 11072962560 -> 0; G1 Old Gen: 9563013104 -> 11068430328; G1 Survivor Space:
> 1660944384
> INFO 09:05:49 G1 Young Generation GC in 367ms. G1 Eden Space:
> 18522046464 -> 0; G1 Old Gen: 4009754608 -> 4732347872; G1 Survivor Space:
> 1778384896 -
> INFO 09:09:30 Redistributing index summaries
> INFO 09:14:52 G1 Young Generation GC in 213ms. G1 Eden Space:
> 15267266560 -> 0; G1 Old Gen: 12155092968 -> 12906235360; G1 Survivor
> Space: 1744830464
> INFO 09:17:23 G1 Young Generation GC in 203ms. G1 Eden Space:
> 14159970304 -> 0; G1 Old Gen: 14779568608 -> 14771180008; G1 Survivor
> Space: 989855744
> INFO 09:17:45 G1 Young Generation GC in 204ms. G1 Eden Space:
> 12616466432 -> 0; G1 Old Gen: 14771180008 -> 15500684768; G1 Survivor
> Space: 1761607680
> INFO 09:20:52 G1 Young Generation GC in 361ms. G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 4351990768 -> 5086824424; G1 Survivor Space:
> 1577058304 -
> INFO 09:23:51 G1 Young Generation GC in 207ms. G1 Eden Space:
> 18891145216 -> 0; G1 Old Gen: 7001879024 -> 7716560880; G1 Survivor Space:
> 1711276032 -
> INFO 09:29:20 G1 Young Generation GC in 202ms. G1 Eden Space:
> 17028874240 -> 0; G1 Survivor Space: 1040187392 -> 1828716544;
> INFO 09:29:40 G1 Young Generation GC in 219ms. G1 Eden Space:
> 15451815936 -> 0; G1 Old Gen: 11802771944 -> 12574988776; G1 Survivor
> Space: 1828716544
> INFO 09:44:51 G1 Young Generation GC in 202ms. G1 Eden Space:
> 18001952768 -> 0; G1 Old Gen: 9325481456 -> 10085596664; G1 Survivor Space:
> 1795162112
> INFO 09:45:23 G1 Young Generation GC in 209ms. G1 Eden Space: 9177137152
> -> 0; G1 Old Gen: 10093985264 -> 12309118960; G1 Survivor Space: 1677721600
> INFO 09:50:33 G1 Young Generation GC in 201ms. G1 Eden Space:
> 12901679104 -> 0; G1 Old Gen: 14487498224 -> 15268560880; G1 Survivor
> Space: 1761607680
> INFO 10:08:09 Saved KeyCache (2989 items) in 92 ms
> INFO 10:09:30 Redistributing index summaries
> INFO 10:22:17 G1 Young Generation GC in 216ms. G1 Eden Space:
> 19495124992 -> 0; G1 Old Gen: 7742571504 -> 7734182904; G1 Survivor Space:
> 1107296256 -
> INFO 10:25:31 G1 Young Generation GC in 206ms. G1 Eden Space:
> 18622709760 -> 0; G1 Survivor Space: 1040187392 -> 1711276032;
> INFO 10:34:54 G1 Young Generation GC in 251ms. G1 Eden Space:
> 18991808512 -> 0; G1 Old Gen: 5066719736 -> 5691775992; G1 Survivor Space:
> 1610612736 -
> INFO 10:40:08 G1 Young Generation GC in 209ms. G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 9000976880 -> 9090653168; G1 Survivor Space:
> 1442840576 -
> INFO 10:40:08 G1 Young Generation GC in 223ms. G1 Eden Space: 469762048
> -> 0; G1 Old Gen: 9090653168 -> 11140124152; G1 Survivor Space: 1862270976
> ->
> INFO 10:52:15 G1 Young Generation GC in 219ms. G1 Eden Space:
> 15820914688 -> 0; G1 Old Gen: 5331537912 -> 5828939768; G1 Survivor Space:
> 1275068416 -
> INFO 10:55:30 G1 Young Generation GC in 207ms. G1 Eden Space:
> 15804137472 -> 0; G1 Old Gen: 7501364216 -> 8001282032; G1 Survivor Space:
> 1459617792 -
> INFO 11:07:57 G1 Young Generation GC in 295ms. G1 Eden Space:
> 19159580672 -> 0; G1 Old Gen: 4301379064 -> 4919709168; G1 Survivor Space:
> 1442840576 -
> INFO 11:09:30 Redistributing index summaries
> INFO 11:13:06 G1 Young Generation GC in 204ms. G1 Eden Space:
> 19126026240 -> 0; G1 Old Gen: 8044675064 -> 8659737080; G1 Survivor Space:
> 1476395008 -
> INFO 11:27:07 G1 Young Generation GC in 202ms. G1 Eden Space:
> 19159580672 -> 0; G1 Old Gen: 4328521712 -> 4905718776; G1 Survivor Space:
> 1442840576 -
> INFO 11:30:08 G1 Young Generation GC in 209ms. G1 Eden Space:
> 19243466752 -> 0; G1 Survivor Space: 1358954496 -> 1828716544;
> INFO 11:30:25 G1 Young Generation GC in 226ms. G1 Eden Space:
> 14277410816 -> 0; G1 Old Gen: 6349296624 -> 7552639472; G1 Survivor Space:
> 1828716544 -
> INFO 11:32:30 G1 Young Generation GC in 215ms. G1 Eden Space:
> 19142803456 -> 0; G1 Survivor Space: 1459617792 -> 2046820352;
> INFO 11:33:01 G1 Young Generation GC in 247ms. G1 Eden Space:
> 17263755264 -> 0; G1 Old Gen: 7544250872 -> 8698295280; G1 Survivor Space:
> 2046820352 -
> INFO 11:46:29 G1 Young Generation GC in 355ms. G1 Eden Space:
> 17230200832 -> 0; G1 Old Gen: 4769358328 -> 5362416624; G1 Survivor Space:
> 1761607680 -
> INFO 11:49:18 G1 Young Generation GC in 233ms. G1 Eden Space:
> 17867735040 -> 0; G1 Old Gen: 6752829432 -> 6944805360; G1 Survivor Space:
> 1509949440 -
> INFO 11:52:24 G1 Young Generation GC in 209ms. G1 Eden Space:
> 19193135104 -> 0; G1 Survivor Space: 1409286144 -> 1946157056;
> INFO 11:52:47 G1 Young Generation GC in 338ms. G1 Eden Space:
> 18287165440 -> 0; G1 Old Gen: 8740881904 -> 9340426216; G1 Survivor Space:
> 1946157056 -
> INFO 12:04:08 G1 Young Generation GC in 810ms. G1 Eden Space:
> 18119393280 -> 0; G1 Old Gen: 3471267832 -> 5003518440; G1 Survivor Space:
> 1325400064 -
> INFO 12:06:28 G1 Young Generation GC in 345ms. G1 Eden Space: 8673820672
> -> 0; G1 Old Gen: 5318377952 -> 6577762784; G1 Survivor Space: 1409286144 ->
> INFO 12:09:31 G1 Young Generation GC in 204ms. G1 Eden Space:
> 19159580672 -> 0; G1 Old Gen: 6577762784 -> 6586151384; G1 Survivor Space:
> 1442840576 -
> INFO 12:09:31 Redistributing index summaries
> INFO 12:10:01 G1 Young Generation GC in 298ms. G1 Eden Space:
> 17230200832 -> 0; G1 Old Gen: 6586151384 -> 7710758368; G1 Survivor Space:
> 1946157056 -
> INFO 12:12:37 G1 Young Generation GC in 214ms. G1 Eden Space:
> 18773704704 -> 0; G1 Old Gen: 8380219864 -> 8919123416; G1 Survivor Space:
> 1728053248 -
> INFO 12:24:54 G1 Young Generation GC in 299ms. G1 Eden Space:
> 16055795712 -> 0; G1 Old Gen: 4375908856 -> 5009812976; G1 Survivor Space:
> 1392508928 -
> INFO 12:28:34 G1 Young Generation GC in 485ms. G1 Eden Space:
> 13304332288 -> 0; G1 Old Gen: 6481607672 -> 7506585072; G1 Survivor Space:
> 1979711488 -
> INFO 12:44:52 G1 Young Generation GC in 261ms. G1 Eden Space:
> 18773704704 -> 0; G1 Old Gen: 4239590896 -> 4692442608; G1 Survivor Space:
> 1828716544 -
> INFO 12:48:37 G1 Young Generation GC in 251ms. G1 Eden Space:
> 13656653824 -> 0; G1 Old Gen: 6719275504 -> 7776254960; G1 Survivor Space:
> 1862270976 -
> INFO 12:53:47 G1 Young Generation GC in 217ms. G1 Eden Space:
> 17548967936 -> 0; G1 Old Gen: 9667142640 -> 10202159608; G1 Survivor Space:
> 1845493760
> INFO 13:05:40 G1 Young Generation GC in 287ms. G1 Eden Space:
> 18320719872 -> 0; G1 Old Gen: 5298861040 -> 5826686456; G1 Survivor Space:
> 1862270976 -
> INFO 13:09:31 Redistributing index summaries
> INFO 13:11:44 G1 Young Generation GC in 268ms. G1 Eden Space:
> 10871635968 -> 0; G1 Old Gen: 8799649784 -> 9797878264; G1 Survivor Space:
> 1811939328 -
> INFO 13:14:22 G1 Young Generation GC in 202ms. G1 Eden Space:
> 17750294528 -> 0; G1 Survivor Space: 1325400064 -> 1795162112;
> INFO 13:14:40 G1 Young Generation GC in 254ms. G1 Eden Space:
> 14092861440 -> 0; G1 Old Gen: 10509551608 -> 11563311096; G1 Survivor
> Space: 1795162112
> INFO 13:16:33 G1 Young Generation GC in 213ms. G1 Eden Space:
> 16039018496 -> 0; G1 Old Gen: 11619767792 -> 12115354104; G1 Survivor
> Space: 1627389952
> INFO 13:25:40 G1 Young Generation GC in 245ms. G1 Eden Space:
> 18790481920 -> 0; G1 Old Gen: 4253024752 -> 4761733616; G1 Survivor Space:
> 1811939328 -
> INFO 13:27:38 G1 Young Generation GC in 208ms. G1 Eden Space:
> 13958643712 -> 0; G1 Old Gen: 5377097720 -> 6427987960; G1 Survivor Space:
> 1912602624 -
> INFO 13:29:30 G1 Young Generation GC in 207ms. G1 Eden Space:
> 12264144896 -> 0; G1 Old Gen: 6427987960 -> 7337681912; G1 Survivor Space:
> 1795162112 -
> INFO 13:33:32 G1 Young Generation GC in 202ms. G1 Eden Space:
> 18589155328 -> 0; G1 Old Gen: 8506316280 -> 8514704880; G1 Survivor Space:
> 1308622848 -
> INFO 13:35:09 G1 Young Generation GC in 203ms. G1 Eden Space:
> 17532190720 -> 0; G1 Survivor Space: 1258291200 -> 1761607680;
> INFO 13:38:33 G1 Young Generation GC in 216ms. G1 Eden Space:
> 11106516992 -> 0; G1 Old Gen: 11571635704 -> 12698973168; G1 Survivor
> Space: 1795162112
> INFO 13:47:36 G1 Young Generation GC in 229ms. G1 Eden Space:
> 19075694592 -> 0; G1 Old Gen: 3825205752 -> 4303660528; G1 Survivor Space:
> 1526726656 -
> INFO 13:53:43 G1 Young Generation GC in 207ms. G1 Eden Space:
> 14042529792 -> 0; G1 Old Gen: 7324985848 -> 8363582960; G1 Survivor Space:
> 1660944384 -
> INFO 13:55:40 G1 Young Generation GC in 201ms. G1 Eden Space:
> 18874368000 -> 0; G1 Old Gen: 8355194360 -> 8820888560; G1 Survivor Space:
> 1728053248 -
> INFO 13:56:22 G1 Young Generation GC in 206ms. G1 Eden Space:
> 19394461696 -> 0; G1 Survivor Space: 1207959552 -> 1711276032;
> INFO 14:08:09 Saved KeyCache (4563 items) in 189 ms
> INFO 14:09:31 Redistributing index summaries
> INFO 14:12:47 G1 Young Generation GC in 271ms. G1 Eden Space:
> 18874368000 -> 0; G1 Old Gen: 4345299448 -> 4814148600; G1 Survivor Space:
> 1728053248 -
> INFO 14:18:05 G1 Young Generation GC in 224ms. G1 Eden Space:
> 18907922432 -> 0; G1 Old Gen: 7021265392 -> 7483297784; G1 Survivor Space:
> 1627389952 -
> INFO 14:36:05 G1 Young Generation GC in 304ms. G1 Eden Space:
> 18958254080 -> 0; G1 Old Gen: 3976200696 -> 4437455344; G1 Survivor Space:
> 1644167168 -
> INFO 14:36:07 G1 Young Generation GC in 212ms. G1 Eden Space: 1711276032
> -> 0; G1 Old Gen: 4437455344 -> 5830082552; G1 Survivor Space: 1459617792 ->
> INFO 15:00:30 G1 Young Generation GC in 233ms. G1 Eden Space:
> 19159580672 -> 0; G1 Old Gen: 4040399864 -> 4343952368; G1 Survivor Space:
> 1442840576 -
> INFO 15:09:31 Redistributing index summaries
> INFO 15:26:28 G1 Young Generation GC in 234ms. G1 Eden Space:
> 19075694592 -> 0; G1 Old Gen: 4604513264 -> 5064485872; G1 Survivor Space:
> 1526726656 -
> INFO 15:52:08 G1 Young Generation GC in 222ms. G1 Eden Space:
> 19126026240 -> 0; G1 Old Gen: 4118806520 -> 4391313912; G1 Survivor Space:
> 1476395008 -
> INFO 15:55:06 G1 Young Generation GC in 202ms. G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 5267292152 -> 5559551472;
> INFO 16:09:31 Redistributing index summaries
> INFO 16:20:48 G1 Young Generation GC in 241ms. G1 Eden Space:
> 19126026240 -> 0; G1 Old Gen: 3854587896 -> 4287600112; G1 Survivor Space:
> 1476395008 -
> INFO 16:31:07 G1 Young Generation GC in 403ms. G1 Eden Space: 822083584
> -> 0; G1 Old Gen: 9219080688 -> 9994162168; G1 Survivor Space: 889192448 ->
> 2
> INFO 16:33:41 G1 Young Generation GC in 426ms. G1 Eden Space: 620756992
> -> 0; G1 Old Gen: 10147301880 -> 11065889792; G1 Survivor Space: 1090519040
> -
> INFO 16:38:17 G1 Young Generation GC in 835ms. G1 Eden Space: 251658240
> -> 0; G1 Old Gen: 12062818288 -> 13404581312; G1 Survivor Space: 1459617792
> -
> INFO 16:43:08 G1 Young Generation GC in 569ms. G1 Eden Space: 3154116608
> -> 0; G1 Old Gen: 14361297352 -> 15461740984; G1 Survivor Space: 1258291200
> INFO 16:46:08 G1 Young Generation GC in 432ms. G1 Eden Space:
> 18706595840 -> 0; G1 Old Gen: 3256274416 -> 4355653104; G1 Survivor Space:
> 1023410176 -
> INFO 17:09:31 Redistributing index summaries
> INFO 17:26:10 G1 Young Generation GC in 744ms. G1 Eden Space: 201326592
> -> 0; G1 Old Gen: 6969713640 -> 8462801376; G1 Survivor Space: 1577058304
> ->
> INFO 17:29:04 G1 Young Generation GC in 204ms. G1 Eden Space: 603979776
> -> 0; G1 Old Gen: 8454412776 -> 9439609320; G1 Survivor Space: 1107296256
> ->
> INFO 17:41:09 G1 Young Generation GC in 279ms. G1 Eden Space: 352321536
> -> 0; G1 Old Gen: 13615267816 -> 14860613592; G1 Survivor Space: 1358954496
> -
> INFO 17:46:51 G1 Young Generation GC in 493ms. G1 Eden Space: 754974720
> -> 0; G1 Old Gen: 3560839664 -> 4350490576; G1 Survivor Space: 956301312 ->
> 2
> INFO 17:49:09 G1 Young Generation GC in 290ms. G1 Eden Space: 335544320
> -> 0; G1 Old Gen: 4350490576 -> 5603827664; G1 Survivor Space: 1375731712
> ->
> INFO 17:55:17 G1 Young Generation GC in 803ms. G1 Eden Space: 201326592
> -> 0; G1 Old Gen: 6543114704 -> 8036691400; G1 Survivor Space: 1577058304
> ->
> INFO 17:58:07 G1 Young Generation GC in 206ms. G1 Eden Space: 4764729344
> -> 0; G1 Old Gen: 8036691400 -> 8934491600; G1 Survivor Space: 973078528 ->
> INFO 18:07:29 G1 Young Generation GC in 306ms. G1 Eden Space: 704643072
> -> 0; G1 Old Gen: 12163481536 -> 13045510592; G1 Survivor Space: 1006632960
> -
> INFO 18:08:09 Saved KeyCache (5307 items) in 85 ms
> INFO 18:09:31 Redistributing index summaries
> INFO 18:10:11 G1 Young Generation GC in 277ms. G1 Eden Space: 402653184
> -> 0; G1 Old Gen: 13045510592 -> 14252493256; G1 Survivor Space: 1308622848
> -
> INFO 18:15:39 G1 Young Generation GC in 238ms. G1 Eden Space:
> 13052674048 -> 0; G1 Old Gen: 15141437888 -> 15520873416; G1 Survivor
> Space: 1358954496
> INFO 18:20:16 G1 Young Generation GC in 540ms. G1 Eden Space: 7616856064
> -> 0; G1 Old Gen: 3909694456 -> 4886048752; G1 Survivor Space: 1509949440 ->
> INFO 18:23:04 G1 Young Generation GC in 221ms. G1 Eden Space:
> 16894656512 -> 0; G1 Old Gen: 5420089848 -> 6187322368; G1 Survivor Space:
> 939524096 ->
> INFO 18:42:08 G1 Young Generation GC in 708ms. G1 Eden Space: 352321536
> -> 0; G1 Old Gen: 12840221688 -> 14085112816; G1 Survivor Space: 1358954496
> -
> INFO 18:50:13 G1 Young Generation GC in 318ms. G1 Eden Space: 218103808
> -> 0; G1 Old Gen: 3631963120 -> 5033289720; G1 Survivor Space: 1493172224
> ->
> INFO 19:09:31 Redistributing index summaries
> INFO 19:27:02 G1 Young Generation GC in 384ms. G1 Eden Space:
> 12901679104 -> 0; G1 Old Gen: 7034014192 -> 7759489528; G1 Survivor Space:
> 889192448 ->
> INFO 19:31:25 G1 Young Generation GC in 770ms. G1 Eden Space: 218103808
> -> 0; G1 Old Gen: 8506048488 -> 9921951720; G1 Survivor Space: 1493172224
> ->
> INFO 20:09:31 Redistributing index summaries
> INFO 21:09:31 Redistributing index summaries
> INFO 21:19:39 G1 Young Generation GC in 201ms. G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 3917480432 -> 4124623344; G1 Survivor Space:
> 1459617792 -
> INFO 22:04:05 G1 Young Generation GC in 209ms. G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 4420796912 -> 4618141680; G1 Survivor Space:
> 1459617792 -
> INFO 22:07:49 G1 Young Generation GC in 215ms. G1 Eden Space:
> 14713618432 -> 0; G1 Old Gen: 5970255352 -> 6555802608; G1 Survivor Space:
> 1493172224 -
> INFO 22:08:09 Saved KeyCache (5845 items) in 87 ms
> INFO 22:09:31 Redistributing index summaries
> INFO 22:38:04 G1 Young Generation GC in 201ms. G1 Eden Space:
> 19075694592 -> 0; G1 Old Gen: 4248291312 -> 4459621368; G1 Survivor Space:
> 1526726656 -
> INFO 22:44:12 G1 Young Generation GC in 203ms. G1 Eden Space:
> 10770972672 -> 0; G1 Old Gen: 6703617016 -> 7549747184; G1 Survivor Space:
> 1325400064 -
> INFO 22:47:05 G1 Young Generation GC in 207ms. G1 Eden Space:
> 19025362944 -> 0; G1 Old Gen: 7549747184 -> 7771343352; G1 Survivor Space:
> 1577058304 -
> INFO 23:08:59 G1 Young Generation GC in 369ms. G1 Eden Space:
> 15820914688 -> 0; G1 Old Gen: 4823449592 -> 5478502888; G1 Survivor Space:
> 1660944384 -
> INFO 23:09:31 Redistributing index summaries
> INFO 23:45:21 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO 23:58:40 G1 Young Generation GC in 283ms. G1 Eden Space:
> 18941476864 -> 0; G1 Old Gen: 4831838192 -> 5297926640; G1 Survivor Space:
> 1660944384 -
> INFO 00:01:54 G1 Young Generation GC in 211ms. G1 Eden Space:
> 18891145216 -> 0; G1 Old Gen: 6425673712 -> 6912948208; G1 Survivor Space:
> 1711276032 -
> INFO 00:05:37 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO 00:09:31 Redistributing index summaries
> INFO 00:14:33 G1 Young Generation GC in 282ms. G1 Eden Space:
> 14344519680 -> 0; G1 Old Gen: 14159246832 -> 14677156848; G1 Survivor
> Space: 1207959552
> INFO 00:21:32 G1 Young Generation GC in 225ms. G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 4346926576 -> 4865982456; G1 Survivor Space:
> 1560281088 -
> INFO 00:22:09 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO 00:24:31 G1 Young Generation GC in 266ms. G1 Eden Space:
> 19008585728 -> 0; G1 Old Gen: 6106907128 -> 6613907952; G1 Survivor Space:
> 1593835520 -
> INFO 00:29:46 G1 Young Generation GC in 233ms. G1 Eden Space:
> 18236833792 -> 0; G1 Old Gen: 9076474360 -> 9498544624; G1 Survivor Space:
> 1795162112 -
> INFO 00:32:35 G1 Young Generation GC in 226ms. G1 Eden Space:
> 16492003328 -> 0; G1 Old Gen: 10804527088 -> 11351712752; G1 Survivor
> Space: 1811939328
> INFO 00:37:09 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO 00:41:24 G1 Young Generation GC in 330ms. G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 3426117112 -> 3961507312; G1 Survivor Space:
> 1459617792 -
> INFO 00:44:26 G1 Young Generation GC in 325ms. G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 5268046320 -> 5788210120; G1 Survivor Space:
> 1560281088 -
> INFO 00:47:13 G1 Young Generation GC in 209ms. G1 Eden Space:
> 19243466752 -> 0; G1 Old Gen: 7147093960 -> 7138705360; G1 Survivor Space:
> 1358954496 -
> INFO 00:47:28 G1 Young Generation GC in 312ms. G1 Eden Space:
> 18605932544 -> 0; G1 Old Gen: 7138705360 -> 7710094280; G1 Survivor Space:
> 1996488704 -
> INFO 00:50:35 G1 Young Generation GC in 330ms. G1 Eden Space:
> 17934843904 -> 0; G1 Old Gen: 9118416840 -> 9688820680; G1 Survivor Space:
> 1929379840 -
> INFO 00:52:52 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO 00:56:29 G1 Young Generation GC in 251ms. G1 Eden Space:
> 14797504512 -> 0; G1 Old Gen: 13404995520 -> 13787049928; G1 Survivor
> Space: 1358954496
> INFO 00:58:14 G1 Young Generation GC in 662ms. G1 Eden Space: 369098752
> -> 0; G1 Old Gen: 14730395584 -> 15984921520; G1 Survivor Space: 1342177280
> -
> INFO 01:01:25 G1 Young Generation GC in 288ms. G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 2618580968 -> 3127466464; G1 Survivor Space:
> 1560281088 -
> INFO 01:04:17 G1 Young Generation GC in 201ms. G1 Eden Space:
> 19327352832 -> 0; G1 Old Gen: 4521459688 -> 4529848288; G1 Survivor Space:
> 1275068416 -
> INFO 01:04:39 G1 Young Generation GC in 283ms. G1 Eden Space:
> 18723373056 -> 0; G1 Old Gen: 4529848288 -> 5024210408; G1 Survivor Space:
> 1879048192 -
> INFO 01:07:23 G1 Young Generation GC in 294ms. G1 Eden Space:
> 18740150272 -> 0; G1 Old Gen: 6308233704 -> 6875745256; G1 Survivor Space:
> 1862270976 -
> INFO 01:07:53 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO 01:09:31 Redistributing index summaries
> INFO 01:10:12 G1 Young Generation GC in 213ms. G1 Eden Space:
> 19243466752 -> 0; G1 Old Gen: 8262779360 -> 8254390760; G1 Survivor Space:
> 1358954496 -
> INFO 01:10:33 G1 Young Generation GC in 319ms. G1 Eden Space:
> 18639486976 -> 0; G1 Old Gen: 8254390760 -> 8821123552; G1 Survivor Space:
> 1962934272 -
> INFO 01:22:55 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO 01:24:25 G1 Young Generation GC in 227ms. G1 Eden Space:
> 19226689536 -> 0; G1 Old Gen: 4245307376 -> 4236918776; G1 Survivor Space:
> 1375731712 -
> INFO 01:24:48 G1 Young Generation GC in 219ms. G1 Eden Space:
> 15401484288 -> 0; G1 Old Gen: 4236918776 -> 5231503864; G1 Survivor Space:
> 2097152000 -
> INFO 01:42:01 G1 Young Generation GC in 212ms. G1 Eden Space:
> 16005464064 -> 0; G1 Old Gen: 4320133112 -> 5100782576; G1 Survivor Space:
> 1627389952 -
> INFO 01:48:28 G1 Young Generation GC in 209ms. G1 Eden Space:
> 14730395648 -> 0; G1 Old Gen: 8901994480 -> 9402393592; G1 Survivor Space:
> 1291845632 -
> INFO 02:04:18 G1 Young Generation GC in 241ms. G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 7825270264 -> 8327173624; G1 Survivor Space:
> 1560281088 -
> INFO 02:08:09 Saved KeyCache (8634 items) in 94 ms
> INFO 02:09:31 Redistributing index summaries
> INFO 02:14:50 G1 Young Generation GC in 314ms. G1 Eden Space:
> 19075694592 -> 0; G1 Old Gen: 3505807352 -> 4140375528; G1 Survivor Space:
> 1526726656 -
> INFO 02:48:35 G1 Young Generation GC in 321ms. G1 Eden Space:
> 11576279040 -> 0; G1 Old Gen: 3865890800 -> 4482588664; G1 Survivor Space:
> 1543503872 -
> INFO 02:53:54 G1 Young Generation GC in 203ms. G1 Eden Space:
> 17901289472 -> 0; G1 Old Gen: 7927235056 -> 8552357368; G1 Survivor Space:
> 1593835520 -
> INFO 03:04:33 G1 Young Generation GC in 345ms. G1 Eden Space:
> 13505658880 -> 0; G1 Old Gen: 3765058032 -> 4443454952; G1 Survivor Space:
> 1442840576 -
> INFO 03:09:31 Redistributing index summaries
> INFO 03:22:56 G1 Young Generation GC in 250ms. G1 Eden Space:
> 18991808512 -> 0; G1 Old Gen: 4806672880 -> 5340134904; G1 Survivor Space:
> 1610612736 -
> INFO 03:35:13 G1 Young Generation GC in 210ms. G1 Eden Space:
> 14445182976 -> 0; G1 Survivor Space: 1073741824 -> 1744830464;
> INFO 03:38:42 G1 Young Generation GC in 214ms. G1 Eden Space:
> 18303942656 -> 0; G1 Old Gen: 4786540536 -> 5432116720; G1 Survivor Space:
> 1375731712 -
> INFO 03:52:11 G1 Young Generation GC in 207ms. G1 Eden Space: 117440512
> -> 0; G1 Old Gen: 12054831088 -> 10154180600; G1 Survivor Space: 1593835520
> -
> INFO 03:54:48 G1 Young Generation GC in 224ms. G1 Eden Space:
> 16156459008 -> 0; G1 Old Gen: 4303355896 -> 4801558520; G1 Survivor Space:
> 1560281088 -
> INFO 04:09:31 Redistributing index summaries
> INFO 04:16:09 G1 Young Generation GC in 216ms. G1 Eden Space:
> 19142803456 -> 0; G1 Old Gen: 5853311984 -> 6366188536; G1 Survivor Space:
> 1459617792 -
> INFO 04:19:13 G1 Young Generation GC in 229ms. G1 Eden Space:
> 19042140160 -> 0; G1 Old Gen: 7826571256 -> 8457920504; G1 Survivor Space:
> 1560281088 -
> INFO 04:27:41 G1 Young Generation GC in 311ms. G1 Eden Space:
> 13723762688 -> 0; G1 Old Gen: 14176747504 -> 14808230392; G1 Survivor
> Space: 1509949440
> INFO 04:29:56 G1 Young Generation GC in 274ms. G1 Eden Space:
> 18740150272 -> 0; G1 Old Gen: 3376452080 -> 3818638328; G1 Survivor Space:
> 1459617792 -
> INFO 04:33:40 Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB
> INFO 04:49:07 G1 Young Generation GC in 319ms. G1 Eden Space:
> 16005464064 -> 0; G1 Old Gen: 4435714040 -> 5023449072; G1 Survivor Space:
> 1442840576 -
> INFO 04:54:06 G1 Young Generation GC in 208ms. G1 Eden Space:
> 18706595840 -> 0; G1 Old Gen: 7843348472 -> 8466889208; G1 Survivor Space:
> 1476395008 -
> INFO 05:09:31 Redistributing index summaries
>
>
> I found INFO Maximum memory usage reached (2.000GiB), cannot allocate
> chunk of 1.000MiB, does it mean the memory is not enough ?
>
>
> I want to know why adding a new node spent so much time and what need
> I do to complete a new node joining the cluster ?
>
>
> I need some help and really need some suggestions.
>
>
>
> Thanks a lot!
>
>
>
>
>
>
> ------------------------------
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>
>
>
Re: [EXTERNAL] Cassandra cluster add new node slowly
Posted by qf zhou <zh...@gmail.com>.
The cassandra version is 3.0.9.
I have changed the heap size (about 32G). Also, the streaming throughput is set 800MB/sec, and the streaming_socket_timeout_in_ms is default 86400000.
I suspect the compactionthroughput has an influence on the new node joining. The command nodetool | getcompactionthroughput says 'Current compaction throughput: 32 MB/s’.
> 在 2018年1月4日，上午4:59，Durity, Sean R <SE...@homedepot.com> 写道：
>
> You don’t mention the version, but here are some general suggestions
>
> - 2 GB heap is very small for a node, especially with 1 TB+ of data. What is the physical RAM on the host? In general, you want ½ of physical RAM for the JVM. (Look in jvm.options or cassandra-env.sh)
> - You can change the streaming throughput from the existing nodes, if it looks like the new node can handle it. Look at nodetool setstreamthroughput. Default is 200 (MB/sec).
> - You might want to check for a streaming_socket_timeout_in_ms. This has changed over the versions. Some details are at: https://issues.apache.org/jira/browse/CASSANDRA-11839 <https://issues.apache.org/jira/browse/CASSANDRA-11839>. 24 hours is good recommendation.
> - If your new node can’t compact fast enough to keep disk usage down, look at compactionthroughput on that node
> - nodetool netstats | grep –v “100%” is a good way to see what is happening/if anything is stuck. Newer versions give a bit more info on progress.
> - Don’t forget to run cleanup on existing nodes after the new nodes are added.
>
>
>
> Sean Durity
> From: qf zhou [mailto:zhouqf2013@gmail.com]
> Sent: Tuesday, January 02, 2018 10:30 PM
> To: user@cassandra.apache.org
> Subject: [EXTERNAL] Cassandra cluster add new node slowly
>
> The cluster has 3 nodes, and the data in each node is about 1.2 T. I want to add two new nodes to expand the cluster.
>
> Following the instructions from the datastax website, ie, (http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/operations/opsAddNodeToCluster.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.datastax.com_en_archived_cassandra_3.x_cassandra_operations_opsAddNodeToCluster.html&d=DwMFoQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=AfP0BxfTG1o_-2nRtGSjngs45YndH060h39eVlypVso&s=AaiOypurMtWA470038fHMbaXJPIZeYtLzv3Yt9vf1Sc&e=>),
>
> I try to add one node to the cluster. However, it is too slow and time cost too much. After about 24 hours, it still didn’t success.
>
> I run the command: nodetool netstats on the new node, it shows that:
>
> (tb1fullwithstate2 is a big table and 90% of the cluster data is in it. Here I use CompactionStrategy: TimeWindowCompactionStrategy).
>
> /*.*.*.3
> Receiving 136 files, 328573794609 bytes total. Already received 8 files, 5774621188 bytes total
> tb1/tb1fullwithneweststatetest 3758271/3758271 bytes(100%) received from idx:0/*.*.*.3
> system_distributed/repair_history 57534/57534 bytes(100%) received from idx:0/*.*.*.3
> system_distributed/parent_repair_history 507660/507660 bytes(100%) received from idx:0/*.*.*.3
> tb1/tb1_device_last_state_eachday 15754096/15754096 bytes(100%) received from idx:0/*.*.*.3
> mytest1/tb1_test1 8143775/8143775 bytes(100%) received from idx:0/*.*.*.3
> tb1/tb1fullwithstate 2251191007/2251191007 bytes(100%) received from idx:0/*.*.*.3
> applocationinfo/weiyirong_app 2760/2760 bytes(100%) received from idx:0/*.*.*.3
> tb1/tb1fullwithstate2 3490748006/4909554503 bytes(71%) received from idx:0/*.*.*.3
> tb1/tb1fullwithneweststate 4458079/4458079 bytes(100%) received from idx:0/*.*.*.3
> /*.*.*.2
> Receiving 136 files, 336762487360 bytes total. Already received 3 files, 5695770181 bytes total
> system_distributed/repair_history 31684/31684 bytes(100%) received from idx:0/*.*.*.2
> tb1/tb1fullwithstate 908260516/908260516 bytes(100%) received from idx:0/*.*.*.2
> tb1/tb1fullwithstate2 4783622958/4990450588 bytes(95%) received from idx:0/*.*.*.2
> tb1/tb1fullwithneweststate 3855023/3855023 bytes(100%) received from idx:0/*.*.*.2
> /*.*.*.4
> Receiving 132 files, 236250553620 bytes total. Already received 10 files, 3117465128 bytes total
> mytest1/wordstest2 46/46 bytes(100%) received from idx:0/*.*.*.4
> tb1/tb1fullwithneweststatetest 3416891/3416891 bytes(100%) received from idx:0/*.*.*.4
> system_distributed/repair_history 39720/39720 bytes(100%) received from idx:0/*.*.*.4
> system_distributed/parent_repair_history 452250/452250 bytes(100%) received from idx:0/*.*.*.4
> mytest1/weblogs 104/104 bytes(100%) received from idx:0/*.*.*.4
> tb1/tb1_device_last_state_eachday 12670998/12670998 bytes(100%) received from idx:0/*.*.*.4
> mytest1/tb1_test1 3257952/3257952 bytes(100%) received from idx:0/*.*.*.4
> tb1/tb1fullwithstate 647702056/647702056 bytes(100%) received from idx:0/*.*.*.4
> applocationinfo/weiyirong_app 3509/3509 bytes(100%) received from idx:0/*.*.*.4
> tb1/tb1fullwithstate2 2446436305/3566562762 bytes(68%) received from idx:0/*.*.*.4
> tb1/tb1fullwithneweststate 3485297/3485297 bytes(100%) received from idx:0/*.*.*.4
>
>
>
> check the log in the logs/system.log, it shows that:
>
>
> INFO 06:09:33 Updating topology for /*.*.*.2
> INFO 06:09:33 Updating topology for /*.*.*.2
> INFO 06:09:33 Node /*.*.*.3 has restarted, now UP
> INFO 06:09:33 InetAddress /*.*.*.3 is now UP
> INFO 06:09:33 Node /*.*.*.3 state jump to NORMAL
> INFO 06:09:33 Updating topology for /*.*.*.3
> INFO 06:09:33 Updating topology for /*.*.*.3
> INFO 06:09:33 Handshaking version with /*.*.*.2
> INFO 06:09:33 Handshaking version with /*.*.*.3
> INFO 06:09:36 Handshaking version with /*.*.*.4
> INFO 06:10:03 JOINING: Starting to bootstrap...
> INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Executing streaming plan for Bootstrap
> INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.4
> INFO 06:10:04 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.4
> INFO 06:10:04 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints
> INFO 06:10:04 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527790874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
> INFO 06:10:11 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints
> INFO 06:10:11 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527800873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
> INFO 06:10:12 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints
> INFO 06:10:12 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527790881-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
> INFO 06:10:20 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints
> INFO 06:10:20 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527800880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
> INFO 06:10:21 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints
> INFO 06:10:21 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527790884-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
> INFO 06:10:28 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints
> INFO 06:10:28 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527800887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
> INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 132 files(220.025GiB), sending 0 files(0.000KiB)
> INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.2
> INFO 06:10:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.2
> INFO 06:10:34 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints
> INFO 06:10:34 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527810873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
> INFO 06:10:42 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints
> INFO 06:10:42 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527820873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
> INFO 06:10:58 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints
> INFO 06:10:58 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527810881-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
> INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 136 files(313.635GiB), sending 0 files(0.000KiB)
> INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75] Starting streaming to /*.*.*.3
> INFO 06:11:03 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75, ID#0] Beginning stream session with /*.*.*.3
> INFO 06:11:06 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints
> INFO 06:11:06 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527830880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
> INFO 06:11:29 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints
> INFO 06:11:29 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527810889-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
> INFO 06:11:33 [Stream #e8d6c640-ec5e-11e7-9d6a-bfe698600c75 ID#0] Prepare completed. Receiving 136 files(306.008GiB), sending 0 files(0.000KiB)
> INFO 06:11:36 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints
> INFO 06:11:36 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527840887-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
> INFO 06:11:49 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints
> INFO 06:11:49 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527830873-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
> INFO 06:11:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints
> INFO 06:11:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527850874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
> INFO 06:12:26 Deleted hint file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints
> INFO 06:12:26 Finished hinted handoff of file c555e44c-9590-4f45-aea4-f5eca68180b2-1514527840880-1.hints to endpoint c555e44c-9590-4f45-aea4-f5eca6818
> INFO 06:12:45 Deleted hint file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints
> INFO 06:12:45 Finished hinted handoff of file c623bbc0-9839-4d2d-8ff3-db7115719d59-1514527850883-1.hints to endpoint c623bbc0-9839-4d2d-8ff3-db7115719
> INFO 06:12:56 Deleted hint file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints
> INFO 06:12:56 Finished hinted handoff of file c5271e74-19a1-4cee-98d7-dc169cf87e95-1514527860874-1.hints to endpoint c5271e74-19a1-4cee-98d7-dc169cf87
> INFO 06:24:03 G1 Young Generation GC in 242ms. G1 Eden Space: 18924699648 -> 0; G1 Old Gen: 4360876536 -> 5095617016; G1 Survivor Space: 1677721600 -
> INFO 06:26:06 G1 Young Generation GC in 215ms. G1 Eden Space: 18220056576 -> 0; G1 Old Gen: 5981077496 -> 6597610488; G1 Survivor Space: 1493172224 -
> INFO 06:29:04 G1 Young Generation GC in 225ms. G1 Eden Space: 18538823680 -> 0; G1 Old Gen: 7910457840 -> 8860639728; G1 Survivor Space: 1895825408 -
> INFO 06:33:22 G1 Young Generation GC in 214ms. G1 Eden Space: 16559112192 -> 0; G1 Old Gen: 11868321784 -> 11941600752; G1 Survivor Space: 1241513984
> INFO 06:33:33 G1 Young Generation GC in 205ms. G1 Eden Space: 8069840896 -> 0; G1 Old Gen: 11941600752 -> 13683606520; G1 Survivor Space: 1895825408
> INFO 06:41:01 G1 Young Generation GC in 383ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 4770399736 -> 5551887344; G1 Survivor Space: 1728053248 -
> INFO 06:45:07 G1 Young Generation GC in 226ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 8036286448 -> 8739698672; G1 Survivor Space: 1577058304 -
> INFO 06:47:11 G1 Young Generation GC in 203ms. G1 Eden Space: 18538823680 -> 0; G1 Survivor Space: 1191182336 -> 1778384896;
> INFO 06:47:22 G1 Young Generation GC in 241ms. G1 Eden Space: 8086618112 -> 0; G1 Old Gen: 9999220720 -> 11591876088; G1 Survivor Space: 1778384896 -
> INFO 06:49:43 G1 Young Generation GC in 201ms. G1 Eden Space: 16022241280 -> 0; G1 Old Gen: 11591876088 -> 12084766712; G1 Survivor Space: 1644167168
> INFO 06:53:21 G1 Young Generation GC in 322ms. G1 Eden Space: 13237223424 -> 0; G1 Old Gen: 14755561464 -> 15445886448; G1 Survivor Space: 1459617792
> INFO 06:55:34 G1 Young Generation GC in 343ms. G1 Eden Space: 16525557760 -> 0; G1 Old Gen: 2939907056 -> 3636770792; G1 Survivor Space: 1610612736 -
> INFO 07:09:30 Redistributing index summaries
> INFO 07:14:25 G1 Young Generation GC in 270ms. G1 Eden Space: 15602810880 -> 0; G1 Old Gen: 4360518136 -> 5243274224; G1 Survivor Space: 1610612736 -
> INFO 07:16:51 G1 Young Generation GC in 229ms. G1 Eden Space: 19008585728 -> 0; G1 Old Gen: 6132072440 -> 6815145976; G1 Survivor Space: 1593835520 -
> INFO 07:19:34 G1 Young Generation GC in 215ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 8689337336 -> 9071161328; G1 Survivor Space: 1610612736 -
> INFO 07:23:35 G1 Young Generation GC in 293ms. G1 Eden Space: 15753805824 -> 0; G1 Old Gen: 11777605616 -> 12289649640; G1 Survivor Space: 1694498816
> INFO 07:29:42 G1 Young Generation GC in 362ms. G1 Eden Space: 15955132416 -> 0; G1 Old Gen: 3742637560 -> 4410361824; G1 Survivor Space: 1644167168 -
> INFO 07:35:12 G1 Young Generation GC in 208ms. G1 Eden Space: 19478347776 -> 0; G1 Old Gen: 7199737832 -> 7208126432; G1 Survivor Space: 1124073472 -
> INFO 07:35:33 G1 Young Generation GC in 219ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 7208126432 -> 8072478184; G1 Survivor Space: 1862270976 -
> INFO 07:45:19 G1 Young Generation GC in 213ms. G1 Eden Space: 13153337344 -> 0; G1 Old Gen: 15418261472 -> 16123805152; G1 Survivor Space: 1174405120
> INFO 07:48:08 G1 Young Generation GC in 335ms. G1 Eden Space: 18387828736 -> 0; G1 Old Gen: 3852453368 -> 4464797168; G1 Survivor Space: 1644167168 -
> INFO 07:48:08 G1 Young Generation GC in 209ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 4464797168 -> 5989466096; G1 Survivor Space: 1476395008 ->
> INFO 07:50:25 G1 Young Generation GC in 221ms. G1 Eden Space: 17498636288 -> 0; G1 Old Gen: 5981077496 -> 6488668656; G1 Survivor Space: 1593835520 -
> INFO 08:03:31 G1 Young Generation GC in 748ms. G1 Eden Space: 8673820672 -> 0; G1 Old Gen: 4386367472 -> 5874747352; G1 Survivor Space: 1627389952 ->
> INFO 08:09:30 Redistributing index summaries
> INFO 08:10:05 G1 Young Generation GC in 213ms. G1 Eden Space: 18220056576 -> 0; G1 Old Gen: 9276024272 -> 9969348048; G1 Survivor Space: 1711276032 -
> INFO 08:18:59 G1 Young Generation GC in 228ms. G1 Eden Space: 17347641344 -> 0; G1 Old Gen: 4546626040 -> 5180788216; G1 Survivor Space: 1593835520 -
> INFO 08:33:12 G1 Young Generation GC in 207ms. G1 Eden Space: 19411238912 -> 0; G1 Survivor Space: 1191182336 -> 1828716544;
> INFO 08:33:13 G1 Young Generation GC in 273ms. G1 Eden Space: 452984832 -> 0; G1 Old Gen: 4668340728 -> 6320276984; G1 Survivor Space: 1828716544 ->
> INFO 08:41:59 G1 Young Generation GC in 202ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 8701765104 -> 8927144944; G1 Survivor Space: 1577058304 -
> INFO 08:56:31 G1 Young Generation GC in 205ms. G1 Eden Space: 11072962560 -> 0; G1 Old Gen: 9563013104 -> 11068430328; G1 Survivor Space: 1660944384
> INFO 09:05:49 G1 Young Generation GC in 367ms. G1 Eden Space: 18522046464 -> 0; G1 Old Gen: 4009754608 -> 4732347872; G1 Survivor Space: 1778384896 -
> INFO 09:09:30 Redistributing index summaries
> INFO 09:14:52 G1 Young Generation GC in 213ms. G1 Eden Space: 15267266560 -> 0; G1 Old Gen: 12155092968 -> 12906235360; G1 Survivor Space: 1744830464
> INFO 09:17:23 G1 Young Generation GC in 203ms. G1 Eden Space: 14159970304 -> 0; G1 Old Gen: 14779568608 -> 14771180008; G1 Survivor Space: 989855744
> INFO 09:17:45 G1 Young Generation GC in 204ms. G1 Eden Space: 12616466432 -> 0; G1 Old Gen: 14771180008 -> 15500684768; G1 Survivor Space: 1761607680
> INFO 09:20:52 G1 Young Generation GC in 361ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 4351990768 -> 5086824424; G1 Survivor Space: 1577058304 -
> INFO 09:23:51 G1 Young Generation GC in 207ms. G1 Eden Space: 18891145216 -> 0; G1 Old Gen: 7001879024 -> 7716560880; G1 Survivor Space: 1711276032 -
> INFO 09:29:20 G1 Young Generation GC in 202ms. G1 Eden Space: 17028874240 -> 0; G1 Survivor Space: 1040187392 -> 1828716544;
> INFO 09:29:40 G1 Young Generation GC in 219ms. G1 Eden Space: 15451815936 -> 0; G1 Old Gen: 11802771944 -> 12574988776; G1 Survivor Space: 1828716544
> INFO 09:44:51 G1 Young Generation GC in 202ms. G1 Eden Space: 18001952768 -> 0; G1 Old Gen: 9325481456 -> 10085596664; G1 Survivor Space: 1795162112
> INFO 09:45:23 G1 Young Generation GC in 209ms. G1 Eden Space: 9177137152 -> 0; G1 Old Gen: 10093985264 -> 12309118960; G1 Survivor Space: 1677721600
> INFO 09:50:33 G1 Young Generation GC in 201ms. G1 Eden Space: 12901679104 -> 0; G1 Old Gen: 14487498224 -> 15268560880; G1 Survivor Space: 1761607680
> INFO 10:08:09 Saved KeyCache (2989 items) in 92 ms
> INFO 10:09:30 Redistributing index summaries
> INFO 10:22:17 G1 Young Generation GC in 216ms. G1 Eden Space: 19495124992 -> 0; G1 Old Gen: 7742571504 -> 7734182904; G1 Survivor Space: 1107296256 -
> INFO 10:25:31 G1 Young Generation GC in 206ms. G1 Eden Space: 18622709760 -> 0; G1 Survivor Space: 1040187392 -> 1711276032;
> INFO 10:34:54 G1 Young Generation GC in 251ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 5066719736 -> 5691775992; G1 Survivor Space: 1610612736 -
> INFO 10:40:08 G1 Young Generation GC in 209ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 9000976880 -> 9090653168; G1 Survivor Space: 1442840576 -
> INFO 10:40:08 G1 Young Generation GC in 223ms. G1 Eden Space: 469762048 -> 0; G1 Old Gen: 9090653168 -> 11140124152; G1 Survivor Space: 1862270976 ->
> INFO 10:52:15 G1 Young Generation GC in 219ms. G1 Eden Space: 15820914688 -> 0; G1 Old Gen: 5331537912 -> 5828939768; G1 Survivor Space: 1275068416 -
> INFO 10:55:30 G1 Young Generation GC in 207ms. G1 Eden Space: 15804137472 -> 0; G1 Old Gen: 7501364216 -> 8001282032; G1 Survivor Space: 1459617792 -
> INFO 11:07:57 G1 Young Generation GC in 295ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4301379064 -> 4919709168; G1 Survivor Space: 1442840576 -
> INFO 11:09:30 Redistributing index summaries
> INFO 11:13:06 G1 Young Generation GC in 204ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 8044675064 -> 8659737080; G1 Survivor Space: 1476395008 -
> INFO 11:27:07 G1 Young Generation GC in 202ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4328521712 -> 4905718776; G1 Survivor Space: 1442840576 -
> INFO 11:30:08 G1 Young Generation GC in 209ms. G1 Eden Space: 19243466752 -> 0; G1 Survivor Space: 1358954496 -> 1828716544;
> INFO 11:30:25 G1 Young Generation GC in 226ms. G1 Eden Space: 14277410816 -> 0; G1 Old Gen: 6349296624 -> 7552639472; G1 Survivor Space: 1828716544 -
> INFO 11:32:30 G1 Young Generation GC in 215ms. G1 Eden Space: 19142803456 -> 0; G1 Survivor Space: 1459617792 -> 2046820352;
> INFO 11:33:01 G1 Young Generation GC in 247ms. G1 Eden Space: 17263755264 -> 0; G1 Old Gen: 7544250872 -> 8698295280; G1 Survivor Space: 2046820352 -
> INFO 11:46:29 G1 Young Generation GC in 355ms. G1 Eden Space: 17230200832 -> 0; G1 Old Gen: 4769358328 -> 5362416624; G1 Survivor Space: 1761607680 -
> INFO 11:49:18 G1 Young Generation GC in 233ms. G1 Eden Space: 17867735040 -> 0; G1 Old Gen: 6752829432 -> 6944805360; G1 Survivor Space: 1509949440 -
> INFO 11:52:24 G1 Young Generation GC in 209ms. G1 Eden Space: 19193135104 -> 0; G1 Survivor Space: 1409286144 -> 1946157056;
> INFO 11:52:47 G1 Young Generation GC in 338ms. G1 Eden Space: 18287165440 -> 0; G1 Old Gen: 8740881904 -> 9340426216; G1 Survivor Space: 1946157056 -
> INFO 12:04:08 G1 Young Generation GC in 810ms. G1 Eden Space: 18119393280 -> 0; G1 Old Gen: 3471267832 -> 5003518440; G1 Survivor Space: 1325400064 -
> INFO 12:06:28 G1 Young Generation GC in 345ms. G1 Eden Space: 8673820672 -> 0; G1 Old Gen: 5318377952 -> 6577762784; G1 Survivor Space: 1409286144 ->
> INFO 12:09:31 G1 Young Generation GC in 204ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 6577762784 -> 6586151384; G1 Survivor Space: 1442840576 -
> INFO 12:09:31 Redistributing index summaries
> INFO 12:10:01 G1 Young Generation GC in 298ms. G1 Eden Space: 17230200832 -> 0; G1 Old Gen: 6586151384 -> 7710758368; G1 Survivor Space: 1946157056 -
> INFO 12:12:37 G1 Young Generation GC in 214ms. G1 Eden Space: 18773704704 -> 0; G1 Old Gen: 8380219864 -> 8919123416; G1 Survivor Space: 1728053248 -
> INFO 12:24:54 G1 Young Generation GC in 299ms. G1 Eden Space: 16055795712 -> 0; G1 Old Gen: 4375908856 -> 5009812976; G1 Survivor Space: 1392508928 -
> INFO 12:28:34 G1 Young Generation GC in 485ms. G1 Eden Space: 13304332288 -> 0; G1 Old Gen: 6481607672 -> 7506585072; G1 Survivor Space: 1979711488 -
> INFO 12:44:52 G1 Young Generation GC in 261ms. G1 Eden Space: 18773704704 -> 0; G1 Old Gen: 4239590896 -> 4692442608; G1 Survivor Space: 1828716544 -
> INFO 12:48:37 G1 Young Generation GC in 251ms. G1 Eden Space: 13656653824 -> 0; G1 Old Gen: 6719275504 -> 7776254960; G1 Survivor Space: 1862270976 -
> INFO 12:53:47 G1 Young Generation GC in 217ms. G1 Eden Space: 17548967936 -> 0; G1 Old Gen: 9667142640 -> 10202159608; G1 Survivor Space: 1845493760
> INFO 13:05:40 G1 Young Generation GC in 287ms. G1 Eden Space: 18320719872 -> 0; G1 Old Gen: 5298861040 -> 5826686456; G1 Survivor Space: 1862270976 -
> INFO 13:09:31 Redistributing index summaries
> INFO 13:11:44 G1 Young Generation GC in 268ms. G1 Eden Space: 10871635968 -> 0; G1 Old Gen: 8799649784 -> 9797878264; G1 Survivor Space: 1811939328 -
> INFO 13:14:22 G1 Young Generation GC in 202ms. G1 Eden Space: 17750294528 -> 0; G1 Survivor Space: 1325400064 -> 1795162112;
> INFO 13:14:40 G1 Young Generation GC in 254ms. G1 Eden Space: 14092861440 -> 0; G1 Old Gen: 10509551608 -> 11563311096; G1 Survivor Space: 1795162112
> INFO 13:16:33 G1 Young Generation GC in 213ms. G1 Eden Space: 16039018496 -> 0; G1 Old Gen: 11619767792 -> 12115354104; G1 Survivor Space: 1627389952
> INFO 13:25:40 G1 Young Generation GC in 245ms. G1 Eden Space: 18790481920 -> 0; G1 Old Gen: 4253024752 -> 4761733616; G1 Survivor Space: 1811939328 -
> INFO 13:27:38 G1 Young Generation GC in 208ms. G1 Eden Space: 13958643712 -> 0; G1 Old Gen: 5377097720 -> 6427987960; G1 Survivor Space: 1912602624 -
> INFO 13:29:30 G1 Young Generation GC in 207ms. G1 Eden Space: 12264144896 -> 0; G1 Old Gen: 6427987960 -> 7337681912; G1 Survivor Space: 1795162112 -
> INFO 13:33:32 G1 Young Generation GC in 202ms. G1 Eden Space: 18589155328 -> 0; G1 Old Gen: 8506316280 -> 8514704880; G1 Survivor Space: 1308622848 -
> INFO 13:35:09 G1 Young Generation GC in 203ms. G1 Eden Space: 17532190720 -> 0; G1 Survivor Space: 1258291200 -> 1761607680;
> INFO 13:38:33 G1 Young Generation GC in 216ms. G1 Eden Space: 11106516992 -> 0; G1 Old Gen: 11571635704 -> 12698973168; G1 Survivor Space: 1795162112
> INFO 13:47:36 G1 Young Generation GC in 229ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 3825205752 -> 4303660528; G1 Survivor Space: 1526726656 -
> INFO 13:53:43 G1 Young Generation GC in 207ms. G1 Eden Space: 14042529792 -> 0; G1 Old Gen: 7324985848 -> 8363582960; G1 Survivor Space: 1660944384 -
> INFO 13:55:40 G1 Young Generation GC in 201ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 8355194360 -> 8820888560; G1 Survivor Space: 1728053248 -
> INFO 13:56:22 G1 Young Generation GC in 206ms. G1 Eden Space: 19394461696 -> 0; G1 Survivor Space: 1207959552 -> 1711276032;
> INFO 14:08:09 Saved KeyCache (4563 items) in 189 ms
> INFO 14:09:31 Redistributing index summaries
> INFO 14:12:47 G1 Young Generation GC in 271ms. G1 Eden Space: 18874368000 -> 0; G1 Old Gen: 4345299448 -> 4814148600; G1 Survivor Space: 1728053248 -
> INFO 14:18:05 G1 Young Generation GC in 224ms. G1 Eden Space: 18907922432 -> 0; G1 Old Gen: 7021265392 -> 7483297784; G1 Survivor Space: 1627389952 -
> INFO 14:36:05 G1 Young Generation GC in 304ms. G1 Eden Space: 18958254080 -> 0; G1 Old Gen: 3976200696 -> 4437455344; G1 Survivor Space: 1644167168 -
> INFO 14:36:07 G1 Young Generation GC in 212ms. G1 Eden Space: 1711276032 -> 0; G1 Old Gen: 4437455344 -> 5830082552; G1 Survivor Space: 1459617792 ->
> INFO 15:00:30 G1 Young Generation GC in 233ms. G1 Eden Space: 19159580672 -> 0; G1 Old Gen: 4040399864 -> 4343952368; G1 Survivor Space: 1442840576 -
> INFO 15:09:31 Redistributing index summaries
> INFO 15:26:28 G1 Young Generation GC in 234ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 4604513264 -> 5064485872; G1 Survivor Space: 1526726656 -
> INFO 15:52:08 G1 Young Generation GC in 222ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 4118806520 -> 4391313912; G1 Survivor Space: 1476395008 -
> INFO 15:55:06 G1 Young Generation GC in 202ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 5267292152 -> 5559551472;
> INFO 16:09:31 Redistributing index summaries
> INFO 16:20:48 G1 Young Generation GC in 241ms. G1 Eden Space: 19126026240 -> 0; G1 Old Gen: 3854587896 -> 4287600112; G1 Survivor Space: 1476395008 -
> INFO 16:31:07 G1 Young Generation GC in 403ms. G1 Eden Space: 822083584 -> 0; G1 Old Gen: 9219080688 -> 9994162168; G1 Survivor Space: 889192448 -> 2
> INFO 16:33:41 G1 Young Generation GC in 426ms. G1 Eden Space: 620756992 -> 0; G1 Old Gen: 10147301880 -> 11065889792; G1 Survivor Space: 1090519040 -
> INFO 16:38:17 G1 Young Generation GC in 835ms. G1 Eden Space: 251658240 -> 0; G1 Old Gen: 12062818288 -> 13404581312; G1 Survivor Space: 1459617792 -
> INFO 16:43:08 G1 Young Generation GC in 569ms. G1 Eden Space: 3154116608 -> 0; G1 Old Gen: 14361297352 -> 15461740984; G1 Survivor Space: 1258291200
> INFO 16:46:08 G1 Young Generation GC in 432ms. G1 Eden Space: 18706595840 -> 0; G1 Old Gen: 3256274416 -> 4355653104; G1 Survivor Space: 1023410176 -
> INFO 17:09:31 Redistributing index summaries
> INFO 17:26:10 G1 Young Generation GC in 744ms. G1 Eden Space: 201326592 -> 0; G1 Old Gen: 6969713640 -> 8462801376; G1 Survivor Space: 1577058304 ->
> INFO 17:29:04 G1 Young Generation GC in 204ms. G1 Eden Space: 603979776 -> 0; G1 Old Gen: 8454412776 -> 9439609320; G1 Survivor Space: 1107296256 ->
> INFO 17:41:09 G1 Young Generation GC in 279ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 13615267816 -> 14860613592; G1 Survivor Space: 1358954496 -
> INFO 17:46:51 G1 Young Generation GC in 493ms. G1 Eden Space: 754974720 -> 0; G1 Old Gen: 3560839664 -> 4350490576; G1 Survivor Space: 956301312 -> 2
> INFO 17:49:09 G1 Young Generation GC in 290ms. G1 Eden Space: 335544320 -> 0; G1 Old Gen: 4350490576 -> 5603827664; G1 Survivor Space: 1375731712 ->
> INFO 17:55:17 G1 Young Generation GC in 803ms. G1 Eden Space: 201326592 -> 0; G1 Old Gen: 6543114704 -> 8036691400; G1 Survivor Space: 1577058304 ->
> INFO 17:58:07 G1 Young Generation GC in 206ms. G1 Eden Space: 4764729344 -> 0; G1 Old Gen: 8036691400 -> 8934491600; G1 Survivor Space: 973078528 ->
> INFO 18:07:29 G1 Young Generation GC in 306ms. G1 Eden Space: 704643072 -> 0; G1 Old Gen: 12163481536 -> 13045510592; G1 Survivor Space: 1006632960 -
> INFO 18:08:09 Saved KeyCache (5307 items) in 85 ms
> INFO 18:09:31 Redistributing index summaries
> INFO 18:10:11 G1 Young Generation GC in 277ms. G1 Eden Space: 402653184 -> 0; G1 Old Gen: 13045510592 -> 14252493256; G1 Survivor Space: 1308622848 -
> INFO 18:15:39 G1 Young Generation GC in 238ms. G1 Eden Space: 13052674048 -> 0; G1 Old Gen: 15141437888 -> 15520873416; G1 Survivor Space: 1358954496
> INFO 18:20:16 G1 Young Generation GC in 540ms. G1 Eden Space: 7616856064 -> 0; G1 Old Gen: 3909694456 -> 4886048752; G1 Survivor Space: 1509949440 ->
> INFO 18:23:04 G1 Young Generation GC in 221ms. G1 Eden Space: 16894656512 -> 0; G1 Old Gen: 5420089848 -> 6187322368; G1 Survivor Space: 939524096 ->
> INFO 18:42:08 G1 Young Generation GC in 708ms. G1 Eden Space: 352321536 -> 0; G1 Old Gen: 12840221688 -> 14085112816; G1 Survivor Space: 1358954496 -
> INFO 18:50:13 G1 Young Generation GC in 318ms. G1 Eden Space: 218103808 -> 0; G1 Old Gen: 3631963120 -> 5033289720; G1 Survivor Space: 1493172224 ->
> INFO 19:09:31 Redistributing index summaries
> INFO 19:27:02 G1 Young Generation GC in 384ms. G1 Eden Space: 12901679104 -> 0; G1 Old Gen: 7034014192 -> 7759489528; G1 Survivor Space: 889192448 ->
> INFO 19:31:25 G1 Young Generation GC in 770ms. G1 Eden Space: 218103808 -> 0; G1 Old Gen: 8506048488 -> 9921951720; G1 Survivor Space: 1493172224 ->
> INFO 20:09:31 Redistributing index summaries
> INFO 21:09:31 Redistributing index summaries
> INFO 21:19:39 G1 Young Generation GC in 201ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 3917480432 -> 4124623344; G1 Survivor Space: 1459617792 -
> INFO 22:04:05 G1 Young Generation GC in 209ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 4420796912 -> 4618141680; G1 Survivor Space: 1459617792 -
> INFO 22:07:49 G1 Young Generation GC in 215ms. G1 Eden Space: 14713618432 -> 0; G1 Old Gen: 5970255352 -> 6555802608; G1 Survivor Space: 1493172224 -
> INFO 22:08:09 Saved KeyCache (5845 items) in 87 ms
> INFO 22:09:31 Redistributing index summaries
> INFO 22:38:04 G1 Young Generation GC in 201ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 4248291312 -> 4459621368; G1 Survivor Space: 1526726656 -
> INFO 22:44:12 G1 Young Generation GC in 203ms. G1 Eden Space: 10770972672 -> 0; G1 Old Gen: 6703617016 -> 7549747184; G1 Survivor Space: 1325400064 -
> INFO 22:47:05 G1 Young Generation GC in 207ms. G1 Eden Space: 19025362944 -> 0; G1 Old Gen: 7549747184 -> 7771343352; G1 Survivor Space: 1577058304 -
> INFO 23:08:59 G1 Young Generation GC in 369ms. G1 Eden Space: 15820914688 -> 0; G1 Old Gen: 4823449592 -> 5478502888; G1 Survivor Space: 1660944384 -
> INFO 23:09:31 Redistributing index summaries
> INFO 23:45:21 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
> INFO 23:58:40 G1 Young Generation GC in 283ms. G1 Eden Space: 18941476864 -> 0; G1 Old Gen: 4831838192 -> 5297926640; G1 Survivor Space: 1660944384 -
> INFO 00:01:54 G1 Young Generation GC in 211ms. G1 Eden Space: 18891145216 -> 0; G1 Old Gen: 6425673712 -> 6912948208; G1 Survivor Space: 1711276032 -
> INFO 00:05:37 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
> INFO 00:09:31 Redistributing index summaries
> INFO 00:14:33 G1 Young Generation GC in 282ms. G1 Eden Space: 14344519680 -> 0; G1 Old Gen: 14159246832 -> 14677156848; G1 Survivor Space: 1207959552
> INFO 00:21:32 G1 Young Generation GC in 225ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 4346926576 -> 4865982456; G1 Survivor Space: 1560281088 -
> INFO 00:22:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
> INFO 00:24:31 G1 Young Generation GC in 266ms. G1 Eden Space: 19008585728 -> 0; G1 Old Gen: 6106907128 -> 6613907952; G1 Survivor Space: 1593835520 -
> INFO 00:29:46 G1 Young Generation GC in 233ms. G1 Eden Space: 18236833792 -> 0; G1 Old Gen: 9076474360 -> 9498544624; G1 Survivor Space: 1795162112 -
> INFO 00:32:35 G1 Young Generation GC in 226ms. G1 Eden Space: 16492003328 -> 0; G1 Old Gen: 10804527088 -> 11351712752; G1 Survivor Space: 1811939328
> INFO 00:37:09 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
> INFO 00:41:24 G1 Young Generation GC in 330ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 3426117112 -> 3961507312; G1 Survivor Space: 1459617792 -
> INFO 00:44:26 G1 Young Generation GC in 325ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 5268046320 -> 5788210120; G1 Survivor Space: 1560281088 -
> INFO 00:47:13 G1 Young Generation GC in 209ms. G1 Eden Space: 19243466752 -> 0; G1 Old Gen: 7147093960 -> 7138705360; G1 Survivor Space: 1358954496 -
> INFO 00:47:28 G1 Young Generation GC in 312ms. G1 Eden Space: 18605932544 -> 0; G1 Old Gen: 7138705360 -> 7710094280; G1 Survivor Space: 1996488704 -
> INFO 00:50:35 G1 Young Generation GC in 330ms. G1 Eden Space: 17934843904 -> 0; G1 Old Gen: 9118416840 -> 9688820680; G1 Survivor Space: 1929379840 -
> INFO 00:52:52 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
> INFO 00:56:29 G1 Young Generation GC in 251ms. G1 Eden Space: 14797504512 -> 0; G1 Old Gen: 13404995520 -> 13787049928; G1 Survivor Space: 1358954496
> INFO 00:58:14 G1 Young Generation GC in 662ms. G1 Eden Space: 369098752 -> 0; G1 Old Gen: 14730395584 -> 15984921520; G1 Survivor Space: 1342177280 -
> INFO 01:01:25 G1 Young Generation GC in 288ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 2618580968 -> 3127466464; G1 Survivor Space: 1560281088 -
> INFO 01:04:17 G1 Young Generation GC in 201ms. G1 Eden Space: 19327352832 -> 0; G1 Old Gen: 4521459688 -> 4529848288; G1 Survivor Space: 1275068416 -
> INFO 01:04:39 G1 Young Generation GC in 283ms. G1 Eden Space: 18723373056 -> 0; G1 Old Gen: 4529848288 -> 5024210408; G1 Survivor Space: 1879048192 -
> INFO 01:07:23 G1 Young Generation GC in 294ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 6308233704 -> 6875745256; G1 Survivor Space: 1862270976 -
> INFO 01:07:53 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
> INFO 01:09:31 Redistributing index summaries
> INFO 01:10:12 G1 Young Generation GC in 213ms. G1 Eden Space: 19243466752 -> 0; G1 Old Gen: 8262779360 -> 8254390760; G1 Survivor Space: 1358954496 -
> INFO 01:10:33 G1 Young Generation GC in 319ms. G1 Eden Space: 18639486976 -> 0; G1 Old Gen: 8254390760 -> 8821123552; G1 Survivor Space: 1962934272 -
> INFO 01:22:55 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
> INFO 01:24:25 G1 Young Generation GC in 227ms. G1 Eden Space: 19226689536 -> 0; G1 Old Gen: 4245307376 -> 4236918776; G1 Survivor Space: 1375731712 -
> INFO 01:24:48 G1 Young Generation GC in 219ms. G1 Eden Space: 15401484288 -> 0; G1 Old Gen: 4236918776 -> 5231503864; G1 Survivor Space: 2097152000 -
> INFO 01:42:01 G1 Young Generation GC in 212ms. G1 Eden Space: 16005464064 -> 0; G1 Old Gen: 4320133112 -> 5100782576; G1 Survivor Space: 1627389952 -
> INFO 01:48:28 G1 Young Generation GC in 209ms. G1 Eden Space: 14730395648 -> 0; G1 Old Gen: 8901994480 -> 9402393592; G1 Survivor Space: 1291845632 -
> INFO 02:04:18 G1 Young Generation GC in 241ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 7825270264 -> 8327173624; G1 Survivor Space: 1560281088 -
> INFO 02:08:09 Saved KeyCache (8634 items) in 94 ms
> INFO 02:09:31 Redistributing index summaries
> INFO 02:14:50 G1 Young Generation GC in 314ms. G1 Eden Space: 19075694592 -> 0; G1 Old Gen: 3505807352 -> 4140375528; G1 Survivor Space: 1526726656 -
> INFO 02:48:35 G1 Young Generation GC in 321ms. G1 Eden Space: 11576279040 -> 0; G1 Old Gen: 3865890800 -> 4482588664; G1 Survivor Space: 1543503872 -
> INFO 02:53:54 G1 Young Generation GC in 203ms. G1 Eden Space: 17901289472 -> 0; G1 Old Gen: 7927235056 -> 8552357368; G1 Survivor Space: 1593835520 -
> INFO 03:04:33 G1 Young Generation GC in 345ms. G1 Eden Space: 13505658880 -> 0; G1 Old Gen: 3765058032 -> 4443454952; G1 Survivor Space: 1442840576 -
> INFO 03:09:31 Redistributing index summaries
> INFO 03:22:56 G1 Young Generation GC in 250ms. G1 Eden Space: 18991808512 -> 0; G1 Old Gen: 4806672880 -> 5340134904; G1 Survivor Space: 1610612736 -
> INFO 03:35:13 G1 Young Generation GC in 210ms. G1 Eden Space: 14445182976 -> 0; G1 Survivor Space: 1073741824 -> 1744830464;
> INFO 03:38:42 G1 Young Generation GC in 214ms. G1 Eden Space: 18303942656 -> 0; G1 Old Gen: 4786540536 -> 5432116720; G1 Survivor Space: 1375731712 -
> INFO 03:52:11 G1 Young Generation GC in 207ms. G1 Eden Space: 117440512 -> 0; G1 Old Gen: 12054831088 -> 10154180600; G1 Survivor Space: 1593835520 -
> INFO 03:54:48 G1 Young Generation GC in 224ms. G1 Eden Space: 16156459008 -> 0; G1 Old Gen: 4303355896 -> 4801558520; G1 Survivor Space: 1560281088 -
> INFO 04:09:31 Redistributing index summaries
> INFO 04:16:09 G1 Young Generation GC in 216ms. G1 Eden Space: 19142803456 -> 0; G1 Old Gen: 5853311984 -> 6366188536; G1 Survivor Space: 1459617792 -
> INFO 04:19:13 G1 Young Generation GC in 229ms. G1 Eden Space: 19042140160 -> 0; G1 Old Gen: 7826571256 -> 8457920504; G1 Survivor Space: 1560281088 -
> INFO 04:27:41 G1 Young Generation GC in 311ms. G1 Eden Space: 13723762688 -> 0; G1 Old Gen: 14176747504 -> 14808230392; G1 Survivor Space: 1509949440
> INFO 04:29:56 G1 Young Generation GC in 274ms. G1 Eden Space: 18740150272 -> 0; G1 Old Gen: 3376452080 -> 3818638328; G1 Survivor Space: 1459617792 -
> INFO 04:33:40 Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB
> INFO 04:49:07 G1 Young Generation GC in 319ms. G1 Eden Space: 16005464064 -> 0; G1 Old Gen: 4435714040 -> 5023449072; G1 Survivor Space: 1442840576 -
> INFO 04:54:06 G1 Young Generation GC in 208ms. G1 Eden Space: 18706595840 -> 0; G1 Old Gen: 7843348472 -> 8466889208; G1 Survivor Space: 1476395008 -
> INFO 05:09:31 Redistributing index summaries
>
>
> I found INFO Maximum memory usage reached (2.000GiB), cannot allocate chunk of 1.000MiB, does it mean the memory is not enough ?
>
>
> I want to know why adding a new node spent so much time and what need I do to complete a new node joining the cluster ?
>
>
> I need some help and really need some suggestions.
>
>
>
> Thanks a lot!
>
>
>
>
>
>
>
> The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.