You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Zhengqingzheng <zh...@huawei.com> on 2016/04/22 13:02:12 UTC
Ignite cache data size problem.
Hi there,
I am trying to load a table with 47Million records, in which the data size is less than 3gb. However, When I load into the memory ( two vm with 48+32 = 80gb), it is still crushed due to not enough memory space? This problem occurred when I instantiated 6 + 4 nodes.
Why the cache model need so much space ( 3g vs 80g)?
Any idea to explain this issue?
Kind regards,
Kevin
re: Ignite cache data size problem.
Posted by Zhengqingzheng <zh...@huawei.com>.
Hi Vladimir,
My table is very simple, it contains the following information
OrgId (varchar) , oId(varchar), fnum(int), gId(number), msg(varchar), num(number), date(date)
the gId is the primary Index.
--------------------------------------------------------------------------------------------------
And the java class is defined as:
package com.huawei.soa.ignite.test;
import java.io.Serializable;
import java.math.BigDecimal;
import java.util.Date;
import org.apache.ignite.cache.query.annotations.QuerySqlField;
public class UniqueField implements Serializable
{
public String getOrgId()
{
return orgId;
}
public void setOrgId(String orgId)
{
this.orgId = orgId;
}
public String getOId()
{
return oId;
}
public void setOId(String oId)
{
this.oId = oId;
}
public String getGid()
{
return gId;
}
public void setGuid(String gId)
{
this.gId = gId;
}
public int getFNum()
{
return fNum;
}
public void setFNum(int fNum)
{
this.fNum = fNum;
}
public String getMsg()
{
return msg;
}
public void setMsg(String msg)
{
this.msg = msg;
}
public BigDecimal getNum()
{
return num;
}
public void setNum(BigDecimal num)
{
this.num = num;
}
public Date getDate()
{
return date;
}
public void setDate(Date date)
{
this.date = date;
}
@QuerySqlField
private String orgId;
@QuerySqlField(index=true)
private String oId;
@QuerySqlField(index=true)
private String gId;
@QuerySqlField
private int fNum;
@QuerySqlField
private String msg;
@QuerySqlField
private BigDecimal num;
@QuerySqlField
private Date date;
public UniqueField(){};
public UniqueField(
String orgId,
String oId,
String gId,
int fNum,
String msg,
BigDecimal num,
Date date
){
this.orgId=orgId;
this.oId=oId;
this.gId = gId;
this.fNum = fNum;
this.msg = msg;
this.num = num;
this.date = date;
}
}
--------------------------------------------------------------------------------------------------
My configuration file on the server side is listed as follows:
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="dataSource" class="oracle.jdbc.pool.OracleDataSource">
<!-- real data is not listed -->
<property name="URL" value="jdbc:oracle:thin:@xxx.xxx.xxx.xxx:xxx" />
<property name="user" value="xxx"/>
<property name="password" value="xxx" />
</bean>
<!--
Alter configuration below as needed.
-->
<bean id="grid1.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="peerClassLoadingEnabled" value="true" />
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="cacheStoreFactory">
<!-- CacheJdbcPojoStoreExampleFactory is extend from CacheJdbcPojoStoreFactory -->
<bean class="com.huawei.soa.ignite.util.CacheJdbcPojoStoreExampleFactory">
<property name="dataSourceBean" value="dataSource" />
<property name="dialect" >
<bean class="org.apache.ignite.cache.store.jdbc.dialect.OracleDialect"/>
</property>
</bean>
</property>
<!-- defintion from ignite document -->
<property name="memoryMode" value="OFFHEAP_TIERED" />
<!-- 10g offheap allocated-->
<property name="offHeapMaxMemory" value="#{10*1024L*1024L*1024L}" />
<property name="readThrough" value="true" />
<property name="writeThrough" value="true" />
<property name="startSize" value="#{200*1024*1024}" />
<property name="cacheMode" value="PARTITIONED" />
<property name="backups" value="0" />
</bean>
</list>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder" >
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value> xxx.xxx.xxx.xxx:47500..47509</value>
<value> xxx.xxx.xxx.xxx:47500..47509</value>
<value> xxx.xxx.xxx.xxx:47500..47509 </value>
<value> xxx.xxx.xxx.xxx:47500..47509 </value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Kind regards,
Kevin
发件人: Vladimir Ozerov [mailto:vozerov@gridgain.com]
发送时间: 2016年4月22日 20:13
收件人: user@ignite.apache.org
主题: Re: Ignite cache data size problem.
Hi,
It looks like you have relatively small entries (somewhere about 60-70 bytes per key-value pair). Ignite also has some intrinsic overhead, which could be more than actual data in this case. However, I surely would not expect that it will not fit into 80GB.
Could you please share your key and value model classes and your XML configuration to investigate it further?
Vladimir.
On Fri, Apr 22, 2016 at 2:02 PM, Zhengqingzheng <zh...@huawei.com>> wrote:
Hi there,
I am trying to load a table with 47Million records, in which the data size is less than 3gb. However, When I load into the memory ( two vm with 48+32 = 80gb), it is still crushed due to not enough memory space? This problem occurred when I instantiated 6 + 4 nodes.
Why the cache model need so much space ( 3g vs 80g)?
Any idea to explain this issue?
Kind regards,
Kevin
Re: Ignite cache data size problem.
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Kevin,
Yes, log files are created under this directory by default.
Vladimir.
On Tue, Apr 26, 2016 at 3:18 PM, Zhengqingzheng <zh...@huawei.com>
wrote:
> Hi Vladimir,
>
> No problem.
>
> I will re-run the loading process and give you the log file.
>
> To be clear, when you say log file, do you mean files located at
> work/log/*?
>
> I use IgntieCache.loadCache(null, “select * from table”) to load all the
> data.
>
>
>
> Best regards,
>
> Kevin
>
>
>
> *发件人:* Vladimir Ozerov [mailto:vozerov@gridgain.com]
> *发送时间:* 2016年4月26日 20:15
> *收件人:* user@ignite.apache.org
> *主题:* Re: Ignite cache data size problem.
>
>
>
> Hi Kevin,
>
>
>
> Could you please re-run your case and attach Ignite logs and GC logs from
> all participating servers? I would expect that this is either a kind of
> out-of-memory problem, or network saturation. Also please explain how
> exactly do you load data to Ignite? Do you use *DataStreamer*, or may be
> *IgniteCache.loadlCache*, or *IgniteCache.put*?
>
>
>
> Vladimir.
>
>
>
> On Tue, Apr 26, 2016 at 3:01 PM, Zhengqingzheng <zh...@huawei.com>
> wrote:
>
> Hi Vladimir,
>
> Thank you for your help.
>
> I tried to load 1Million records, and calculate each object’s size
> (including key and value object, where key is string type), summarized
> together and found the total memory consumption is 130Mb.
>
>
>
> Because ./ignitevisor.sh command only shows number of records, and no
> data allocation information, I don’t know how much memory has been consumed
> for each type of cache.
>
>
>
> My result is as follows:
>
> Log:
>
> Found big object: [Ljava.util.HashMap$Node;@24833164 size:
> 30.888206481933594Mb
>
> Found big object: java.util.HashMap@15819383 size: 30.88824462890625Mb
>
> Found big object: java.util.HashSet@10236420 size: 30.888259887695312Mb
>
> key size: 32388688 human readable data: 30.888259887695312Mb
>
> Found big object: [Lorg.jsr166.ConcurrentHashMap8$Node;@29556439 size:
> 129.99818420410156Mb
>
> Found big object: org.jsr166.ConcurrentHashMap8@19238297 size:
> *129.99822235107422Mb*
>
> value size: 136313016 human readable data: 129.99822235107422Mb
>
> *the whole number of record is 47Million, so the data size inside the
> cache should be 130*47=**6110Mb(around 6Gb).*
>
>
>
> However, when I try to load the whole data into the cache, I still get
> exceptions:
>
> The exception information is listed as follows:
>
> 1. ---------------------------------- exception info from client
> ----------------------------------------
>
> Before exception occurred, I have 10 nodes on two servers, server1 (48G
> ram) has 6nodes, each node is assigned with 7gb jvm heap; server2 (32gb
> ram) has 4 nodes with the same jvm settings as previous one.
>
> After exception, client stopped, and 8 node left (server1’s node remain,
> no exception occurred on this server), server2 ( 2 nodes remain, two nodes
> droped)
>
> The total number of records loaded is 37Million.
>
>
>
> [19:01:47] Topology snapshot [ver=77, servers=9, clients=1, CPUs=20,
> heap=64.0GB]
>
> [19:01:47,463][SEVERE][pub-#46%null%][GridTaskWorker] Failed to obtain
> remote job result policy for result from ComputeTask.result(..) method
> (will fail the whole task): GridJobResultImpl [job=C2 [],
> sib=GridJobSiblingImpl
> [sesId=0dbc0f15451-880a2dd1-bc95-4084-a705-4effcec5d2cd,
> jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924,
> nodeId=832bed3e-dc5d-4743-9853-127e3b516924, isJobDone=false],
> jobCtx=GridJobContextImpl
> [jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924, timeoutObj=null,
> attrs={}], node=TcpDiscoveryNode [id=832bed3e-dc5d-4743-9853-127e3b516924,
> addrs=[0:0:0:0:0:0:0:1%lo, 10.120.70.122, 127.0.0.1],
> sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /0:0:0:0:0:0:0:1%lo:47500, /
> 10.120.70.122:47500, /127.0.0.1:47500], discPort=47500, order=7,
> intOrder=7, lastExchangeTime=1461663619161, loc=false,
> ver=1.5.0#20151229-sha1:f1f8cda2, isClient=false], ex=class
> o.a.i.cluster.ClusterTopologyException: Node has left grid:
> 832bed3e-dc5d-4743-9853-127e3b516924, hasRes=true, isCancelled=false,
> isOccupied=true]
>
> class org.apache.ignite.cluster.ClusterTopologyException: Node has left
> grid: 832bed3e-dc5d-4743-9853-127e3b516924
>
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1315)
>
> at
> org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1.run(GridTaskProcessor.java:1246)
>
> at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6453)
>
> at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:788)
>
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> [19:02:16] Ignite node stopped OK [uptime=01:21:52:963]
>
>
>
> 2 ------------------------------ also, I use the recommended settings for
> the jvm and get the following log information
> ------------------------------------------------
>
> Java HotSpot(TM) Client VM (25.66-b18) for windows-x86 JRE (1.8.0_66-b18),
> built on Nov 9 2015 10:58:29 by "java_re" with MS VC++ 10.0 (VS2010)
>
> Memory: 4k page, physical 6291064k(1912832k free), swap 12580288k(6996288k
> free)
>
> CommandLine flags: -XX:GCLogFileSize=104857600
> -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824
> -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation
> -XX:-UseLargePagesIndividualAllocation
>
> 2016-04-26T17:40:15.609+0800: 4.839: [Full GC (Metadata GC Threshold)
> 2016-04-26T17:40:15.610+0800: 4.839: [Tenured: 0K->10330K(699072K),
> 0.0656011 secs] 162451K->10330K(1013632K), [Metaspace:
> 12085K->12085K(12672K)], 0.0659450 secs] [Times: user=0.06 sys=0.00,
> real=0.07 secs]
>
> 2016-04-26T17:40:24.837+0800: 14.044: [Full GC (Metadata GC Threshold)
> 2016-04-26T17:40:24.837+0800: 14.044: [Tenured: 10330K->32179K(699072K),
> 0.1810716 secs] 282679K->32179K(1013632K), [Metaspace:
> 20221K->20221K(20864K)], 0.1812825 secs] [Times: user=0.17 sys=0.00,
> real=0.18 secs]
>
> 2016-04-26T17:40:53.699+0800: 42.937: [GC (Allocation Failure)
> 2016-04-26T17:40:53.699+0800: 42.937: [DefNew: 279616K->8837K(314560K),
> 0.0242718 secs] 311795K->41016K(1013632K), 0.0244783 secs] [Times:
> user=0.03 sys=0.00, real=0.03 secs]
>
> 2016-04-26T17:41:26.411+0800: 75.516: [GC (Allocation Failure)
> 2016-04-26T17:41:26.412+0800: 75.516: [DefNew: 288453K->2685K(314560K),
> 0.0116993 secs] 320632K->34864K(1013632K), 0.0119035 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:42:01.895+0800: 110.834: [GC (Allocation Failure)
> 2016-04-26T17:42:01.895+0800: 110.834: [DefNew: 282301K->2689K(314560K),
> 0.0111799 secs] 314480K->34868K(1013632K), 0.0113629 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:42:37.354+0800: 145.818: [GC (Allocation Failure)
> 2016-04-26T17:42:37.354+0800: 145.818: [DefNew: 282305K->2695K(314560K),
> 0.0102415 secs] 314484K->34874K(1013632K), 0.0104181 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:43:13.549+0800: 181.641: [GC (Allocation Failure)
> 2016-04-26T17:43:13.549+0800: 181.641: [DefNew: 282311K->2893K(314560K),
> 0.0134864 secs] 314490K->35072K(1013632K), 0.0136970 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:43:50.453+0800: 218.129: [GC (Allocation Failure)
> 2016-04-26T17:43:50.453+0800: 218.129: [DefNew: 282509K->2708K(314560K),
> 0.0160308 secs] 314688K->34887K(1013632K), 0.0162459 secs] [Times:
> user=0.03 sys=0.00, real=0.03 secs]
>
> 2016-04-26T17:44:27.317+0800: 254.604: [GC (Allocation Failure)
> 2016-04-26T17:44:27.317+0800: 254.605: [DefNew: 282324K->2715K(314560K),
> 0.0128055 secs] 314503K->34894K(1013632K), 0.0130086 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:45:03.944+0800: 291.040: [GC (Allocation Failure)
> 2016-04-26T17:45:03.944+0800: 291.040: [DefNew: 282331K->2724K(314560K),
> 0.0109078 secs] 314510K->34903K(1013632K), 0.0110838 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:45:41.069+0800: 327.816: [GC (Allocation Failure)
> 2016-04-26T17:45:41.069+0800: 327.816: [DefNew: 282340K->2732K(314560K),
> 0.0128223 secs] 314519K->34912K(1013632K), 0.0130151 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:46:18.108+0800: 364.538: [GC (Allocation Failure)
> 2016-04-26T17:46:18.108+0800: 364.538: [DefNew: 282348K->2738K(314560K),
> 0.0123320 secs] 314528K->34917K(1013632K), 0.0125273 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:46:54.492+0800: 400.975: [GC (Allocation Failure)
> 2016-04-26T17:46:54.492+0800: 400.975: [DefNew: 282354K->2747K(314560K),
> 0.0126284 secs] 314533K->34926K(1013632K), 0.0128229 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:47:36.805+0800: 443.099: [GC (Allocation Failure)
> 2016-04-26T17:47:36.805+0800: 443.099: [DefNew: 282363K->2755K(314560K),
> 0.0137048 secs] 314542K->34934K(1013632K), 0.0139138 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:48:13.500+0800: 480.054: [GC (Allocation Failure)
> 2016-04-26T17:48:13.500+0800: 480.054: [DefNew: 282371K->2764K(314560K),
> 0.0134777 secs] 314550K->34943K(1013632K), 0.0137571 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:48:50.568+0800: 517.242: [GC (Allocation Failure)
> 2016-04-26T17:48:50.568+0800: 517.242: [DefNew: 282380K->2769K(314560K),
> 0.0121474 secs] 314559K->34948K(1013632K), 0.0125373 secs] [Times:
> user=0.00 sys=0.02, real=0.02 secs]
>
> 2016-04-26T17:49:28.506+0800: 555.103: [GC (Allocation Failure)
> 2016-04-26T17:49:28.507+0800: 555.103: [DefNew: 282385K->2777K(314560K),
> 0.0133794 secs] 314564K->34957K(1013632K), 0.0135732 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:50:05.869+0800: 592.355: [GC (Allocation Failure)
> 2016-04-26T17:50:05.869+0800: 592.355: [DefNew: 282393K->1713K(314560K),
> 0.0129486 secs] 314573K->34965K(1013632K), 0.0131366 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:50:43.336+0800: 629.663: [GC (Allocation Failure)
> 2016-04-26T17:50:43.336+0800: 629.663: [DefNew: 281329K->1688K(314560K),
> 0.0107519 secs] 314581K->34971K(1013632K), 0.0109511 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:51:20.603+0800: 666.805: [GC (Allocation Failure)
> 2016-04-26T17:51:20.604+0800: 666.806: [DefNew: 281304K->2043K(314560K),
> 0.0125329 secs] 314587K->35333K(1013632K), 0.0127661 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:51:57.981+0800: 704.195: [GC (Allocation Failure)
> 2016-04-26T17:51:57.981+0800: 704.195: [DefNew: 281659K->1688K(314560K),
> 0.0083589 secs] 314949K->34985K(1013632K), 0.0085578 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:52:35.234+0800: 741.413: [GC (Allocation Failure)
> 2016-04-26T17:52:35.234+0800: 741.413: [DefNew: 281304K->1686K(314560K),
> 0.0080466 secs] 314601K->34990K(1013632K), 0.0082145 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:53:11.734+0800: 778.046: [GC (Allocation Failure)
> 2016-04-26T17:53:11.734+0800: 778.046: [DefNew: 281302K->1689K(314560K),
> 0.0106016 secs] 314606K->35000K(1013632K), 0.0108028 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:53:48.679+0800: 814.992: [GC (Allocation Failure)
> 2016-04-26T17:53:48.679+0800: 814.992: [DefNew: 281305K->1689K(314560K),
> 0.0085276 secs] 314616K->35007K(1013632K), 0.0086721 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:54:24.734+0800: 851.078: [GC (Allocation Failure)
> 2016-04-26T17:54:24.735+0800: 851.078: [DefNew: 281305K->1685K(314560K),
> 0.0130525 secs] 314623K->35012K(1013632K), 0.0132933 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:55:01.614+0800: 888.090: [GC (Allocation Failure)
> 2016-04-26T17:55:01.614+0800: 888.090: [DefNew: 281301K->1686K(314560K),
> 0.0084343 secs] 314628K->35021K(1013632K), 0.0086352 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:55:38.728+0800: 925.129: [GC (Allocation Failure)
> 2016-04-26T17:55:38.728+0800: 925.129: [DefNew: 281302K->1684K(314560K),
> 0.0081086 secs] 314637K->35026K(1013632K), 0.0082835 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:56:15.257+0800: 961.859: [GC (Allocation Failure)
> 2016-04-26T17:56:15.257+0800: 961.859: [DefNew: 281300K->1683K(314560K),
> 0.0097599 secs] 314642K->35032K(1013632K), 0.0099144 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:56:51.417+0800: 998.123: [GC (Allocation Failure)
> 2016-04-26T17:56:51.417+0800: 998.123: [DefNew: 281299K->1681K(314560K),
> 0.0095046 secs] 314648K->35038K(1013632K), 0.0096970 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:57:27.838+0800: 1034.605: [GC (Allocation Failure)
> 2016-04-26T17:57:27.838+0800: 1034.605: [DefNew: 281297K->1681K(314560K),
> 0.0104843 secs] 314654K->35045K(1013632K), 0.0106871 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:58:04.008+0800: 1070.815: [GC (Allocation Failure)
> 2016-04-26T17:58:04.008+0800: 1070.815: [DefNew: 281297K->1680K(314560K),
> 0.0087810 secs] 314661K->35051K(1013632K), 0.0089486 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:58:40.916+0800: 1107.629: [GC (Allocation Failure)
> 2016-04-26T17:58:40.916+0800: 1107.629: [DefNew: 281296K->1679K(314560K),
> 0.0123532 secs] 314667K->35058K(1013632K), 0.0125745 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:59:17.297+0800: 1143.843: [GC (Allocation Failure)
> 2016-04-26T17:59:17.297+0800: 1143.843: [DefNew: 281295K->1682K(314560K),
> 0.0099356 secs] 314674K->35069K(1013632K), 0.0101206 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:59:54.265+0800: 1180.679: [GC (Allocation Failure)
> 2016-04-26T17:59:54.265+0800: 1180.680: [DefNew: 281298K->1678K(314560K),
> 0.0089453 secs] 314685K->35071K(1013632K), 0.0091503 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:00:31.910+0800: 1218.043: [GC (Allocation Failure)
> 2016-04-26T18:00:31.910+0800: 1218.043: [DefNew: 281294K->1677K(314560K),
> 0.0082765 secs] 314687K->35078K(1013632K), 0.0084452 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:01:07.853+0800: 1254.048: [GC (Allocation Failure)
> 2016-04-26T18:01:07.854+0800: 1254.048: [DefNew: 281293K->1678K(314560K),
> 0.0089447 secs] 314694K->35085K(1013632K), 0.0091405 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:01:44.497+0800: 1290.474: [GC (Allocation Failure)
> 2016-04-26T18:01:44.497+0800: 1290.474: [DefNew: 281294K->1678K(314560K),
> 0.0076934 secs] 314701K->35091K(1013632K), 0.0078666 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:02:27.863+0800: 1333.673: [GC (Allocation Failure)
> 2016-04-26T18:02:27.863+0800: 1333.674: [DefNew: 281294K->1674K(314560K),
> 0.0084192 secs] 314707K->35098K(1013632K), 0.0085597 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:03:04.588+0800: 1370.424: [GC (Allocation Failure)
> 2016-04-26T18:03:04.589+0800: 1370.424: [DefNew: 281290K->1677K(314560K),
> 0.0086385 secs] 314714K->35107K(1013632K), 0.0087911 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:03:41.646+0800: 1407.396: [GC (Allocation Failure)
> 2016-04-26T18:03:41.646+0800: 1407.396: [DefNew: 281293K->1674K(314560K),
> 0.0079049 secs] 314723K->35111K(1013632K), 0.0080862 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:04:18.618+0800: 1444.458: [GC (Allocation Failure)
> 2016-04-26T18:04:18.618+0800: 1444.458: [DefNew: 281290K->1674K(314560K),
> 0.0089752 secs] 314727K->35118K(1013632K), 0.0091841 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:04:55.359+0800: 1481.281: [GC (Allocation Failure)
> 2016-04-26T18:04:55.359+0800: 1481.281: [DefNew: 281290K->1674K(314560K),
> 0.0091816 secs] 314734K->35124K(1013632K), 0.0093663 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:05:31.696+0800: 1517.690: [GC (Allocation Failure)
> 2016-04-26T18:05:31.696+0800: 1517.690: [DefNew: 281290K->1680K(314560K),
> 0.0097599 secs] 314740K->35136K(1013632K), 0.0099613 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:06:08.293+0800: 1554.357: [GC (Allocation Failure)
> 2016-04-26T18:06:08.293+0800: 1554.357: [DefNew: 281296K->1681K(314560K),
> 0.0102206 secs] 314752K->35144K(1013632K), 0.0104262 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:06:44.886+0800: 1590.882: [GC (Allocation Failure)
> 2016-04-26T18:06:44.886+0800: 1590.882: [DefNew: 281297K->1680K(314560K),
> 0.0090525 secs] 314760K->35150K(1013632K), 0.0092456 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:07:20.852+0800: 1626.984: [GC (Allocation Failure)
> 2016-04-26T18:07:20.852+0800: 1626.984: [DefNew: 281296K->1682K(314560K),
> 0.0095817 secs] 314766K->35158K(1013632K), 0.0097445 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:07:57.290+0800: 1663.543: [GC (Allocation Failure)
> 2016-04-26T18:07:57.290+0800: 1663.543: [DefNew: 281298K->1682K(314560K),
> 0.0076180 secs] 314774K->35164K(1013632K), 0.0077856 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:08:34.599+0800: 1700.805: [GC (Allocation Failure)
> 2016-04-26T18:08:34.599+0800: 1700.805: [DefNew: 281298K->1680K(314560K),
> 0.0136244 secs] 314780K->35170K(1013632K), 0.0138406 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:09:10.999+0800: 1737.438: [GC (Allocation Failure)
> 2016-04-26T18:09:10.999+0800: 1737.438: [DefNew: 281296K->1681K(314560K),
> 0.0086762 secs] 314786K->35177K(1013632K), 0.0088701 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:09:47.695+0800: 1774.086: [GC (Allocation Failure)
> 2016-04-26T18:09:47.695+0800: 1774.086: [DefNew: 281297K->1679K(314560K),
> 0.0079063 secs] 314793K->35182K(1013632K), 0.0080798 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:10:24.630+0800: 1811.127: [GC (Allocation Failure)
> 2016-04-26T18:10:24.630+0800: 1811.127: [DefNew: 281295K->1680K(314560K),
> 0.0100454 secs] 314798K->35189K(1013632K), 0.0101801 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:11:00.972+0800: 1847.581: [GC (Allocation Failure)
> 2016-04-26T18:11:00.972+0800: 1847.581: [DefNew: 281296K->1679K(314560K),
> 0.0083639 secs] 314805K->35195K(1013632K), 0.0085452 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:11:37.280+0800: 1883.810: [GC (Allocation Failure)
> 2016-04-26T18:11:37.280+0800: 1883.810: [DefNew: 281295K->1679K(314560K),
> 0.0084648 secs] 314811K->35202K(1013632K), 0.0086475 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:12:13.694+0800: 1920.386: [GC (Allocation Failure)
> 2016-04-26T18:12:13.694+0800: 1920.386: [DefNew: 281295K->1677K(314560K),
> 0.0080820 secs] 314818K->35207K(1013632K), 0.0082119 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:12:49.894+0800: 1956.802: [GC (Allocation Failure)
> 2016-04-26T18:12:49.894+0800: 1956.803: [DefNew: 281293K->1677K(314560K),
> 0.0146823 secs] 314823K->35213K(1013632K), 0.0149620 secs] [Times:
> user=0.00 sys=0.02, real=0.01 secs]
>
> 2016-04-26T18:13:26.430+0800: 1993.317: [GC (Allocation Failure)
> 2016-04-26T18:13:26.430+0800: 1993.317: [DefNew: 281293K->1677K(314560K),
> 0.0088413 secs] 314829K->35220K(1013632K), 0.0090009 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:14:03.233+0800: 2030.321: [GC (Allocation Failure)
> 2016-04-26T18:14:03.233+0800: 2030.321: [DefNew: 281293K->1676K(314560K),
> 0.0078476 secs] 314836K->35225K(1013632K), 0.0079887 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:14:40.063+0800: 2067.102: [GC (Allocation Failure)
> 2016-04-26T18:14:40.063+0800: 2067.102: [DefNew: 281292K->1677K(314560K),
> 0.0092786 secs] 314841K->35232K(1013632K), 0.0094601 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:15:16.165+0800: 2103.376: [GC (Allocation Failure)
> 2016-04-26T18:15:16.165+0800: 2103.376: [DefNew: 281293K->2106K(314560K),
> 0.0135981 secs] 314848K->35675K(1013632K), 0.0138093 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:15:52.922+0800: 2139.884: [GC (Allocation Failure)
> 2016-04-26T18:15:52.922+0800: 2139.884: [DefNew: 281722K->1669K(314560K),
> 0.0123189 secs] 315291K->35244K(1013632K), 0.0125281 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:16:30.496+0800: 2177.184: [GC (Allocation Failure)
> 2016-04-26T18:16:30.496+0800: 2177.184: [DefNew: 281285K->1669K(314560K),
> 0.0089190 secs] 314860K->35251K(1013632K), 0.0090788 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:17:06.767+0800: 2213.365: [GC (Allocation Failure)
> 2016-04-26T18:17:06.767+0800: 2213.365: [DefNew: 281285K->1826K(314560K),
> 0.0079870 secs] 314867K->35415K(1013632K), 0.0081401 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:17:43.991+0800: 2250.422: [GC (Allocation Failure)
> 2016-04-26T18:17:43.991+0800: 2250.422: [DefNew: 281442K->1669K(314560K),
> 0.0115439 secs] 315031K->35264K(1013632K), 0.0117428 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:18:20.528+0800: 2286.733: [GC (Allocation Failure)
> 2016-04-26T18:18:20.528+0800: 2286.733: [DefNew: 281285K->1668K(314560K),
> 0.0082332 secs] 314880K->35270K(1013632K), 0.0084053 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:18:57.215+0800: 2323.271: [GC (Allocation Failure)
> 2016-04-26T18:18:57.215+0800: 2323.271: [DefNew: 281284K->1671K(314560K),
> 0.0077934 secs] 314886K->35277K(1013632K), 0.0079697 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:19:34.151+0800: 2359.977: [GC (Allocation Failure)
> 2016-04-26T18:19:34.151+0800: 2359.977: [DefNew: 281287K->1669K(314560K),
> 0.0078809 secs] 314893K->35283K(1013632K), 0.0080410 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:20:11.106+0800: 2396.706: [GC (Allocation Failure)
> 2016-04-26T18:20:11.106+0800: 2396.706: [DefNew: 281285K->1673K(314560K),
> 0.0092928 secs] 314899K->35293K(1013632K), 0.0094864 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:20:47.129+0800: 2432.801: [GC (Allocation Failure)
> 2016-04-26T18:20:47.129+0800: 2432.801: [DefNew: 281289K->1672K(314560K),
> 0.0089699 secs] 314909K->35298K(1013632K), 0.0091397 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:21:23.028+0800: 2468.633: [GC (Allocation Failure)
> 2016-04-26T18:21:23.028+0800: 2468.633: [DefNew: 281288K->1671K(314560K),
> 0.0124404 secs] 314914K->35303K(1013632K), 0.0128810 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:21:59.745+0800: 2505.506: [GC (Allocation Failure)
> 2016-04-26T18:21:59.745+0800: 2505.506: [DefNew: 281287K->1671K(314560K),
> 0.0080019 secs] 314919K->35309K(1013632K), 0.0082167 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:22:36.639+0800: 2542.570: [GC (Allocation Failure)
> 2016-04-26T18:22:36.639+0800: 2542.570: [DefNew: 281287K->1670K(314560K),
> 0.0082147 secs] 314925K->35315K(1013632K), 0.0083938 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:23:12.390+0800: 2578.421: [GC (Allocation Failure)
> 2016-04-26T18:23:12.390+0800: 2578.421: [DefNew: 281286K->2207K(314560K),
> 0.0114121 secs] 314931K->35857K(1013632K), 0.0116115 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:23:48.693+0800: 2615.071: [GC (Allocation Failure)
> 2016-04-26T18:23:48.693+0800: 2615.071: [DefNew: 281823K->1798K(314560K),
> 0.0104047 secs] 315473K->35455K(1013632K), 0.0105935 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:24:25.698+0800: 2652.227: [GC (Allocation Failure)
> 2016-04-26T18:24:25.698+0800: 2652.228: [DefNew: 281414K->1795K(314560K),
> 0.0102968 secs] 315071K->35459K(1013632K), 0.0104854 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:25:01.801+0800: 2688.457: [GC (Allocation Failure)
> 2016-04-26T18:25:01.801+0800: 2688.458: [DefNew: 281411K->2126K(314560K),
> 0.0090964 secs] 315075K->35795K(1013632K), 0.0093529 secs] [Times:
> user=0.00 sys=0.02, real=0.01 secs]
>
> 2016-04-26T18:25:39.055+0800: 2725.732: [GC (Allocation Failure)
> 2016-04-26T18:25:39.055+0800: 2725.732: [DefNew: 281742K->1799K(314560K),
> 0.0085785 secs] 315411K->35474K(1013632K), 0.0087168 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:26:27.232+0800: 2773.853: [GC (Allocation Failure)
> 2016-04-26T18:26:27.232+0800: 2773.853: [DefNew: 281415K->1800K(314560K),
> 0.0091224 secs] 315090K->35482K(1013632K), 0.0094059 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:27:05.583+0800: 2812.453: [GC (Allocation Failure)
> 2016-04-26T18:27:05.584+0800: 2812.453: [DefNew: 281416K->1801K(314560K),
> 0.0083206 secs] 315098K->35489K(1013632K), 0.0084969 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:27:43.412+0800: 2850.382: [GC (Allocation Failure)
> 2016-04-26T18:27:43.412+0800: 2850.382: [DefNew: 281417K->1801K(314560K),
> 0.0095576 secs] 315105K->35495K(1013632K), 0.0097616 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:28:20.067+0800: 2887.191: [GC (Allocation Failure)
> 2016-04-26T18:28:20.067+0800: 2887.191: [DefNew: 281417K->1799K(314560K),
> 0.0121244 secs] 315111K->35501K(1013632K), 0.0123463 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:28:55.431+0800: 2923.064: [GC (Allocation Failure)
> 2016-04-26T18:28:55.431+0800: 2923.064: [DefNew: 281415K->1798K(314560K),
> 0.0088000 secs] 315117K->35506K(1013632K), 0.0089771 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:29:30.761+0800: 2958.335: [GC (Allocation Failure)
> 2016-04-26T18:29:30.761+0800: 2958.335: [DefNew: 281414K->1798K(314560K),
> 0.0113492 secs] 315122K->35512K(1013632K), 0.0115526 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:30:05.795+0800: 2993.528: [GC (Allocation Failure)
> 2016-04-26T18:30:05.795+0800: 2993.528: [DefNew: 281414K->1797K(314560K),
> 0.0112305 secs] 315128K->35518K(1013632K), 0.0114227 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:30:41.648+0800: 3029.322: [GC (Allocation Failure)
> 2016-04-26T18:30:41.648+0800: 3029.322: [DefNew: 281413K->1797K(314560K),
> 0.0174930 secs] 315134K->35524K(1013632K), 0.0177282 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:31:34.467+0800: 3082.237: [GC (Allocation Failure)
> 2016-04-26T18:31:34.467+0800: 3082.237: [DefNew: 281413K->1800K(314560K),
> 0.0095434 secs] 315140K->35534K(1013632K), 0.0097186 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:32:14.021+0800: 3121.810: [GC (Allocation Failure)
> 2016-04-26T18:32:14.021+0800: 3121.810: [DefNew: 281416K->1800K(314560K),
> 0.0096049 secs] 315150K->35540K(1013632K), 0.0097856 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:33:07.874+0800: 3175.537: [GC (Allocation Failure)
> 2016-04-26T18:33:07.874+0800: 3175.537: [DefNew: 281416K->1697K(314560K),
> 0.0135850 secs] 315156K->35550K(1013632K), 0.0138194 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:33:47.621+0800: 3215.029: [GC (Allocation Failure)
> 2016-04-26T18:33:47.621+0800: 3215.029: [DefNew: 281313K->1694K(314560K),
> 0.0080256 secs] 315166K->35566K(1013632K), 0.0081854 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:34:24.160+0800: 3251.370: [GC (Allocation Failure)
> 2016-04-26T18:34:24.160+0800: 3251.371: [DefNew: 281310K->1691K(314560K),
> 0.0086257 secs] 315182K->35570K(1013632K), 0.0087810 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:34:59.507+0800: 3286.713: [GC (Allocation Failure)
> 2016-04-26T18:34:59.507+0800: 3286.713: [DefNew: 281307K->1690K(314560K),
> 0.0082689 secs] 315186K->35576K(1013632K), 0.0084583 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:35:34.974+0800: 3322.008: [GC (Allocation Failure)
> 2016-04-26T18:35:34.974+0800: 3322.008: [DefNew: 281306K->1690K(314560K),
> 0.0134400 secs] 315192K->35582K(1013632K), 0.0136509 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:36:11.007+0800: 3357.907: [GC (Allocation Failure)
> 2016-04-26T18:36:11.007+0800: 3357.907: [DefNew: 281306K->1689K(314560K),
> 0.0118568 secs] 315198K->35590K(1013632K), 0.0120744 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:36:48.227+0800: 3394.956: [GC (Allocation Failure)
> 2016-04-26T18:36:48.227+0800: 3394.956: [DefNew: 281305K->1688K(314560K),
> 0.0083156 secs] 315206K->35596K(1013632K), 0.0084603 secs] [Times:
> user=0.03 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:37:24.551+0800: 3430.991: [GC (Allocation Failure)
> 2016-04-26T18:37:24.551+0800: 3430.991: [DefNew: 281304K->1688K(314560K),
> 0.0091559 secs] 315212K->35602K(1013632K), 0.0093369 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:38:01.307+0800: 3467.736: [GC (Allocation Failure)
> 2016-04-26T18:38:01.307+0800: 3467.736: [DefNew: 281304K->1685K(314560K),
> 0.0085799 secs] 315218K->35609K(1013632K), 0.0087592 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:38:37.590+0800: 3503.935: [GC (Allocation Failure)
> 2016-04-26T18:38:37.590+0800: 3503.935: [DefNew: 281301K->1685K(314560K),
> 0.0103468 secs] 315225K->35616K(1013632K), 0.0105709 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:39:14.026+0800: 3540.397: [GC (Allocation Failure)
> 2016-04-26T18:39:14.026+0800: 3540.397: [DefNew: 281301K->1685K(314560K),
> 0.0119247 secs] 315232K->35622K(1013632K), 0.0121515 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:39:50.866+0800: 3577.377: [GC (Allocation Failure)
> 2016-04-26T18:39:50.866+0800: 3577.377: [DefNew: 281301K->1687K(314560K),
> 0.0134316 secs] 315238K->35630K(1013632K), 0.0136453 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:40:28.204+0800: 3614.598: [GC (Allocation Failure)
> 2016-04-26T18:40:28.204+0800: 3614.598: [DefNew: 281303K->1687K(314560K),
> 0.0093311 secs] 315246K->35635K(1013632K), 0.0095685 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:41:04.400+0800: 3650.792: [GC (Allocation Failure)
> 2016-04-26T18:41:04.400+0800: 3650.792: [DefNew: 281303K->1684K(314560K),
> 0.0114953 secs] 315251K->35642K(1013632K), 0.0116892 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:41:41.313+0800: 3687.569: [GC (Allocation Failure)
> 2016-04-26T18:41:41.313+0800: 3687.569: [DefNew: 281300K->1683K(314560K),
> 0.0078890 secs] 315258K->35648K(1013632K), 0.0080298 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:42:16.550+0800: 3722.774: [GC (Allocation Failure)
> 2016-04-26T18:42:16.550+0800: 3722.774: [DefNew: 281299K->1679K(314560K),
> 0.0124507 secs] 315264K->35654K(1013632K), 0.0126969 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:42:52.776+0800: 3759.034: [GC (Allocation Failure)
> 2016-04-26T18:42:52.776+0800: 3759.034: [DefNew: 281295K->1673K(314560K),
> 0.0105041 secs] 315270K->35661K(1013632K), 0.0106768 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:43:28.412+0800: 3794.638: [GC (Allocation Failure)
> 2016-04-26T18:43:28.412+0800: 3794.638: [DefNew: 281289K->1674K(314560K),
> 0.0103795 secs] 315277K->35669K(1013632K), 0.0105525 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:44:04.536+0800: 3830.831: [GC (Allocation Failure)
> 2016-04-26T18:44:04.536+0800: 3830.831: [DefNew: 281290K->1674K(314560K),
> 0.0079088 secs] 315285K->35675K(1013632K), 0.0080555 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:44:40.190+0800: 3866.449: [GC (Allocation Failure)
> 2016-04-26T18:44:40.190+0800: 3866.449: [DefNew: 281290K->1673K(314560K),
> 0.0095096 secs] 315291K->35681K(1013632K), 0.0096861 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:45:16.528+0800: 3902.765: [GC (Allocation Failure)
> 2016-04-26T18:45:16.528+0800: 3902.765: [DefNew: 281289K->1673K(314560K),
> 0.0130461 secs] 315297K->35687K(1013632K), 0.0132673 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:45:52.574+0800: 3939.001: [GC (Allocation Failure)
> 2016-04-26T18:45:52.574+0800: 3939.001: [DefNew: 281289K->1680K(314560K),
> 0.0101569 secs] 315303K->35700K(1013632K), 0.0103066 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:46:28.413+0800: 3975.223: [GC (Allocation Failure)
> 2016-04-26T18:46:28.413+0800: 3975.223: [DefNew: 281296K->1679K(314560K),
> 0.0080063 secs] 315316K->35706K(1013632K), 0.0081820 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:47:04.190+0800: 4011.234: [GC (Allocation Failure)
> 2016-04-26T18:47:04.190+0800: 4011.234: [DefNew: 281295K->1678K(314560K),
> 0.0102572 secs] 315322K->35712K(1013632K), 0.0104334 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:47:40.161+0800: 4047.120: [GC (Allocation Failure)
> 2016-04-26T18:47:40.161+0800: 4047.120: [DefNew: 281294K->1677K(314560K),
> 0.0109581 secs] 315328K->35718K(1013632K), 0.0111609 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:48:15.352+0800: 4082.428: [GC (Allocation Failure)
> 2016-04-26T18:48:15.352+0800: 4082.428: [DefNew: 281293K->1677K(314560K),
> 0.0087567 secs] 315334K->35724K(1013632K), 0.0089472 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:48:51.254+0800: 4118.407: [GC (Allocation Failure)
> 2016-04-26T18:48:51.254+0800: 4118.407: [DefNew: 281293K->1677K(314560K),
> 0.0115241 secs] 315340K->35731K(1013632K), 0.0117504 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:49:26.547+0800: 4153.705: [GC (Allocation Failure)
> 2016-04-26T18:49:26.547+0800: 4153.705: [DefNew: 281293K->1967K(314560K),
> 0.0095867 secs] 315347K->36027K(1013632K), 0.0097683 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:50:02.043+0800: 4189.206: [GC (Allocation Failure)
> 2016-04-26T18:50:02.043+0800: 4189.206: [DefNew: 281583K->1676K(314560K),
> 0.0096753 secs] 315643K->35742K(1013632K), 0.0098599 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:50:37.663+0800: 4224.505: [GC (Allocation Failure)
> 2016-04-26T18:50:37.663+0800: 4224.505: [DefNew: 281292K->1675K(314560K),
> 0.0077731 secs] 315358K->35749K(1013632K), 0.0079524 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:51:13.058+0800: 4259.780: [GC (Allocation Failure)
> 2016-04-26T18:51:13.058+0800: 4259.780: [DefNew: 281291K->1676K(314560K),
> 0.0086972 secs] 315365K->35756K(1013632K), 0.0088771 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:51:48.677+0800: 4295.269: [GC (Allocation Failure)
> 2016-04-26T18:51:48.677+0800: 4295.269: [DefNew: 281292K->1931K(314560K),
> 0.0099264 secs] 315372K->36016K(1013632K), 0.0101337 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:52:23.956+0800: 4330.493: [GC (Allocation Failure)
> 2016-04-26T18:52:23.956+0800: 4330.493: [DefNew: 281547K->1673K(314560K),
> 0.0079896 secs] 315632K->35766K(1013632K), 0.0081558 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:52:59.459+0800: 4366.025: [GC (Allocation Failure)
> 2016-04-26T18:52:59.459+0800: 4366.025: [DefNew: 281289K->1942K(314560K),
> 0.0099881 secs] 315382K->36041K(1013632K), 0.0101999 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:53:35.248+0800: 4401.571: [GC (Allocation Failure)
> 2016-04-26T18:53:35.248+0800: 4401.571: [DefNew: 281558K->1675K(314560K),
> 0.0080323 secs] 315657K->35779K(1013632K), 0.0081949 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:54:11.231+0800: 4437.434: [GC (Allocation Failure)
> 2016-04-26T18:54:11.231+0800: 4437.434: [DefNew: 281291K->1672K(314560K),
> 0.0107709 secs] 315395K->35785K(1013632K), 0.0109682 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:54:47.360+0800: 4473.741: [GC (Allocation Failure)
> 2016-04-26T18:54:47.360+0800: 4473.741: [DefNew: 281288K->1675K(314560K),
> 0.0077845 secs] 315401K->35796K(1013632K), 0.0079611 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:55:22.937+0800: 4509.361: [GC (Allocation Failure)
> 2016-04-26T18:55:22.937+0800: 4509.361: [DefNew: 281291K->1675K(314560K),
> 0.0082094 secs] 315412K->35802K(1013632K), 0.0083421 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:55:58.273+0800: 4544.708: [GC (Allocation Failure)
> 2016-04-26T18:55:58.273+0800: 4544.708: [DefNew: 281291K->1674K(314560K),
> 0.0100806 secs] 315418K->35807K(1013632K), 0.0102764 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:56:33.184+0800: 4579.506: [GC (Allocation Failure)
> 2016-04-26T18:56:33.184+0800: 4579.507: [DefNew: 281290K->1674K(314560K),
> 0.0111419 secs] 315423K->35814K(1013632K), 0.0113590 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:57:09.091+0800: 4615.482: [GC (Allocation Failure)
> 2016-04-26T18:57:09.091+0800: 4615.482: [DefNew: 281290K->1672K(314560K),
> 0.0139571 secs] 315430K->35818K(1013632K), 0.0141755 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:57:45.582+0800: 4651.989: [GC (Allocation Failure)
> 2016-04-26T18:57:45.582+0800: 4651.989: [DefNew: 281288K->1672K(314560K),
> 0.0089576 secs] 315434K->35824K(1013632K), 0.0091738 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:58:21.593+0800: 4688.001: [GC (Allocation Failure)
> 2016-04-26T18:58:21.593+0800: 4688.002: [DefNew: 281288K->1676K(314560K),
> 0.0158752 secs] 315440K->35835K(1013632K), 0.0160361 secs] [Times:
> user=0.00 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:58:56.204+0800: 4722.662: [GC (Allocation Failure)
> 2016-04-26T18:58:56.204+0800: 4722.662: [DefNew: 281292K->1675K(314560K),
> 0.0076781 secs] 315451K->35841K(1013632K), 0.0078622 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:59:31.386+0800: 4757.760: [GC (Allocation Failure)
> 2016-04-26T18:59:31.386+0800: 4757.760: [DefNew: 281291K->1678K(314560K),
> 0.0079359 secs] 315457K->35849K(1013632K), 0.0080991 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T19:00:07.775+0800: 4794.332: [GC (Allocation Failure)
> 2016-04-26T19:00:07.775+0800: 4794.332: [DefNew: 281294K->1676K(314560K),
> 0.0087637 secs] 315465K->35854K(1013632K), 0.0089416 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T19:00:44.243+0800: 4830.820: [GC (Allocation Failure)
> 2016-04-26T19:00:44.243+0800: 4830.820: [DefNew: 281292K->1674K(314560K),
> 0.0098652 secs] 315470K->35859K(1013632K), 0.0101359 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T19:01:19.025+0800: 4865.773: [GC (Allocation Failure)
> 2016-04-26T19:01:19.025+0800: 4865.773: [DefNew: 281290K->1686K(314560K),
> 0.0077619 secs] 315475K->35877K(1013632K), 0.0079340 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T19:01:47.622+0800: 4894.541: [GC (Allocation Failure)
> 2016-04-26T19:01:47.622+0800: 4894.541: [DefNew: 281302K->2540K(314560K),
> 0.0189695 secs] 315493K->36737K(1013632K), 0.0192055 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> Heap
>
> def new generation total 314560K, used 268425K [0x04600000, 0x19b50000,
> 0x19b50000)
>
> eden space 279616K, 95% used [0x04600000, 0x149a73e0, 0x15710000)
>
> from space 34944K, 7% used [0x17930000, 0x17bab258, 0x19b50000)
>
> to space 34944K, 0% used [0x15710000, 0x15710000, 0x17930000)
>
> tenured generation total 699072K, used 34196K [0x19b50000, 0x44600000,
> 0x44600000)
>
> the space 699072K, 4% used [0x19b50000, 0x1bcb5300, 0x1bcb5400,
> 0x44600000)
>
> Metaspace used 21496K, capacity 21761K, committed 21888K, reserved
> 21888K
>
>
>
>
>
> *发件人:* Vladimir Ozerov [mailto:vozerov@gridgain.com]
> *发送时间:* 2016年4月25日 20:21
> *收件人:* user@ignite.apache.org
> *主题:* Re: Ignite cache data size problem.
>
>
>
> Hi Kevin,
>
>
>
> I performed several experiments. Essentially, I put 1M entries of the
> class you provided with fields initialized as follows:
>
>
>
> *for *(*int *i = 0; i < 1_000_000; i++) {
> UniqueField field = *new *UniqueField();
>
> field.setDate(*new *Date());
> field.setGuid(UUID.*randomUUID*().toString());
> field.setMsg(String.*valueOf*(i));
> field.setNum(BigDecimal.*valueOf*(ThreadLocalRandom.*current*().nextDouble()));
> field.setOId(String.*valueOf*(i));
> field.setOrgId(String.*valueOf*(i));
>
> cache.put(i, field);
> }
>
>
>
> My results are:
>
> 1) Onheap, no indexes - about 400Mb is required to store 1M objects, or
> ~20Gb for 47M objects.
>
> 2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
>
> 3) Offheap, with indexes - about 400Mb offheap memory is required, or
> ~20Gb for all your objects.
>
>
>
> Could you please provide more information on the error you receive? Also,
> you could try load entries in a batches of a well-defined size (say, 1M),
> and see what happens to the system. I expect you should see similar numbers.
>
>
>
> Vladimir.
>
>
>
> On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zh...@huawei.com>
> wrote:
>
> BTW, I created 4 + 3 nodes on two servers.
> each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
>
> kind regards,
> Kevin
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
>
>
re: Ignite cache data size problem.
Posted by Zhengqingzheng <zh...@huawei.com>.
Hi Vladimir,
No problem.
I will re-run the loading process and give you the log file.
To be clear, when you say log file, do you mean files located at work/log/*?
I use IgntieCache.loadCache(null, “select * from table”) to load all the data.
Best regards,
Kevin
发件人: Vladimir Ozerov [mailto:vozerov@gridgain.com]
发送时间: 2016年4月26日 20:15
收件人: user@ignite.apache.org
主题: Re: Ignite cache data size problem.
Hi Kevin,
Could you please re-run your case and attach Ignite logs and GC logs from all participating servers? I would expect that this is either a kind of out-of-memory problem, or network saturation. Also please explain how exactly do you load data to Ignite? Do you use DataStreamer, or may be IgniteCache.loadlCache, or IgniteCache.put?
Vladimir.
On Tue, Apr 26, 2016 at 3:01 PM, Zhengqingzheng <zh...@huawei.com>> wrote:
Hi Vladimir,
Thank you for your help.
I tried to load 1Million records, and calculate each object’s size (including key and value object, where key is string type), summarized together and found the total memory consumption is 130Mb.
Because ./ignitevisor.sh command only shows number of records, and no data allocation information, I don’t know how much memory has been consumed for each type of cache.
My result is as follows:
Log:
Found big object: [Ljava.util.HashMap$Node;@24833164 size: 30.888206481933594Mb
Found big object: java.util.HashMap@15819383<ma...@15819383> size: 30.88824462890625Mb
Found big object: java.util.HashSet@10236420<ma...@10236420> size: 30.888259887695312Mb
key size: 32388688 human readable data: 30.888259887695312Mb
Found big object: [Lorg.jsr166.ConcurrentHashMap8$Node;@29556439 size: 129.99818420410156Mb
Found big object: org.jsr166.ConcurrentHashMap8@19238297<ma...@19238297> size: 129.99822235107422Mb
value size: 136313016 human readable data: 129.99822235107422Mb
the whole number of record is 47Million, so the data size inside the cache should be 130*47=6110Mb(around 6Gb).
However, when I try to load the whole data into the cache, I still get exceptions:
The exception information is listed as follows:
1. ---------------------------------- exception info from client ----------------------------------------
Before exception occurred, I have 10 nodes on two servers, server1 (48G ram) has 6nodes, each node is assigned with 7gb jvm heap; server2 (32gb ram) has 4 nodes with the same jvm settings as previous one.
After exception, client stopped, and 8 node left (server1’s node remain, no exception occurred on this server), server2 ( 2 nodes remain, two nodes droped)
The total number of records loaded is 37Million.
[19:01:47] Topology snapshot [ver=77, servers=9, clients=1, CPUs=20, heap=64.0GB]
[19:01:47,463][SEVERE][pub-#46%null%][GridTaskWorker] Failed to obtain remote job result policy for result from ComputeTask.result(..) method (will fail the whole task): GridJobResultImpl [job=C2 [], sib=GridJobSiblingImpl [sesId=0dbc0f15451-880a2dd1-bc95-4084-a705-4effcec5d2cd, jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924, nodeId=832bed3e-dc5d-4743-9853-127e3b516924, isJobDone=false], jobCtx=GridJobContextImpl [jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924, timeoutObj=null, attrs={}], node=TcpDiscoveryNode [id=832bed3e-dc5d-4743-9853-127e3b516924, addrs=[0:0:0:0:0:0:0:1%lo, 10.120.70.122, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /0:0:0:0:0:0:0:1%lo:47500, /10.120.70.122:47500<http://10.120.70.122:47500>, /127.0.0.1:47500<http://127.0.0.1:47500>], discPort=47500, order=7, intOrder=7, lastExchangeTime=1461663619161, loc=false, ver=1.5.0#20151229-sha1:f1f8cda2, isClient=false], ex=class o.a.i.cluster.ClusterTopologyException: Node has left grid: 832bed3e-dc5d-4743-9853-127e3b516924, hasRes=true, isCancelled=false, isOccupied=true]
class org.apache.ignite.cluster.ClusterTopologyException: Node has left grid: 832bed3e-dc5d-4743-9853-127e3b516924
at org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1315)
at org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1.run(GridTaskProcessor.java:1246)
at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6453)
at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:788)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[19:02:16] Ignite node stopped OK [uptime=01:21:52:963]
2 ------------------------------ also, I use the recommended settings for the jvm and get the following log information ------------------------------------------------
Java HotSpot(TM) Client VM (25.66-b18) for windows-x86 JRE (1.8.0_66-b18), built on Nov 9 2015 10:58:29 by "java_re" with MS VC++ 10.0 (VS2010)
Memory: 4k page, physical 6291064k(1912832k free), swap 12580288k(6996288k free)
CommandLine flags: -XX:GCLogFileSize=104857600 -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:-UseLargePagesIndividualAllocation
2016-04-26T17:40:15.609+0800: 4.839: [Full GC (Metadata GC Threshold) 2016-04-26T17:40:15.610+0800: 4.839: [Tenured: 0K->10330K(699072K), 0.0656011 secs] 162451K->10330K(1013632K), [Metaspace: 12085K->12085K(12672K)], 0.0659450 secs] [Times: user=0.06 sys=0.00, real=0.07 secs]
2016-04-26T17:40:24.837+0800: 14.044: [Full GC (Metadata GC Threshold) 2016-04-26T17:40:24.837+0800: 14.044: [Tenured: 10330K->32179K(699072K), 0.1810716 secs] 282679K->32179K(1013632K), [Metaspace: 20221K->20221K(20864K)], 0.1812825 secs] [Times: user=0.17 sys=0.00, real=0.18 secs]
2016-04-26T17:40:53.699+0800: 42.937: [GC (Allocation Failure) 2016-04-26T17:40:53.699+0800: 42.937: [DefNew: 279616K->8837K(314560K), 0.0242718 secs] 311795K->41016K(1013632K), 0.0244783 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2016-04-26T17:41:26.411+0800: 75.516: [GC (Allocation Failure) 2016-04-26T17:41:26.412+0800: 75.516: [DefNew: 288453K->2685K(314560K), 0.0116993 secs] 320632K->34864K(1013632K), 0.0119035 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:42:01.895+0800: 110.834: [GC (Allocation Failure) 2016-04-26T17:42:01.895+0800: 110.834: [DefNew: 282301K->2689K(314560K), 0.0111799 secs] 314480K->34868K(1013632K), 0.0113629 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:42:37.354+0800: 145.818: [GC (Allocation Failure) 2016-04-26T17:42:37.354+0800: 145.818: [DefNew: 282305K->2695K(314560K), 0.0102415 secs] 314484K->34874K(1013632K), 0.0104181 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:43:13.549+0800: 181.641: [GC (Allocation Failure) 2016-04-26T17:43:13.549+0800: 181.641: [DefNew: 282311K->2893K(314560K), 0.0134864 secs] 314490K->35072K(1013632K), 0.0136970 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:43:50.453+0800: 218.129: [GC (Allocation Failure) 2016-04-26T17:43:50.453+0800: 218.129: [DefNew: 282509K->2708K(314560K), 0.0160308 secs] 314688K->34887K(1013632K), 0.0162459 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2016-04-26T17:44:27.317+0800: 254.604: [GC (Allocation Failure) 2016-04-26T17:44:27.317+0800: 254.605: [DefNew: 282324K->2715K(314560K), 0.0128055 secs] 314503K->34894K(1013632K), 0.0130086 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:45:03.944+0800: 291.040: [GC (Allocation Failure) 2016-04-26T17:45:03.944+0800: 291.040: [DefNew: 282331K->2724K(314560K), 0.0109078 secs] 314510K->34903K(1013632K), 0.0110838 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:45:41.069+0800: 327.816: [GC (Allocation Failure) 2016-04-26T17:45:41.069+0800: 327.816: [DefNew: 282340K->2732K(314560K), 0.0128223 secs] 314519K->34912K(1013632K), 0.0130151 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:46:18.108+0800: 364.538: [GC (Allocation Failure) 2016-04-26T17:46:18.108+0800: 364.538: [DefNew: 282348K->2738K(314560K), 0.0123320 secs] 314528K->34917K(1013632K), 0.0125273 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:46:54.492+0800: 400.975: [GC (Allocation Failure) 2016-04-26T17:46:54.492+0800: 400.975: [DefNew: 282354K->2747K(314560K), 0.0126284 secs] 314533K->34926K(1013632K), 0.0128229 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:47:36.805+0800: 443.099: [GC (Allocation Failure) 2016-04-26T17:47:36.805+0800: 443.099: [DefNew: 282363K->2755K(314560K), 0.0137048 secs] 314542K->34934K(1013632K), 0.0139138 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:48:13.500+0800: 480.054: [GC (Allocation Failure) 2016-04-26T17:48:13.500+0800: 480.054: [DefNew: 282371K->2764K(314560K), 0.0134777 secs] 314550K->34943K(1013632K), 0.0137571 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:48:50.568+0800: 517.242: [GC (Allocation Failure) 2016-04-26T17:48:50.568+0800: 517.242: [DefNew: 282380K->2769K(314560K), 0.0121474 secs] 314559K->34948K(1013632K), 0.0125373 secs] [Times: user=0.00 sys=0.02, real=0.02 secs]
2016-04-26T17:49:28.506+0800: 555.103: [GC (Allocation Failure) 2016-04-26T17:49:28.507+0800: 555.103: [DefNew: 282385K->2777K(314560K), 0.0133794 secs] 314564K->34957K(1013632K), 0.0135732 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:50:05.869+0800: 592.355: [GC (Allocation Failure) 2016-04-26T17:50:05.869+0800: 592.355: [DefNew: 282393K->1713K(314560K), 0.0129486 secs] 314573K->34965K(1013632K), 0.0131366 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:50:43.336+0800: 629.663: [GC (Allocation Failure) 2016-04-26T17:50:43.336+0800: 629.663: [DefNew: 281329K->1688K(314560K), 0.0107519 secs] 314581K->34971K(1013632K), 0.0109511 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:51:20.603+0800: 666.805: [GC (Allocation Failure) 2016-04-26T17:51:20.604+0800: 666.806: [DefNew: 281304K->2043K(314560K), 0.0125329 secs] 314587K->35333K(1013632K), 0.0127661 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:51:57.981+0800: 704.195: [GC (Allocation Failure) 2016-04-26T17:51:57.981+0800: 704.195: [DefNew: 281659K->1688K(314560K), 0.0083589 secs] 314949K->34985K(1013632K), 0.0085578 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:52:35.234+0800: 741.413: [GC (Allocation Failure) 2016-04-26T17:52:35.234+0800: 741.413: [DefNew: 281304K->1686K(314560K), 0.0080466 secs] 314601K->34990K(1013632K), 0.0082145 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:53:11.734+0800: 778.046: [GC (Allocation Failure) 2016-04-26T17:53:11.734+0800: 778.046: [DefNew: 281302K->1689K(314560K), 0.0106016 secs] 314606K->35000K(1013632K), 0.0108028 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:53:48.679+0800: 814.992: [GC (Allocation Failure) 2016-04-26T17:53:48.679+0800: 814.992: [DefNew: 281305K->1689K(314560K), 0.0085276 secs] 314616K->35007K(1013632K), 0.0086721 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:54:24.734+0800: 851.078: [GC (Allocation Failure) 2016-04-26T17:54:24.735+0800: 851.078: [DefNew: 281305K->1685K(314560K), 0.0130525 secs] 314623K->35012K(1013632K), 0.0132933 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:55:01.614+0800: 888.090: [GC (Allocation Failure) 2016-04-26T17:55:01.614+0800: 888.090: [DefNew: 281301K->1686K(314560K), 0.0084343 secs] 314628K->35021K(1013632K), 0.0086352 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:55:38.728+0800: 925.129: [GC (Allocation Failure) 2016-04-26T17:55:38.728+0800: 925.129: [DefNew: 281302K->1684K(314560K), 0.0081086 secs] 314637K->35026K(1013632K), 0.0082835 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:56:15.257+0800: 961.859: [GC (Allocation Failure) 2016-04-26T17:56:15.257+0800: 961.859: [DefNew: 281300K->1683K(314560K), 0.0097599 secs] 314642K->35032K(1013632K), 0.0099144 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:56:51.417+0800: 998.123: [GC (Allocation Failure) 2016-04-26T17:56:51.417+0800: 998.123: [DefNew: 281299K->1681K(314560K), 0.0095046 secs] 314648K->35038K(1013632K), 0.0096970 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:57:27.838+0800: 1034.605: [GC (Allocation Failure) 2016-04-26T17:57:27.838+0800: 1034.605: [DefNew: 281297K->1681K(314560K), 0.0104843 secs] 314654K->35045K(1013632K), 0.0106871 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:58:04.008+0800: 1070.815: [GC (Allocation Failure) 2016-04-26T17:58:04.008+0800: 1070.815: [DefNew: 281297K->1680K(314560K), 0.0087810 secs] 314661K->35051K(1013632K), 0.0089486 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:58:40.916+0800: 1107.629: [GC (Allocation Failure) 2016-04-26T17:58:40.916+0800: 1107.629: [DefNew: 281296K->1679K(314560K), 0.0123532 secs] 314667K->35058K(1013632K), 0.0125745 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:59:17.297+0800: 1143.843: [GC (Allocation Failure) 2016-04-26T17:59:17.297+0800: 1143.843: [DefNew: 281295K->1682K(314560K), 0.0099356 secs] 314674K->35069K(1013632K), 0.0101206 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:59:54.265+0800: 1180.679: [GC (Allocation Failure) 2016-04-26T17:59:54.265+0800: 1180.680: [DefNew: 281298K->1678K(314560K), 0.0089453 secs] 314685K->35071K(1013632K), 0.0091503 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:00:31.910+0800: 1218.043: [GC (Allocation Failure) 2016-04-26T18:00:31.910+0800: 1218.043: [DefNew: 281294K->1677K(314560K), 0.0082765 secs] 314687K->35078K(1013632K), 0.0084452 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:01:07.853+0800: 1254.048: [GC (Allocation Failure) 2016-04-26T18:01:07.854+0800: 1254.048: [DefNew: 281293K->1678K(314560K), 0.0089447 secs] 314694K->35085K(1013632K), 0.0091405 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:01:44.497+0800: 1290.474: [GC (Allocation Failure) 2016-04-26T18:01:44.497+0800: 1290.474: [DefNew: 281294K->1678K(314560K), 0.0076934 secs] 314701K->35091K(1013632K), 0.0078666 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:02:27.863+0800: 1333.673: [GC (Allocation Failure) 2016-04-26T18:02:27.863+0800: 1333.674: [DefNew: 281294K->1674K(314560K), 0.0084192 secs] 314707K->35098K(1013632K), 0.0085597 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:03:04.588+0800: 1370.424: [GC (Allocation Failure) 2016-04-26T18:03:04.589+0800: 1370.424: [DefNew: 281290K->1677K(314560K), 0.0086385 secs] 314714K->35107K(1013632K), 0.0087911 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:03:41.646+0800: 1407.396: [GC (Allocation Failure) 2016-04-26T18:03:41.646+0800: 1407.396: [DefNew: 281293K->1674K(314560K), 0.0079049 secs] 314723K->35111K(1013632K), 0.0080862 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:04:18.618+0800: 1444.458: [GC (Allocation Failure) 2016-04-26T18:04:18.618+0800: 1444.458: [DefNew: 281290K->1674K(314560K), 0.0089752 secs] 314727K->35118K(1013632K), 0.0091841 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:04:55.359+0800: 1481.281: [GC (Allocation Failure) 2016-04-26T18:04:55.359+0800: 1481.281: [DefNew: 281290K->1674K(314560K), 0.0091816 secs] 314734K->35124K(1013632K), 0.0093663 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:05:31.696+0800: 1517.690: [GC (Allocation Failure) 2016-04-26T18:05:31.696+0800: 1517.690: [DefNew: 281290K->1680K(314560K), 0.0097599 secs] 314740K->35136K(1013632K), 0.0099613 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:06:08.293+0800: 1554.357: [GC (Allocation Failure) 2016-04-26T18:06:08.293+0800: 1554.357: [DefNew: 281296K->1681K(314560K), 0.0102206 secs] 314752K->35144K(1013632K), 0.0104262 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:06:44.886+0800: 1590.882: [GC (Allocation Failure) 2016-04-26T18:06:44.886+0800: 1590.882: [DefNew: 281297K->1680K(314560K), 0.0090525 secs] 314760K->35150K(1013632K), 0.0092456 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:07:20.852+0800: 1626.984: [GC (Allocation Failure) 2016-04-26T18:07:20.852+0800: 1626.984: [DefNew: 281296K->1682K(314560K), 0.0095817 secs] 314766K->35158K(1013632K), 0.0097445 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:07:57.290+0800: 1663.543: [GC (Allocation Failure) 2016-04-26T18:07:57.290+0800: 1663.543: [DefNew: 281298K->1682K(314560K), 0.0076180 secs] 314774K->35164K(1013632K), 0.0077856 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:08:34.599+0800: 1700.805: [GC (Allocation Failure) 2016-04-26T18:08:34.599+0800: 1700.805: [DefNew: 281298K->1680K(314560K), 0.0136244 secs] 314780K->35170K(1013632K), 0.0138406 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:09:10.999+0800: 1737.438: [GC (Allocation Failure) 2016-04-26T18:09:10.999+0800: 1737.438: [DefNew: 281296K->1681K(314560K), 0.0086762 secs] 314786K->35177K(1013632K), 0.0088701 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:09:47.695+0800: 1774.086: [GC (Allocation Failure) 2016-04-26T18:09:47.695+0800: 1774.086: [DefNew: 281297K->1679K(314560K), 0.0079063 secs] 314793K->35182K(1013632K), 0.0080798 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:10:24.630+0800: 1811.127: [GC (Allocation Failure) 2016-04-26T18:10:24.630+0800: 1811.127: [DefNew: 281295K->1680K(314560K), 0.0100454 secs] 314798K->35189K(1013632K), 0.0101801 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:11:00.972+0800: 1847.581: [GC (Allocation Failure) 2016-04-26T18:11:00.972+0800: 1847.581: [DefNew: 281296K->1679K(314560K), 0.0083639 secs] 314805K->35195K(1013632K), 0.0085452 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:11:37.280+0800: 1883.810: [GC (Allocation Failure) 2016-04-26T18:11:37.280+0800: 1883.810: [DefNew: 281295K->1679K(314560K), 0.0084648 secs] 314811K->35202K(1013632K), 0.0086475 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:12:13.694+0800: 1920.386: [GC (Allocation Failure) 2016-04-26T18:12:13.694+0800: 1920.386: [DefNew: 281295K->1677K(314560K), 0.0080820 secs] 314818K->35207K(1013632K), 0.0082119 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:12:49.894+0800: 1956.802: [GC (Allocation Failure) 2016-04-26T18:12:49.894+0800: 1956.803: [DefNew: 281293K->1677K(314560K), 0.0146823 secs] 314823K->35213K(1013632K), 0.0149620 secs] [Times: user=0.00 sys=0.02, real=0.01 secs]
2016-04-26T18:13:26.430+0800: 1993.317: [GC (Allocation Failure) 2016-04-26T18:13:26.430+0800: 1993.317: [DefNew: 281293K->1677K(314560K), 0.0088413 secs] 314829K->35220K(1013632K), 0.0090009 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:14:03.233+0800: 2030.321: [GC (Allocation Failure) 2016-04-26T18:14:03.233+0800: 2030.321: [DefNew: 281293K->1676K(314560K), 0.0078476 secs] 314836K->35225K(1013632K), 0.0079887 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:14:40.063+0800: 2067.102: [GC (Allocation Failure) 2016-04-26T18:14:40.063+0800: 2067.102: [DefNew: 281292K->1677K(314560K), 0.0092786 secs] 314841K->35232K(1013632K), 0.0094601 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:15:16.165+0800: 2103.376: [GC (Allocation Failure) 2016-04-26T18:15:16.165+0800: 2103.376: [DefNew: 281293K->2106K(314560K), 0.0135981 secs] 314848K->35675K(1013632K), 0.0138093 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:15:52.922+0800: 2139.884: [GC (Allocation Failure) 2016-04-26T18:15:52.922+0800: 2139.884: [DefNew: 281722K->1669K(314560K), 0.0123189 secs] 315291K->35244K(1013632K), 0.0125281 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:16:30.496+0800: 2177.184: [GC (Allocation Failure) 2016-04-26T18:16:30.496+0800: 2177.184: [DefNew: 281285K->1669K(314560K), 0.0089190 secs] 314860K->35251K(1013632K), 0.0090788 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:17:06.767+0800: 2213.365: [GC (Allocation Failure) 2016-04-26T18:17:06.767+0800: 2213.365: [DefNew: 281285K->1826K(314560K), 0.0079870 secs] 314867K->35415K(1013632K), 0.0081401 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:17:43.991+0800: 2250.422: [GC (Allocation Failure) 2016-04-26T18:17:43.991+0800: 2250.422: [DefNew: 281442K->1669K(314560K), 0.0115439 secs] 315031K->35264K(1013632K), 0.0117428 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:18:20.528+0800: 2286.733: [GC (Allocation Failure) 2016-04-26T18:18:20.528+0800: 2286.733: [DefNew: 281285K->1668K(314560K), 0.0082332 secs] 314880K->35270K(1013632K), 0.0084053 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:18:57.215+0800: 2323.271: [GC (Allocation Failure) 2016-04-26T18:18:57.215+0800: 2323.271: [DefNew: 281284K->1671K(314560K), 0.0077934 secs] 314886K->35277K(1013632K), 0.0079697 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:19:34.151+0800: 2359.977: [GC (Allocation Failure) 2016-04-26T18:19:34.151+0800: 2359.977: [DefNew: 281287K->1669K(314560K), 0.0078809 secs] 314893K->35283K(1013632K), 0.0080410 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:20:11.106+0800: 2396.706: [GC (Allocation Failure) 2016-04-26T18:20:11.106+0800: 2396.706: [DefNew: 281285K->1673K(314560K), 0.0092928 secs] 314899K->35293K(1013632K), 0.0094864 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:20:47.129+0800: 2432.801: [GC (Allocation Failure) 2016-04-26T18:20:47.129+0800: 2432.801: [DefNew: 281289K->1672K(314560K), 0.0089699 secs] 314909K->35298K(1013632K), 0.0091397 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:21:23.028+0800: 2468.633: [GC (Allocation Failure) 2016-04-26T18:21:23.028+0800: 2468.633: [DefNew: 281288K->1671K(314560K), 0.0124404 secs] 314914K->35303K(1013632K), 0.0128810 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:21:59.745+0800: 2505.506: [GC (Allocation Failure) 2016-04-26T18:21:59.745+0800: 2505.506: [DefNew: 281287K->1671K(314560K), 0.0080019 secs] 314919K->35309K(1013632K), 0.0082167 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:22:36.639+0800: 2542.570: [GC (Allocation Failure) 2016-04-26T18:22:36.639+0800: 2542.570: [DefNew: 281287K->1670K(314560K), 0.0082147 secs] 314925K->35315K(1013632K), 0.0083938 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:23:12.390+0800: 2578.421: [GC (Allocation Failure) 2016-04-26T18:23:12.390+0800: 2578.421: [DefNew: 281286K->2207K(314560K), 0.0114121 secs] 314931K->35857K(1013632K), 0.0116115 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:23:48.693+0800: 2615.071: [GC (Allocation Failure) 2016-04-26T18:23:48.693+0800: 2615.071: [DefNew: 281823K->1798K(314560K), 0.0104047 secs] 315473K->35455K(1013632K), 0.0105935 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:24:25.698+0800: 2652.227: [GC (Allocation Failure) 2016-04-26T18:24:25.698+0800: 2652.228: [DefNew: 281414K->1795K(314560K), 0.0102968 secs] 315071K->35459K(1013632K), 0.0104854 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:25:01.801+0800: 2688.457: [GC (Allocation Failure) 2016-04-26T18:25:01.801+0800: 2688.458: [DefNew: 281411K->2126K(314560K), 0.0090964 secs] 315075K->35795K(1013632K), 0.0093529 secs] [Times: user=0.00 sys=0.02, real=0.01 secs]
2016-04-26T18:25:39.055+0800: 2725.732: [GC (Allocation Failure) 2016-04-26T18:25:39.055+0800: 2725.732: [DefNew: 281742K->1799K(314560K), 0.0085785 secs] 315411K->35474K(1013632K), 0.0087168 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:26:27.232+0800: 2773.853: [GC (Allocation Failure) 2016-04-26T18:26:27.232+0800: 2773.853: [DefNew: 281415K->1800K(314560K), 0.0091224 secs] 315090K->35482K(1013632K), 0.0094059 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:27:05.583+0800: 2812.453: [GC (Allocation Failure) 2016-04-26T18:27:05.584+0800: 2812.453: [DefNew: 281416K->1801K(314560K), 0.0083206 secs] 315098K->35489K(1013632K), 0.0084969 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:27:43.412+0800: 2850.382: [GC (Allocation Failure) 2016-04-26T18:27:43.412+0800: 2850.382: [DefNew: 281417K->1801K(314560K), 0.0095576 secs] 315105K->35495K(1013632K), 0.0097616 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:28:20.067+0800: 2887.191: [GC (Allocation Failure) 2016-04-26T18:28:20.067+0800: 2887.191: [DefNew: 281417K->1799K(314560K), 0.0121244 secs] 315111K->35501K(1013632K), 0.0123463 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:28:55.431+0800: 2923.064: [GC (Allocation Failure) 2016-04-26T18:28:55.431+0800: 2923.064: [DefNew: 281415K->1798K(314560K), 0.0088000 secs] 315117K->35506K(1013632K), 0.0089771 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:29:30.761+0800: 2958.335: [GC (Allocation Failure) 2016-04-26T18:29:30.761+0800: 2958.335: [DefNew: 281414K->1798K(314560K), 0.0113492 secs] 315122K->35512K(1013632K), 0.0115526 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:30:05.795+0800: 2993.528: [GC (Allocation Failure) 2016-04-26T18:30:05.795+0800: 2993.528: [DefNew: 281414K->1797K(314560K), 0.0112305 secs] 315128K->35518K(1013632K), 0.0114227 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:30:41.648+0800: 3029.322: [GC (Allocation Failure) 2016-04-26T18:30:41.648+0800: 3029.322: [DefNew: 281413K->1797K(314560K), 0.0174930 secs] 315134K->35524K(1013632K), 0.0177282 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:31:34.467+0800: 3082.237: [GC (Allocation Failure) 2016-04-26T18:31:34.467+0800: 3082.237: [DefNew: 281413K->1800K(314560K), 0.0095434 secs] 315140K->35534K(1013632K), 0.0097186 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:32:14.021+0800: 3121.810: [GC (Allocation Failure) 2016-04-26T18:32:14.021+0800: 3121.810: [DefNew: 281416K->1800K(314560K), 0.0096049 secs] 315150K->35540K(1013632K), 0.0097856 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:33:07.874+0800: 3175.537: [GC (Allocation Failure) 2016-04-26T18:33:07.874+0800: 3175.537: [DefNew: 281416K->1697K(314560K), 0.0135850 secs] 315156K->35550K(1013632K), 0.0138194 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:33:47.621+0800: 3215.029: [GC (Allocation Failure) 2016-04-26T18:33:47.621+0800: 3215.029: [DefNew: 281313K->1694K(314560K), 0.0080256 secs] 315166K->35566K(1013632K), 0.0081854 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:34:24.160+0800: 3251.370: [GC (Allocation Failure) 2016-04-26T18:34:24.160+0800: 3251.371: [DefNew: 281310K->1691K(314560K), 0.0086257 secs] 315182K->35570K(1013632K), 0.0087810 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:34:59.507+0800: 3286.713: [GC (Allocation Failure) 2016-04-26T18:34:59.507+0800: 3286.713: [DefNew: 281307K->1690K(314560K), 0.0082689 secs] 315186K->35576K(1013632K), 0.0084583 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:35:34.974+0800: 3322.008: [GC (Allocation Failure) 2016-04-26T18:35:34.974+0800: 3322.008: [DefNew: 281306K->1690K(314560K), 0.0134400 secs] 315192K->35582K(1013632K), 0.0136509 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:36:11.007+0800: 3357.907: [GC (Allocation Failure) 2016-04-26T18:36:11.007+0800: 3357.907: [DefNew: 281306K->1689K(314560K), 0.0118568 secs] 315198K->35590K(1013632K), 0.0120744 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:36:48.227+0800: 3394.956: [GC (Allocation Failure) 2016-04-26T18:36:48.227+0800: 3394.956: [DefNew: 281305K->1688K(314560K), 0.0083156 secs] 315206K->35596K(1013632K), 0.0084603 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2016-04-26T18:37:24.551+0800: 3430.991: [GC (Allocation Failure) 2016-04-26T18:37:24.551+0800: 3430.991: [DefNew: 281304K->1688K(314560K), 0.0091559 secs] 315212K->35602K(1013632K), 0.0093369 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:38:01.307+0800: 3467.736: [GC (Allocation Failure) 2016-04-26T18:38:01.307+0800: 3467.736: [DefNew: 281304K->1685K(314560K), 0.0085799 secs] 315218K->35609K(1013632K), 0.0087592 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:38:37.590+0800: 3503.935: [GC (Allocation Failure) 2016-04-26T18:38:37.590+0800: 3503.935: [DefNew: 281301K->1685K(314560K), 0.0103468 secs] 315225K->35616K(1013632K), 0.0105709 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:39:14.026+0800: 3540.397: [GC (Allocation Failure) 2016-04-26T18:39:14.026+0800: 3540.397: [DefNew: 281301K->1685K(314560K), 0.0119247 secs] 315232K->35622K(1013632K), 0.0121515 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:39:50.866+0800: 3577.377: [GC (Allocation Failure) 2016-04-26T18:39:50.866+0800: 3577.377: [DefNew: 281301K->1687K(314560K), 0.0134316 secs] 315238K->35630K(1013632K), 0.0136453 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:40:28.204+0800: 3614.598: [GC (Allocation Failure) 2016-04-26T18:40:28.204+0800: 3614.598: [DefNew: 281303K->1687K(314560K), 0.0093311 secs] 315246K->35635K(1013632K), 0.0095685 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:41:04.400+0800: 3650.792: [GC (Allocation Failure) 2016-04-26T18:41:04.400+0800: 3650.792: [DefNew: 281303K->1684K(314560K), 0.0114953 secs] 315251K->35642K(1013632K), 0.0116892 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:41:41.313+0800: 3687.569: [GC (Allocation Failure) 2016-04-26T18:41:41.313+0800: 3687.569: [DefNew: 281300K->1683K(314560K), 0.0078890 secs] 315258K->35648K(1013632K), 0.0080298 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:42:16.550+0800: 3722.774: [GC (Allocation Failure) 2016-04-26T18:42:16.550+0800: 3722.774: [DefNew: 281299K->1679K(314560K), 0.0124507 secs] 315264K->35654K(1013632K), 0.0126969 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:42:52.776+0800: 3759.034: [GC (Allocation Failure) 2016-04-26T18:42:52.776+0800: 3759.034: [DefNew: 281295K->1673K(314560K), 0.0105041 secs] 315270K->35661K(1013632K), 0.0106768 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:43:28.412+0800: 3794.638: [GC (Allocation Failure) 2016-04-26T18:43:28.412+0800: 3794.638: [DefNew: 281289K->1674K(314560K), 0.0103795 secs] 315277K->35669K(1013632K), 0.0105525 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:44:04.536+0800: 3830.831: [GC (Allocation Failure) 2016-04-26T18:44:04.536+0800: 3830.831: [DefNew: 281290K->1674K(314560K), 0.0079088 secs] 315285K->35675K(1013632K), 0.0080555 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:44:40.190+0800: 3866.449: [GC (Allocation Failure) 2016-04-26T18:44:40.190+0800: 3866.449: [DefNew: 281290K->1673K(314560K), 0.0095096 secs] 315291K->35681K(1013632K), 0.0096861 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:45:16.528+0800: 3902.765: [GC (Allocation Failure) 2016-04-26T18:45:16.528+0800: 3902.765: [DefNew: 281289K->1673K(314560K), 0.0130461 secs] 315297K->35687K(1013632K), 0.0132673 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:45:52.574+0800: 3939.001: [GC (Allocation Failure) 2016-04-26T18:45:52.574+0800: 3939.001: [DefNew: 281289K->1680K(314560K), 0.0101569 secs] 315303K->35700K(1013632K), 0.0103066 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:46:28.413+0800: 3975.223: [GC (Allocation Failure) 2016-04-26T18:46:28.413+0800: 3975.223: [DefNew: 281296K->1679K(314560K), 0.0080063 secs] 315316K->35706K(1013632K), 0.0081820 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:47:04.190+0800: 4011.234: [GC (Allocation Failure) 2016-04-26T18:47:04.190+0800: 4011.234: [DefNew: 281295K->1678K(314560K), 0.0102572 secs] 315322K->35712K(1013632K), 0.0104334 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:47:40.161+0800: 4047.120: [GC (Allocation Failure) 2016-04-26T18:47:40.161+0800: 4047.120: [DefNew: 281294K->1677K(314560K), 0.0109581 secs] 315328K->35718K(1013632K), 0.0111609 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:48:15.352+0800: 4082.428: [GC (Allocation Failure) 2016-04-26T18:48:15.352+0800: 4082.428: [DefNew: 281293K->1677K(314560K), 0.0087567 secs] 315334K->35724K(1013632K), 0.0089472 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:48:51.254+0800: 4118.407: [GC (Allocation Failure) 2016-04-26T18:48:51.254+0800: 4118.407: [DefNew: 281293K->1677K(314560K), 0.0115241 secs] 315340K->35731K(1013632K), 0.0117504 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:49:26.547+0800: 4153.705: [GC (Allocation Failure) 2016-04-26T18:49:26.547+0800: 4153.705: [DefNew: 281293K->1967K(314560K), 0.0095867 secs] 315347K->36027K(1013632K), 0.0097683 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:50:02.043+0800: 4189.206: [GC (Allocation Failure) 2016-04-26T18:50:02.043+0800: 4189.206: [DefNew: 281583K->1676K(314560K), 0.0096753 secs] 315643K->35742K(1013632K), 0.0098599 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:50:37.663+0800: 4224.505: [GC (Allocation Failure) 2016-04-26T18:50:37.663+0800: 4224.505: [DefNew: 281292K->1675K(314560K), 0.0077731 secs] 315358K->35749K(1013632K), 0.0079524 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:51:13.058+0800: 4259.780: [GC (Allocation Failure) 2016-04-26T18:51:13.058+0800: 4259.780: [DefNew: 281291K->1676K(314560K), 0.0086972 secs] 315365K->35756K(1013632K), 0.0088771 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:51:48.677+0800: 4295.269: [GC (Allocation Failure) 2016-04-26T18:51:48.677+0800: 4295.269: [DefNew: 281292K->1931K(314560K), 0.0099264 secs] 315372K->36016K(1013632K), 0.0101337 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:52:23.956+0800: 4330.493: [GC (Allocation Failure) 2016-04-26T18:52:23.956+0800: 4330.493: [DefNew: 281547K->1673K(314560K), 0.0079896 secs] 315632K->35766K(1013632K), 0.0081558 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:52:59.459+0800: 4366.025: [GC (Allocation Failure) 2016-04-26T18:52:59.459+0800: 4366.025: [DefNew: 281289K->1942K(314560K), 0.0099881 secs] 315382K->36041K(1013632K), 0.0101999 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:53:35.248+0800: 4401.571: [GC (Allocation Failure) 2016-04-26T18:53:35.248+0800: 4401.571: [DefNew: 281558K->1675K(314560K), 0.0080323 secs] 315657K->35779K(1013632K), 0.0081949 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:54:11.231+0800: 4437.434: [GC (Allocation Failure) 2016-04-26T18:54:11.231+0800: 4437.434: [DefNew: 281291K->1672K(314560K), 0.0107709 secs] 315395K->35785K(1013632K), 0.0109682 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:54:47.360+0800: 4473.741: [GC (Allocation Failure) 2016-04-26T18:54:47.360+0800: 4473.741: [DefNew: 281288K->1675K(314560K), 0.0077845 secs] 315401K->35796K(1013632K), 0.0079611 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:55:22.937+0800: 4509.361: [GC (Allocation Failure) 2016-04-26T18:55:22.937+0800: 4509.361: [DefNew: 281291K->1675K(314560K), 0.0082094 secs] 315412K->35802K(1013632K), 0.0083421 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:55:58.273+0800: 4544.708: [GC (Allocation Failure) 2016-04-26T18:55:58.273+0800: 4544.708: [DefNew: 281291K->1674K(314560K), 0.0100806 secs] 315418K->35807K(1013632K), 0.0102764 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:56:33.184+0800: 4579.506: [GC (Allocation Failure) 2016-04-26T18:56:33.184+0800: 4579.507: [DefNew: 281290K->1674K(314560K), 0.0111419 secs] 315423K->35814K(1013632K), 0.0113590 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:57:09.091+0800: 4615.482: [GC (Allocation Failure) 2016-04-26T18:57:09.091+0800: 4615.482: [DefNew: 281290K->1672K(314560K), 0.0139571 secs] 315430K->35818K(1013632K), 0.0141755 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:57:45.582+0800: 4651.989: [GC (Allocation Failure) 2016-04-26T18:57:45.582+0800: 4651.989: [DefNew: 281288K->1672K(314560K), 0.0089576 secs] 315434K->35824K(1013632K), 0.0091738 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:58:21.593+0800: 4688.001: [GC (Allocation Failure) 2016-04-26T18:58:21.593+0800: 4688.002: [DefNew: 281288K->1676K(314560K), 0.0158752 secs] 315440K->35835K(1013632K), 0.0160361 secs] [Times: user=0.00 sys=0.00, real=0.02 secs]
2016-04-26T18:58:56.204+0800: 4722.662: [GC (Allocation Failure) 2016-04-26T18:58:56.204+0800: 4722.662: [DefNew: 281292K->1675K(314560K), 0.0076781 secs] 315451K->35841K(1013632K), 0.0078622 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:59:31.386+0800: 4757.760: [GC (Allocation Failure) 2016-04-26T18:59:31.386+0800: 4757.760: [DefNew: 281291K->1678K(314560K), 0.0079359 secs] 315457K->35849K(1013632K), 0.0080991 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T19:00:07.775+0800: 4794.332: [GC (Allocation Failure) 2016-04-26T19:00:07.775+0800: 4794.332: [DefNew: 281294K->1676K(314560K), 0.0087637 secs] 315465K->35854K(1013632K), 0.0089416 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T19:00:44.243+0800: 4830.820: [GC (Allocation Failure) 2016-04-26T19:00:44.243+0800: 4830.820: [DefNew: 281292K->1674K(314560K), 0.0098652 secs] 315470K->35859K(1013632K), 0.0101359 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T19:01:19.025+0800: 4865.773: [GC (Allocation Failure) 2016-04-26T19:01:19.025+0800: 4865.773: [DefNew: 281290K->1686K(314560K), 0.0077619 secs] 315475K->35877K(1013632K), 0.0079340 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T19:01:47.622+0800: 4894.541: [GC (Allocation Failure) 2016-04-26T19:01:47.622+0800: 4894.541: [DefNew: 281302K->2540K(314560K), 0.0189695 secs] 315493K->36737K(1013632K), 0.0192055 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
Heap
def new generation total 314560K, used 268425K [0x04600000, 0x19b50000, 0x19b50000)
eden space 279616K, 95% used [0x04600000, 0x149a73e0, 0x15710000)
from space 34944K, 7% used [0x17930000, 0x17bab258, 0x19b50000)
to space 34944K, 0% used [0x15710000, 0x15710000, 0x17930000)
tenured generation total 699072K, used 34196K [0x19b50000, 0x44600000, 0x44600000)
the space 699072K, 4% used [0x19b50000, 0x1bcb5300, 0x1bcb5400, 0x44600000)
Metaspace used 21496K, capacity 21761K, committed 21888K, reserved 21888K
发件人: Vladimir Ozerov [mailto:vozerov@gridgain.com<ma...@gridgain.com>]
发送时间: 2016年4月25日 20:21
收件人: user@ignite.apache.org<ma...@ignite.apache.org>
主题: Re: Ignite cache data size problem.
Hi Kevin,
I performed several experiments. Essentially, I put 1M entries of the class you provided with fields initialized as follows:
for (int i = 0; i < 1_000_000; i++) {
UniqueField field = new UniqueField();
field.setDate(new Date());
field.setGuid(UUID.randomUUID().toString());
field.setMsg(String.valueOf(i));
field.setNum(BigDecimal.valueOf(ThreadLocalRandom.current().nextDouble()));
field.setOId(String.valueOf(i));
field.setOrgId(String.valueOf(i));
cache.put(i, field);
}
My results are:
1) Onheap, no indexes - about 400Mb is required to store 1M objects, or ~20Gb for 47M objects.
2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
3) Offheap, with indexes - about 400Mb offheap memory is required, or ~20Gb for all your objects.
Could you please provide more information on the error you receive? Also, you could try load entries in a batches of a well-defined size (say, 1M), and see what happens to the system. I expect you should see similar numbers.
Vladimir.
On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zh...@huawei.com>> wrote:
BTW, I created 4 + 3 nodes on two servers.
each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
kind regards,
Kevin
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite cache data size problem.
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Kevin,
Could you please re-run your case and attach Ignite logs and GC logs from
all participating servers? I would expect that this is either a kind of
out-of-memory problem, or network saturation. Also please explain how
exactly do you load data to Ignite? Do you use *DataStreamer*, or may be
*IgniteCache.loadlCache*, or *IgniteCache.put*?
Vladimir.
On Tue, Apr 26, 2016 at 3:01 PM, Zhengqingzheng <zh...@huawei.com>
wrote:
> Hi Vladimir,
>
> Thank you for your help.
>
> I tried to load 1Million records, and calculate each object’s size
> (including key and value object, where key is string type), summarized
> together and found the total memory consumption is 130Mb.
>
>
>
> Because ./ignitevisor.sh command only shows number of records, and no
> data allocation information, I don’t know how much memory has been consumed
> for each type of cache.
>
>
>
> My result is as follows:
>
> Log:
>
> Found big object: [Ljava.util.HashMap$Node;@24833164 size:
> 30.888206481933594Mb
>
> Found big object: java.util.HashMap@15819383 size: 30.88824462890625Mb
>
> Found big object: java.util.HashSet@10236420 size: 30.888259887695312Mb
>
> key size: 32388688 human readable data: 30.888259887695312Mb
>
> Found big object: [Lorg.jsr166.ConcurrentHashMap8$Node;@29556439 size:
> 129.99818420410156Mb
>
> Found big object: org.jsr166.ConcurrentHashMap8@19238297 size:
> *129.99822235107422Mb*
>
> value size: 136313016 human readable data: 129.99822235107422Mb
>
> *the whole number of record is 47Million, so the data size inside the
> cache should be 130*47=**6110Mb(around 6Gb).*
>
>
>
> However, when I try to load the whole data into the cache, I still get
> exceptions:
>
> The exception information is listed as follows:
>
> 1. ---------------------------------- exception info from client
> ----------------------------------------
>
> Before exception occurred, I have 10 nodes on two servers, server1 (48G
> ram) has 6nodes, each node is assigned with 7gb jvm heap; server2 (32gb
> ram) has 4 nodes with the same jvm settings as previous one.
>
> After exception, client stopped, and 8 node left (server1’s node remain,
> no exception occurred on this server), server2 ( 2 nodes remain, two nodes
> droped)
>
> The total number of records loaded is 37Million.
>
>
>
> [19:01:47] Topology snapshot [ver=77, servers=9, clients=1, CPUs=20,
> heap=64.0GB]
>
> [19:01:47,463][SEVERE][pub-#46%null%][GridTaskWorker] Failed to obtain
> remote job result policy for result from ComputeTask.result(..) method
> (will fail the whole task): GridJobResultImpl [job=C2 [],
> sib=GridJobSiblingImpl
> [sesId=0dbc0f15451-880a2dd1-bc95-4084-a705-4effcec5d2cd,
> jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924,
> nodeId=832bed3e-dc5d-4743-9853-127e3b516924, isJobDone=false],
> jobCtx=GridJobContextImpl
> [jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924, timeoutObj=null,
> attrs={}], node=TcpDiscoveryNode [id=832bed3e-dc5d-4743-9853-127e3b516924,
> addrs=[0:0:0:0:0:0:0:1%lo, 10.120.70.122, 127.0.0.1],
> sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /0:0:0:0:0:0:0:1%lo:47500, /
> 10.120.70.122:47500, /127.0.0.1:47500], discPort=47500, order=7,
> intOrder=7, lastExchangeTime=1461663619161, loc=false,
> ver=1.5.0#20151229-sha1:f1f8cda2, isClient=false], ex=class
> o.a.i.cluster.ClusterTopologyException: Node has left grid:
> 832bed3e-dc5d-4743-9853-127e3b516924, hasRes=true, isCancelled=false,
> isOccupied=true]
>
> class org.apache.ignite.cluster.ClusterTopologyException: Node has left
> grid: 832bed3e-dc5d-4743-9853-127e3b516924
>
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1315)
>
> at
> org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1.run(GridTaskProcessor.java:1246)
>
> at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6453)
>
> at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:788)
>
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> [19:02:16] Ignite node stopped OK [uptime=01:21:52:963]
>
>
>
> 2 ------------------------------ also, I use the recommended settings for
> the jvm and get the following log information
> ------------------------------------------------
>
> Java HotSpot(TM) Client VM (25.66-b18) for windows-x86 JRE (1.8.0_66-b18),
> built on Nov 9 2015 10:58:29 by "java_re" with MS VC++ 10.0 (VS2010)
>
> Memory: 4k page, physical 6291064k(1912832k free), swap 12580288k(6996288k
> free)
>
> CommandLine flags: -XX:GCLogFileSize=104857600
> -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824
> -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation
> -XX:-UseLargePagesIndividualAllocation
>
> 2016-04-26T17:40:15.609+0800: 4.839: [Full GC (Metadata GC Threshold)
> 2016-04-26T17:40:15.610+0800: 4.839: [Tenured: 0K->10330K(699072K),
> 0.0656011 secs] 162451K->10330K(1013632K), [Metaspace:
> 12085K->12085K(12672K)], 0.0659450 secs] [Times: user=0.06 sys=0.00,
> real=0.07 secs]
>
> 2016-04-26T17:40:24.837+0800: 14.044: [Full GC (Metadata GC Threshold)
> 2016-04-26T17:40:24.837+0800: 14.044: [Tenured: 10330K->32179K(699072K),
> 0.1810716 secs] 282679K->32179K(1013632K), [Metaspace:
> 20221K->20221K(20864K)], 0.1812825 secs] [Times: user=0.17 sys=0.00,
> real=0.18 secs]
>
> 2016-04-26T17:40:53.699+0800: 42.937: [GC (Allocation Failure)
> 2016-04-26T17:40:53.699+0800: 42.937: [DefNew: 279616K->8837K(314560K),
> 0.0242718 secs] 311795K->41016K(1013632K), 0.0244783 secs] [Times:
> user=0.03 sys=0.00, real=0.03 secs]
>
> 2016-04-26T17:41:26.411+0800: 75.516: [GC (Allocation Failure)
> 2016-04-26T17:41:26.412+0800: 75.516: [DefNew: 288453K->2685K(314560K),
> 0.0116993 secs] 320632K->34864K(1013632K), 0.0119035 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:42:01.895+0800: 110.834: [GC (Allocation Failure)
> 2016-04-26T17:42:01.895+0800: 110.834: [DefNew: 282301K->2689K(314560K),
> 0.0111799 secs] 314480K->34868K(1013632K), 0.0113629 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:42:37.354+0800: 145.818: [GC (Allocation Failure)
> 2016-04-26T17:42:37.354+0800: 145.818: [DefNew: 282305K->2695K(314560K),
> 0.0102415 secs] 314484K->34874K(1013632K), 0.0104181 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:43:13.549+0800: 181.641: [GC (Allocation Failure)
> 2016-04-26T17:43:13.549+0800: 181.641: [DefNew: 282311K->2893K(314560K),
> 0.0134864 secs] 314490K->35072K(1013632K), 0.0136970 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:43:50.453+0800: 218.129: [GC (Allocation Failure)
> 2016-04-26T17:43:50.453+0800: 218.129: [DefNew: 282509K->2708K(314560K),
> 0.0160308 secs] 314688K->34887K(1013632K), 0.0162459 secs] [Times:
> user=0.03 sys=0.00, real=0.03 secs]
>
> 2016-04-26T17:44:27.317+0800: 254.604: [GC (Allocation Failure)
> 2016-04-26T17:44:27.317+0800: 254.605: [DefNew: 282324K->2715K(314560K),
> 0.0128055 secs] 314503K->34894K(1013632K), 0.0130086 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:45:03.944+0800: 291.040: [GC (Allocation Failure)
> 2016-04-26T17:45:03.944+0800: 291.040: [DefNew: 282331K->2724K(314560K),
> 0.0109078 secs] 314510K->34903K(1013632K), 0.0110838 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:45:41.069+0800: 327.816: [GC (Allocation Failure)
> 2016-04-26T17:45:41.069+0800: 327.816: [DefNew: 282340K->2732K(314560K),
> 0.0128223 secs] 314519K->34912K(1013632K), 0.0130151 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:46:18.108+0800: 364.538: [GC (Allocation Failure)
> 2016-04-26T17:46:18.108+0800: 364.538: [DefNew: 282348K->2738K(314560K),
> 0.0123320 secs] 314528K->34917K(1013632K), 0.0125273 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:46:54.492+0800: 400.975: [GC (Allocation Failure)
> 2016-04-26T17:46:54.492+0800: 400.975: [DefNew: 282354K->2747K(314560K),
> 0.0126284 secs] 314533K->34926K(1013632K), 0.0128229 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:47:36.805+0800: 443.099: [GC (Allocation Failure)
> 2016-04-26T17:47:36.805+0800: 443.099: [DefNew: 282363K->2755K(314560K),
> 0.0137048 secs] 314542K->34934K(1013632K), 0.0139138 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:48:13.500+0800: 480.054: [GC (Allocation Failure)
> 2016-04-26T17:48:13.500+0800: 480.054: [DefNew: 282371K->2764K(314560K),
> 0.0134777 secs] 314550K->34943K(1013632K), 0.0137571 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:48:50.568+0800: 517.242: [GC (Allocation Failure)
> 2016-04-26T17:48:50.568+0800: 517.242: [DefNew: 282380K->2769K(314560K),
> 0.0121474 secs] 314559K->34948K(1013632K), 0.0125373 secs] [Times:
> user=0.00 sys=0.02, real=0.02 secs]
>
> 2016-04-26T17:49:28.506+0800: 555.103: [GC (Allocation Failure)
> 2016-04-26T17:49:28.507+0800: 555.103: [DefNew: 282385K->2777K(314560K),
> 0.0133794 secs] 314564K->34957K(1013632K), 0.0135732 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:50:05.869+0800: 592.355: [GC (Allocation Failure)
> 2016-04-26T17:50:05.869+0800: 592.355: [DefNew: 282393K->1713K(314560K),
> 0.0129486 secs] 314573K->34965K(1013632K), 0.0131366 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:50:43.336+0800: 629.663: [GC (Allocation Failure)
> 2016-04-26T17:50:43.336+0800: 629.663: [DefNew: 281329K->1688K(314560K),
> 0.0107519 secs] 314581K->34971K(1013632K), 0.0109511 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:51:20.603+0800: 666.805: [GC (Allocation Failure)
> 2016-04-26T17:51:20.604+0800: 666.806: [DefNew: 281304K->2043K(314560K),
> 0.0125329 secs] 314587K->35333K(1013632K), 0.0127661 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:51:57.981+0800: 704.195: [GC (Allocation Failure)
> 2016-04-26T17:51:57.981+0800: 704.195: [DefNew: 281659K->1688K(314560K),
> 0.0083589 secs] 314949K->34985K(1013632K), 0.0085578 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:52:35.234+0800: 741.413: [GC (Allocation Failure)
> 2016-04-26T17:52:35.234+0800: 741.413: [DefNew: 281304K->1686K(314560K),
> 0.0080466 secs] 314601K->34990K(1013632K), 0.0082145 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:53:11.734+0800: 778.046: [GC (Allocation Failure)
> 2016-04-26T17:53:11.734+0800: 778.046: [DefNew: 281302K->1689K(314560K),
> 0.0106016 secs] 314606K->35000K(1013632K), 0.0108028 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:53:48.679+0800: 814.992: [GC (Allocation Failure)
> 2016-04-26T17:53:48.679+0800: 814.992: [DefNew: 281305K->1689K(314560K),
> 0.0085276 secs] 314616K->35007K(1013632K), 0.0086721 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:54:24.734+0800: 851.078: [GC (Allocation Failure)
> 2016-04-26T17:54:24.735+0800: 851.078: [DefNew: 281305K->1685K(314560K),
> 0.0130525 secs] 314623K->35012K(1013632K), 0.0132933 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:55:01.614+0800: 888.090: [GC (Allocation Failure)
> 2016-04-26T17:55:01.614+0800: 888.090: [DefNew: 281301K->1686K(314560K),
> 0.0084343 secs] 314628K->35021K(1013632K), 0.0086352 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:55:38.728+0800: 925.129: [GC (Allocation Failure)
> 2016-04-26T17:55:38.728+0800: 925.129: [DefNew: 281302K->1684K(314560K),
> 0.0081086 secs] 314637K->35026K(1013632K), 0.0082835 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:56:15.257+0800: 961.859: [GC (Allocation Failure)
> 2016-04-26T17:56:15.257+0800: 961.859: [DefNew: 281300K->1683K(314560K),
> 0.0097599 secs] 314642K->35032K(1013632K), 0.0099144 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:56:51.417+0800: 998.123: [GC (Allocation Failure)
> 2016-04-26T17:56:51.417+0800: 998.123: [DefNew: 281299K->1681K(314560K),
> 0.0095046 secs] 314648K->35038K(1013632K), 0.0096970 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:57:27.838+0800: 1034.605: [GC (Allocation Failure)
> 2016-04-26T17:57:27.838+0800: 1034.605: [DefNew: 281297K->1681K(314560K),
> 0.0104843 secs] 314654K->35045K(1013632K), 0.0106871 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:58:04.008+0800: 1070.815: [GC (Allocation Failure)
> 2016-04-26T17:58:04.008+0800: 1070.815: [DefNew: 281297K->1680K(314560K),
> 0.0087810 secs] 314661K->35051K(1013632K), 0.0089486 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:58:40.916+0800: 1107.629: [GC (Allocation Failure)
> 2016-04-26T17:58:40.916+0800: 1107.629: [DefNew: 281296K->1679K(314560K),
> 0.0123532 secs] 314667K->35058K(1013632K), 0.0125745 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:59:17.297+0800: 1143.843: [GC (Allocation Failure)
> 2016-04-26T17:59:17.297+0800: 1143.843: [DefNew: 281295K->1682K(314560K),
> 0.0099356 secs] 314674K->35069K(1013632K), 0.0101206 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:59:54.265+0800: 1180.679: [GC (Allocation Failure)
> 2016-04-26T17:59:54.265+0800: 1180.680: [DefNew: 281298K->1678K(314560K),
> 0.0089453 secs] 314685K->35071K(1013632K), 0.0091503 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:00:31.910+0800: 1218.043: [GC (Allocation Failure)
> 2016-04-26T18:00:31.910+0800: 1218.043: [DefNew: 281294K->1677K(314560K),
> 0.0082765 secs] 314687K->35078K(1013632K), 0.0084452 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:01:07.853+0800: 1254.048: [GC (Allocation Failure)
> 2016-04-26T18:01:07.854+0800: 1254.048: [DefNew: 281293K->1678K(314560K),
> 0.0089447 secs] 314694K->35085K(1013632K), 0.0091405 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:01:44.497+0800: 1290.474: [GC (Allocation Failure)
> 2016-04-26T18:01:44.497+0800: 1290.474: [DefNew: 281294K->1678K(314560K),
> 0.0076934 secs] 314701K->35091K(1013632K), 0.0078666 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:02:27.863+0800: 1333.673: [GC (Allocation Failure)
> 2016-04-26T18:02:27.863+0800: 1333.674: [DefNew: 281294K->1674K(314560K),
> 0.0084192 secs] 314707K->35098K(1013632K), 0.0085597 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:03:04.588+0800: 1370.424: [GC (Allocation Failure)
> 2016-04-26T18:03:04.589+0800: 1370.424: [DefNew: 281290K->1677K(314560K),
> 0.0086385 secs] 314714K->35107K(1013632K), 0.0087911 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:03:41.646+0800: 1407.396: [GC (Allocation Failure)
> 2016-04-26T18:03:41.646+0800: 1407.396: [DefNew: 281293K->1674K(314560K),
> 0.0079049 secs] 314723K->35111K(1013632K), 0.0080862 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:04:18.618+0800: 1444.458: [GC (Allocation Failure)
> 2016-04-26T18:04:18.618+0800: 1444.458: [DefNew: 281290K->1674K(314560K),
> 0.0089752 secs] 314727K->35118K(1013632K), 0.0091841 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:04:55.359+0800: 1481.281: [GC (Allocation Failure)
> 2016-04-26T18:04:55.359+0800: 1481.281: [DefNew: 281290K->1674K(314560K),
> 0.0091816 secs] 314734K->35124K(1013632K), 0.0093663 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:05:31.696+0800: 1517.690: [GC (Allocation Failure)
> 2016-04-26T18:05:31.696+0800: 1517.690: [DefNew: 281290K->1680K(314560K),
> 0.0097599 secs] 314740K->35136K(1013632K), 0.0099613 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:06:08.293+0800: 1554.357: [GC (Allocation Failure)
> 2016-04-26T18:06:08.293+0800: 1554.357: [DefNew: 281296K->1681K(314560K),
> 0.0102206 secs] 314752K->35144K(1013632K), 0.0104262 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:06:44.886+0800: 1590.882: [GC (Allocation Failure)
> 2016-04-26T18:06:44.886+0800: 1590.882: [DefNew: 281297K->1680K(314560K),
> 0.0090525 secs] 314760K->35150K(1013632K), 0.0092456 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:07:20.852+0800: 1626.984: [GC (Allocation Failure)
> 2016-04-26T18:07:20.852+0800: 1626.984: [DefNew: 281296K->1682K(314560K),
> 0.0095817 secs] 314766K->35158K(1013632K), 0.0097445 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:07:57.290+0800: 1663.543: [GC (Allocation Failure)
> 2016-04-26T18:07:57.290+0800: 1663.543: [DefNew: 281298K->1682K(314560K),
> 0.0076180 secs] 314774K->35164K(1013632K), 0.0077856 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:08:34.599+0800: 1700.805: [GC (Allocation Failure)
> 2016-04-26T18:08:34.599+0800: 1700.805: [DefNew: 281298K->1680K(314560K),
> 0.0136244 secs] 314780K->35170K(1013632K), 0.0138406 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:09:10.999+0800: 1737.438: [GC (Allocation Failure)
> 2016-04-26T18:09:10.999+0800: 1737.438: [DefNew: 281296K->1681K(314560K),
> 0.0086762 secs] 314786K->35177K(1013632K), 0.0088701 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:09:47.695+0800: 1774.086: [GC (Allocation Failure)
> 2016-04-26T18:09:47.695+0800: 1774.086: [DefNew: 281297K->1679K(314560K),
> 0.0079063 secs] 314793K->35182K(1013632K), 0.0080798 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:10:24.630+0800: 1811.127: [GC (Allocation Failure)
> 2016-04-26T18:10:24.630+0800: 1811.127: [DefNew: 281295K->1680K(314560K),
> 0.0100454 secs] 314798K->35189K(1013632K), 0.0101801 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:11:00.972+0800: 1847.581: [GC (Allocation Failure)
> 2016-04-26T18:11:00.972+0800: 1847.581: [DefNew: 281296K->1679K(314560K),
> 0.0083639 secs] 314805K->35195K(1013632K), 0.0085452 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:11:37.280+0800: 1883.810: [GC (Allocation Failure)
> 2016-04-26T18:11:37.280+0800: 1883.810: [DefNew: 281295K->1679K(314560K),
> 0.0084648 secs] 314811K->35202K(1013632K), 0.0086475 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:12:13.694+0800: 1920.386: [GC (Allocation Failure)
> 2016-04-26T18:12:13.694+0800: 1920.386: [DefNew: 281295K->1677K(314560K),
> 0.0080820 secs] 314818K->35207K(1013632K), 0.0082119 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:12:49.894+0800: 1956.802: [GC (Allocation Failure)
> 2016-04-26T18:12:49.894+0800: 1956.803: [DefNew: 281293K->1677K(314560K),
> 0.0146823 secs] 314823K->35213K(1013632K), 0.0149620 secs] [Times:
> user=0.00 sys=0.02, real=0.01 secs]
>
> 2016-04-26T18:13:26.430+0800: 1993.317: [GC (Allocation Failure)
> 2016-04-26T18:13:26.430+0800: 1993.317: [DefNew: 281293K->1677K(314560K),
> 0.0088413 secs] 314829K->35220K(1013632K), 0.0090009 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:14:03.233+0800: 2030.321: [GC (Allocation Failure)
> 2016-04-26T18:14:03.233+0800: 2030.321: [DefNew: 281293K->1676K(314560K),
> 0.0078476 secs] 314836K->35225K(1013632K), 0.0079887 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:14:40.063+0800: 2067.102: [GC (Allocation Failure)
> 2016-04-26T18:14:40.063+0800: 2067.102: [DefNew: 281292K->1677K(314560K),
> 0.0092786 secs] 314841K->35232K(1013632K), 0.0094601 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:15:16.165+0800: 2103.376: [GC (Allocation Failure)
> 2016-04-26T18:15:16.165+0800: 2103.376: [DefNew: 281293K->2106K(314560K),
> 0.0135981 secs] 314848K->35675K(1013632K), 0.0138093 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:15:52.922+0800: 2139.884: [GC (Allocation Failure)
> 2016-04-26T18:15:52.922+0800: 2139.884: [DefNew: 281722K->1669K(314560K),
> 0.0123189 secs] 315291K->35244K(1013632K), 0.0125281 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:16:30.496+0800: 2177.184: [GC (Allocation Failure)
> 2016-04-26T18:16:30.496+0800: 2177.184: [DefNew: 281285K->1669K(314560K),
> 0.0089190 secs] 314860K->35251K(1013632K), 0.0090788 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:17:06.767+0800: 2213.365: [GC (Allocation Failure)
> 2016-04-26T18:17:06.767+0800: 2213.365: [DefNew: 281285K->1826K(314560K),
> 0.0079870 secs] 314867K->35415K(1013632K), 0.0081401 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:17:43.991+0800: 2250.422: [GC (Allocation Failure)
> 2016-04-26T18:17:43.991+0800: 2250.422: [DefNew: 281442K->1669K(314560K),
> 0.0115439 secs] 315031K->35264K(1013632K), 0.0117428 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:18:20.528+0800: 2286.733: [GC (Allocation Failure)
> 2016-04-26T18:18:20.528+0800: 2286.733: [DefNew: 281285K->1668K(314560K),
> 0.0082332 secs] 314880K->35270K(1013632K), 0.0084053 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:18:57.215+0800: 2323.271: [GC (Allocation Failure)
> 2016-04-26T18:18:57.215+0800: 2323.271: [DefNew: 281284K->1671K(314560K),
> 0.0077934 secs] 314886K->35277K(1013632K), 0.0079697 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:19:34.151+0800: 2359.977: [GC (Allocation Failure)
> 2016-04-26T18:19:34.151+0800: 2359.977: [DefNew: 281287K->1669K(314560K),
> 0.0078809 secs] 314893K->35283K(1013632K), 0.0080410 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:20:11.106+0800: 2396.706: [GC (Allocation Failure)
> 2016-04-26T18:20:11.106+0800: 2396.706: [DefNew: 281285K->1673K(314560K),
> 0.0092928 secs] 314899K->35293K(1013632K), 0.0094864 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:20:47.129+0800: 2432.801: [GC (Allocation Failure)
> 2016-04-26T18:20:47.129+0800: 2432.801: [DefNew: 281289K->1672K(314560K),
> 0.0089699 secs] 314909K->35298K(1013632K), 0.0091397 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:21:23.028+0800: 2468.633: [GC (Allocation Failure)
> 2016-04-26T18:21:23.028+0800: 2468.633: [DefNew: 281288K->1671K(314560K),
> 0.0124404 secs] 314914K->35303K(1013632K), 0.0128810 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:21:59.745+0800: 2505.506: [GC (Allocation Failure)
> 2016-04-26T18:21:59.745+0800: 2505.506: [DefNew: 281287K->1671K(314560K),
> 0.0080019 secs] 314919K->35309K(1013632K), 0.0082167 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:22:36.639+0800: 2542.570: [GC (Allocation Failure)
> 2016-04-26T18:22:36.639+0800: 2542.570: [DefNew: 281287K->1670K(314560K),
> 0.0082147 secs] 314925K->35315K(1013632K), 0.0083938 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:23:12.390+0800: 2578.421: [GC (Allocation Failure)
> 2016-04-26T18:23:12.390+0800: 2578.421: [DefNew: 281286K->2207K(314560K),
> 0.0114121 secs] 314931K->35857K(1013632K), 0.0116115 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:23:48.693+0800: 2615.071: [GC (Allocation Failure)
> 2016-04-26T18:23:48.693+0800: 2615.071: [DefNew: 281823K->1798K(314560K),
> 0.0104047 secs] 315473K->35455K(1013632K), 0.0105935 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:24:25.698+0800: 2652.227: [GC (Allocation Failure)
> 2016-04-26T18:24:25.698+0800: 2652.228: [DefNew: 281414K->1795K(314560K),
> 0.0102968 secs] 315071K->35459K(1013632K), 0.0104854 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:25:01.801+0800: 2688.457: [GC (Allocation Failure)
> 2016-04-26T18:25:01.801+0800: 2688.458: [DefNew: 281411K->2126K(314560K),
> 0.0090964 secs] 315075K->35795K(1013632K), 0.0093529 secs] [Times:
> user=0.00 sys=0.02, real=0.01 secs]
>
> 2016-04-26T18:25:39.055+0800: 2725.732: [GC (Allocation Failure)
> 2016-04-26T18:25:39.055+0800: 2725.732: [DefNew: 281742K->1799K(314560K),
> 0.0085785 secs] 315411K->35474K(1013632K), 0.0087168 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:26:27.232+0800: 2773.853: [GC (Allocation Failure)
> 2016-04-26T18:26:27.232+0800: 2773.853: [DefNew: 281415K->1800K(314560K),
> 0.0091224 secs] 315090K->35482K(1013632K), 0.0094059 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:27:05.583+0800: 2812.453: [GC (Allocation Failure)
> 2016-04-26T18:27:05.584+0800: 2812.453: [DefNew: 281416K->1801K(314560K),
> 0.0083206 secs] 315098K->35489K(1013632K), 0.0084969 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:27:43.412+0800: 2850.382: [GC (Allocation Failure)
> 2016-04-26T18:27:43.412+0800: 2850.382: [DefNew: 281417K->1801K(314560K),
> 0.0095576 secs] 315105K->35495K(1013632K), 0.0097616 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:28:20.067+0800: 2887.191: [GC (Allocation Failure)
> 2016-04-26T18:28:20.067+0800: 2887.191: [DefNew: 281417K->1799K(314560K),
> 0.0121244 secs] 315111K->35501K(1013632K), 0.0123463 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:28:55.431+0800: 2923.064: [GC (Allocation Failure)
> 2016-04-26T18:28:55.431+0800: 2923.064: [DefNew: 281415K->1798K(314560K),
> 0.0088000 secs] 315117K->35506K(1013632K), 0.0089771 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:29:30.761+0800: 2958.335: [GC (Allocation Failure)
> 2016-04-26T18:29:30.761+0800: 2958.335: [DefNew: 281414K->1798K(314560K),
> 0.0113492 secs] 315122K->35512K(1013632K), 0.0115526 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:30:05.795+0800: 2993.528: [GC (Allocation Failure)
> 2016-04-26T18:30:05.795+0800: 2993.528: [DefNew: 281414K->1797K(314560K),
> 0.0112305 secs] 315128K->35518K(1013632K), 0.0114227 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:30:41.648+0800: 3029.322: [GC (Allocation Failure)
> 2016-04-26T18:30:41.648+0800: 3029.322: [DefNew: 281413K->1797K(314560K),
> 0.0174930 secs] 315134K->35524K(1013632K), 0.0177282 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:31:34.467+0800: 3082.237: [GC (Allocation Failure)
> 2016-04-26T18:31:34.467+0800: 3082.237: [DefNew: 281413K->1800K(314560K),
> 0.0095434 secs] 315140K->35534K(1013632K), 0.0097186 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:32:14.021+0800: 3121.810: [GC (Allocation Failure)
> 2016-04-26T18:32:14.021+0800: 3121.810: [DefNew: 281416K->1800K(314560K),
> 0.0096049 secs] 315150K->35540K(1013632K), 0.0097856 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:33:07.874+0800: 3175.537: [GC (Allocation Failure)
> 2016-04-26T18:33:07.874+0800: 3175.537: [DefNew: 281416K->1697K(314560K),
> 0.0135850 secs] 315156K->35550K(1013632K), 0.0138194 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:33:47.621+0800: 3215.029: [GC (Allocation Failure)
> 2016-04-26T18:33:47.621+0800: 3215.029: [DefNew: 281313K->1694K(314560K),
> 0.0080256 secs] 315166K->35566K(1013632K), 0.0081854 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:34:24.160+0800: 3251.370: [GC (Allocation Failure)
> 2016-04-26T18:34:24.160+0800: 3251.371: [DefNew: 281310K->1691K(314560K),
> 0.0086257 secs] 315182K->35570K(1013632K), 0.0087810 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:34:59.507+0800: 3286.713: [GC (Allocation Failure)
> 2016-04-26T18:34:59.507+0800: 3286.713: [DefNew: 281307K->1690K(314560K),
> 0.0082689 secs] 315186K->35576K(1013632K), 0.0084583 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:35:34.974+0800: 3322.008: [GC (Allocation Failure)
> 2016-04-26T18:35:34.974+0800: 3322.008: [DefNew: 281306K->1690K(314560K),
> 0.0134400 secs] 315192K->35582K(1013632K), 0.0136509 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:36:11.007+0800: 3357.907: [GC (Allocation Failure)
> 2016-04-26T18:36:11.007+0800: 3357.907: [DefNew: 281306K->1689K(314560K),
> 0.0118568 secs] 315198K->35590K(1013632K), 0.0120744 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:36:48.227+0800: 3394.956: [GC (Allocation Failure)
> 2016-04-26T18:36:48.227+0800: 3394.956: [DefNew: 281305K->1688K(314560K),
> 0.0083156 secs] 315206K->35596K(1013632K), 0.0084603 secs] [Times:
> user=0.03 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:37:24.551+0800: 3430.991: [GC (Allocation Failure)
> 2016-04-26T18:37:24.551+0800: 3430.991: [DefNew: 281304K->1688K(314560K),
> 0.0091559 secs] 315212K->35602K(1013632K), 0.0093369 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:38:01.307+0800: 3467.736: [GC (Allocation Failure)
> 2016-04-26T18:38:01.307+0800: 3467.736: [DefNew: 281304K->1685K(314560K),
> 0.0085799 secs] 315218K->35609K(1013632K), 0.0087592 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:38:37.590+0800: 3503.935: [GC (Allocation Failure)
> 2016-04-26T18:38:37.590+0800: 3503.935: [DefNew: 281301K->1685K(314560K),
> 0.0103468 secs] 315225K->35616K(1013632K), 0.0105709 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:39:14.026+0800: 3540.397: [GC (Allocation Failure)
> 2016-04-26T18:39:14.026+0800: 3540.397: [DefNew: 281301K->1685K(314560K),
> 0.0119247 secs] 315232K->35622K(1013632K), 0.0121515 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:39:50.866+0800: 3577.377: [GC (Allocation Failure)
> 2016-04-26T18:39:50.866+0800: 3577.377: [DefNew: 281301K->1687K(314560K),
> 0.0134316 secs] 315238K->35630K(1013632K), 0.0136453 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:40:28.204+0800: 3614.598: [GC (Allocation Failure)
> 2016-04-26T18:40:28.204+0800: 3614.598: [DefNew: 281303K->1687K(314560K),
> 0.0093311 secs] 315246K->35635K(1013632K), 0.0095685 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:41:04.400+0800: 3650.792: [GC (Allocation Failure)
> 2016-04-26T18:41:04.400+0800: 3650.792: [DefNew: 281303K->1684K(314560K),
> 0.0114953 secs] 315251K->35642K(1013632K), 0.0116892 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:41:41.313+0800: 3687.569: [GC (Allocation Failure)
> 2016-04-26T18:41:41.313+0800: 3687.569: [DefNew: 281300K->1683K(314560K),
> 0.0078890 secs] 315258K->35648K(1013632K), 0.0080298 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:42:16.550+0800: 3722.774: [GC (Allocation Failure)
> 2016-04-26T18:42:16.550+0800: 3722.774: [DefNew: 281299K->1679K(314560K),
> 0.0124507 secs] 315264K->35654K(1013632K), 0.0126969 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:42:52.776+0800: 3759.034: [GC (Allocation Failure)
> 2016-04-26T18:42:52.776+0800: 3759.034: [DefNew: 281295K->1673K(314560K),
> 0.0105041 secs] 315270K->35661K(1013632K), 0.0106768 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:43:28.412+0800: 3794.638: [GC (Allocation Failure)
> 2016-04-26T18:43:28.412+0800: 3794.638: [DefNew: 281289K->1674K(314560K),
> 0.0103795 secs] 315277K->35669K(1013632K), 0.0105525 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:44:04.536+0800: 3830.831: [GC (Allocation Failure)
> 2016-04-26T18:44:04.536+0800: 3830.831: [DefNew: 281290K->1674K(314560K),
> 0.0079088 secs] 315285K->35675K(1013632K), 0.0080555 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:44:40.190+0800: 3866.449: [GC (Allocation Failure)
> 2016-04-26T18:44:40.190+0800: 3866.449: [DefNew: 281290K->1673K(314560K),
> 0.0095096 secs] 315291K->35681K(1013632K), 0.0096861 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:45:16.528+0800: 3902.765: [GC (Allocation Failure)
> 2016-04-26T18:45:16.528+0800: 3902.765: [DefNew: 281289K->1673K(314560K),
> 0.0130461 secs] 315297K->35687K(1013632K), 0.0132673 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:45:52.574+0800: 3939.001: [GC (Allocation Failure)
> 2016-04-26T18:45:52.574+0800: 3939.001: [DefNew: 281289K->1680K(314560K),
> 0.0101569 secs] 315303K->35700K(1013632K), 0.0103066 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:46:28.413+0800: 3975.223: [GC (Allocation Failure)
> 2016-04-26T18:46:28.413+0800: 3975.223: [DefNew: 281296K->1679K(314560K),
> 0.0080063 secs] 315316K->35706K(1013632K), 0.0081820 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:47:04.190+0800: 4011.234: [GC (Allocation Failure)
> 2016-04-26T18:47:04.190+0800: 4011.234: [DefNew: 281295K->1678K(314560K),
> 0.0102572 secs] 315322K->35712K(1013632K), 0.0104334 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:47:40.161+0800: 4047.120: [GC (Allocation Failure)
> 2016-04-26T18:47:40.161+0800: 4047.120: [DefNew: 281294K->1677K(314560K),
> 0.0109581 secs] 315328K->35718K(1013632K), 0.0111609 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:48:15.352+0800: 4082.428: [GC (Allocation Failure)
> 2016-04-26T18:48:15.352+0800: 4082.428: [DefNew: 281293K->1677K(314560K),
> 0.0087567 secs] 315334K->35724K(1013632K), 0.0089472 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:48:51.254+0800: 4118.407: [GC (Allocation Failure)
> 2016-04-26T18:48:51.254+0800: 4118.407: [DefNew: 281293K->1677K(314560K),
> 0.0115241 secs] 315340K->35731K(1013632K), 0.0117504 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:49:26.547+0800: 4153.705: [GC (Allocation Failure)
> 2016-04-26T18:49:26.547+0800: 4153.705: [DefNew: 281293K->1967K(314560K),
> 0.0095867 secs] 315347K->36027K(1013632K), 0.0097683 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:50:02.043+0800: 4189.206: [GC (Allocation Failure)
> 2016-04-26T18:50:02.043+0800: 4189.206: [DefNew: 281583K->1676K(314560K),
> 0.0096753 secs] 315643K->35742K(1013632K), 0.0098599 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:50:37.663+0800: 4224.505: [GC (Allocation Failure)
> 2016-04-26T18:50:37.663+0800: 4224.505: [DefNew: 281292K->1675K(314560K),
> 0.0077731 secs] 315358K->35749K(1013632K), 0.0079524 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:51:13.058+0800: 4259.780: [GC (Allocation Failure)
> 2016-04-26T18:51:13.058+0800: 4259.780: [DefNew: 281291K->1676K(314560K),
> 0.0086972 secs] 315365K->35756K(1013632K), 0.0088771 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:51:48.677+0800: 4295.269: [GC (Allocation Failure)
> 2016-04-26T18:51:48.677+0800: 4295.269: [DefNew: 281292K->1931K(314560K),
> 0.0099264 secs] 315372K->36016K(1013632K), 0.0101337 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:52:23.956+0800: 4330.493: [GC (Allocation Failure)
> 2016-04-26T18:52:23.956+0800: 4330.493: [DefNew: 281547K->1673K(314560K),
> 0.0079896 secs] 315632K->35766K(1013632K), 0.0081558 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:52:59.459+0800: 4366.025: [GC (Allocation Failure)
> 2016-04-26T18:52:59.459+0800: 4366.025: [DefNew: 281289K->1942K(314560K),
> 0.0099881 secs] 315382K->36041K(1013632K), 0.0101999 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:53:35.248+0800: 4401.571: [GC (Allocation Failure)
> 2016-04-26T18:53:35.248+0800: 4401.571: [DefNew: 281558K->1675K(314560K),
> 0.0080323 secs] 315657K->35779K(1013632K), 0.0081949 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:54:11.231+0800: 4437.434: [GC (Allocation Failure)
> 2016-04-26T18:54:11.231+0800: 4437.434: [DefNew: 281291K->1672K(314560K),
> 0.0107709 secs] 315395K->35785K(1013632K), 0.0109682 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:54:47.360+0800: 4473.741: [GC (Allocation Failure)
> 2016-04-26T18:54:47.360+0800: 4473.741: [DefNew: 281288K->1675K(314560K),
> 0.0077845 secs] 315401K->35796K(1013632K), 0.0079611 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:55:22.937+0800: 4509.361: [GC (Allocation Failure)
> 2016-04-26T18:55:22.937+0800: 4509.361: [DefNew: 281291K->1675K(314560K),
> 0.0082094 secs] 315412K->35802K(1013632K), 0.0083421 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:55:58.273+0800: 4544.708: [GC (Allocation Failure)
> 2016-04-26T18:55:58.273+0800: 4544.708: [DefNew: 281291K->1674K(314560K),
> 0.0100806 secs] 315418K->35807K(1013632K), 0.0102764 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:56:33.184+0800: 4579.506: [GC (Allocation Failure)
> 2016-04-26T18:56:33.184+0800: 4579.507: [DefNew: 281290K->1674K(314560K),
> 0.0111419 secs] 315423K->35814K(1013632K), 0.0113590 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:57:09.091+0800: 4615.482: [GC (Allocation Failure)
> 2016-04-26T18:57:09.091+0800: 4615.482: [DefNew: 281290K->1672K(314560K),
> 0.0139571 secs] 315430K->35818K(1013632K), 0.0141755 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:57:45.582+0800: 4651.989: [GC (Allocation Failure)
> 2016-04-26T18:57:45.582+0800: 4651.989: [DefNew: 281288K->1672K(314560K),
> 0.0089576 secs] 315434K->35824K(1013632K), 0.0091738 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:58:21.593+0800: 4688.001: [GC (Allocation Failure)
> 2016-04-26T18:58:21.593+0800: 4688.002: [DefNew: 281288K->1676K(314560K),
> 0.0158752 secs] 315440K->35835K(1013632K), 0.0160361 secs] [Times:
> user=0.00 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:58:56.204+0800: 4722.662: [GC (Allocation Failure)
> 2016-04-26T18:58:56.204+0800: 4722.662: [DefNew: 281292K->1675K(314560K),
> 0.0076781 secs] 315451K->35841K(1013632K), 0.0078622 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:59:31.386+0800: 4757.760: [GC (Allocation Failure)
> 2016-04-26T18:59:31.386+0800: 4757.760: [DefNew: 281291K->1678K(314560K),
> 0.0079359 secs] 315457K->35849K(1013632K), 0.0080991 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T19:00:07.775+0800: 4794.332: [GC (Allocation Failure)
> 2016-04-26T19:00:07.775+0800: 4794.332: [DefNew: 281294K->1676K(314560K),
> 0.0087637 secs] 315465K->35854K(1013632K), 0.0089416 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T19:00:44.243+0800: 4830.820: [GC (Allocation Failure)
> 2016-04-26T19:00:44.243+0800: 4830.820: [DefNew: 281292K->1674K(314560K),
> 0.0098652 secs] 315470K->35859K(1013632K), 0.0101359 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T19:01:19.025+0800: 4865.773: [GC (Allocation Failure)
> 2016-04-26T19:01:19.025+0800: 4865.773: [DefNew: 281290K->1686K(314560K),
> 0.0077619 secs] 315475K->35877K(1013632K), 0.0079340 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T19:01:47.622+0800: 4894.541: [GC (Allocation Failure)
> 2016-04-26T19:01:47.622+0800: 4894.541: [DefNew: 281302K->2540K(314560K),
> 0.0189695 secs] 315493K->36737K(1013632K), 0.0192055 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> Heap
>
> def new generation total 314560K, used 268425K [0x04600000, 0x19b50000,
> 0x19b50000)
>
> eden space 279616K, 95% used [0x04600000, 0x149a73e0, 0x15710000)
>
> from space 34944K, 7% used [0x17930000, 0x17bab258, 0x19b50000)
>
> to space 34944K, 0% used [0x15710000, 0x15710000, 0x17930000)
>
> tenured generation total 699072K, used 34196K [0x19b50000, 0x44600000,
> 0x44600000)
>
> the space 699072K, 4% used [0x19b50000, 0x1bcb5300, 0x1bcb5400,
> 0x44600000)
>
> Metaspace used 21496K, capacity 21761K, committed 21888K, reserved
> 21888K
>
>
>
>
>
> *发件人:* Vladimir Ozerov [mailto:vozerov@gridgain.com]
> *发送时间:* 2016年4月25日 20:21
> *收件人:* user@ignite.apache.org
> *主题:* Re: Ignite cache data size problem.
>
>
>
> Hi Kevin,
>
>
>
> I performed several experiments. Essentially, I put 1M entries of the
> class you provided with fields initialized as follows:
>
>
>
> *for *(*int *i = 0; i < 1_000_000; i++) {
> UniqueField field = *new *UniqueField();
>
> field.setDate(*new *Date());
> field.setGuid(UUID.*randomUUID*().toString());
> field.setMsg(String.*valueOf*(i));
> field.setNum(BigDecimal.*valueOf*(ThreadLocalRandom.*current*().nextDouble()));
> field.setOId(String.*valueOf*(i));
> field.setOrgId(String.*valueOf*(i));
>
> cache.put(i, field);
> }
>
>
>
> My results are:
>
> 1) Onheap, no indexes - about 400Mb is required to store 1M objects, or
> ~20Gb for 47M objects.
>
> 2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
>
> 3) Offheap, with indexes - about 400Mb offheap memory is required, or
> ~20Gb for all your objects.
>
>
>
> Could you please provide more information on the error you receive? Also,
> you could try load entries in a batches of a well-defined size (say, 1M),
> and see what happens to the system. I expect you should see similar numbers.
>
>
>
> Vladimir.
>
>
>
> On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zh...@huawei.com>
> wrote:
>
> BTW, I created 4 + 3 nodes on two servers.
> each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
>
> kind regards,
> Kevin
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
re: Ignite cache data size problem.
Posted by Zhengqingzheng <zh...@huawei.com>.
Hi Vladimir,
Thank you for your help.
I tried to load 1Million records, and calculate each object’s size (including key and value object, where key is string type), summarized together and found the total memory consumption is 130Mb.
Because ./ignitevisor.sh command only shows number of records, and no data allocation information, I don’t know how much memory has been consumed for each type of cache.
My result is as follows:
Log:
Found big object: [Ljava.util.HashMap$Node;@24833164 size: 30.888206481933594Mb
Found big object: java.util.HashMap@15819383 size: 30.88824462890625Mb
Found big object: java.util.HashSet@10236420 size: 30.888259887695312Mb
key size: 32388688 human readable data: 30.888259887695312Mb
Found big object: [Lorg.jsr166.ConcurrentHashMap8$Node;@29556439 size: 129.99818420410156Mb
Found big object: org.jsr166.ConcurrentHashMap8@19238297 size: 129.99822235107422Mb
value size: 136313016 human readable data: 129.99822235107422Mb
the whole number of record is 47Million, so the data size inside the cache should be 130*47=6110Mb(around 6Gb).
However, when I try to load the whole data into the cache, I still get exceptions:
The exception information is listed as follows:
1. ---------------------------------- exception info from client ----------------------------------------
Before exception occurred, I have 10 nodes on two servers, server1 (48G ram) has 6nodes, each node is assigned with 7gb jvm heap; server2 (32gb ram) has 4 nodes with the same jvm settings as previous one.
After exception, client stopped, and 8 node left (server1’s node remain, no exception occurred on this server), server2 ( 2 nodes remain, two nodes droped)
The total number of records loaded is 37Million.
[19:01:47] Topology snapshot [ver=77, servers=9, clients=1, CPUs=20, heap=64.0GB]
[19:01:47,463][SEVERE][pub-#46%null%][GridTaskWorker] Failed to obtain remote job result policy for result from ComputeTask.result(..) method (will fail the whole task): GridJobResultImpl [job=C2 [], sib=GridJobSiblingImpl [sesId=0dbc0f15451-880a2dd1-bc95-4084-a705-4effcec5d2cd, jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924, nodeId=832bed3e-dc5d-4743-9853-127e3b516924, isJobDone=false], jobCtx=GridJobContextImpl [jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924, timeoutObj=null, attrs={}], node=TcpDiscoveryNode [id=832bed3e-dc5d-4743-9853-127e3b516924, addrs=[0:0:0:0:0:0:0:1%lo, 10.120.70.122, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /0:0:0:0:0:0:0:1%lo:47500, /10.120.70.122:47500, /127.0.0.1:47500], discPort=47500, order=7, intOrder=7, lastExchangeTime=1461663619161, loc=false, ver=1.5.0#20151229-sha1:f1f8cda2, isClient=false], ex=class o.a.i.cluster.ClusterTopologyException: Node has left grid: 832bed3e-dc5d-4743-9853-127e3b516924, hasRes=true, isCancelled=false, isOccupied=true]
class org.apache.ignite.cluster.ClusterTopologyException: Node has left grid: 832bed3e-dc5d-4743-9853-127e3b516924
at org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1315)
at org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1.run(GridTaskProcessor.java:1246)
at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6453)
at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:788)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[19:02:16] Ignite node stopped OK [uptime=01:21:52:963]
2 ------------------------------ also, I use the recommended settings for the jvm and get the following log information ------------------------------------------------
Java HotSpot(TM) Client VM (25.66-b18) for windows-x86 JRE (1.8.0_66-b18), built on Nov 9 2015 10:58:29 by "java_re" with MS VC++ 10.0 (VS2010)
Memory: 4k page, physical 6291064k(1912832k free), swap 12580288k(6996288k free)
CommandLine flags: -XX:GCLogFileSize=104857600 -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:-UseLargePagesIndividualAllocation
2016-04-26T17:40:15.609+0800: 4.839: [Full GC (Metadata GC Threshold) 2016-04-26T17:40:15.610+0800: 4.839: [Tenured: 0K->10330K(699072K), 0.0656011 secs] 162451K->10330K(1013632K), [Metaspace: 12085K->12085K(12672K)], 0.0659450 secs] [Times: user=0.06 sys=0.00, real=0.07 secs]
2016-04-26T17:40:24.837+0800: 14.044: [Full GC (Metadata GC Threshold) 2016-04-26T17:40:24.837+0800: 14.044: [Tenured: 10330K->32179K(699072K), 0.1810716 secs] 282679K->32179K(1013632K), [Metaspace: 20221K->20221K(20864K)], 0.1812825 secs] [Times: user=0.17 sys=0.00, real=0.18 secs]
2016-04-26T17:40:53.699+0800: 42.937: [GC (Allocation Failure) 2016-04-26T17:40:53.699+0800: 42.937: [DefNew: 279616K->8837K(314560K), 0.0242718 secs] 311795K->41016K(1013632K), 0.0244783 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2016-04-26T17:41:26.411+0800: 75.516: [GC (Allocation Failure) 2016-04-26T17:41:26.412+0800: 75.516: [DefNew: 288453K->2685K(314560K), 0.0116993 secs] 320632K->34864K(1013632K), 0.0119035 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:42:01.895+0800: 110.834: [GC (Allocation Failure) 2016-04-26T17:42:01.895+0800: 110.834: [DefNew: 282301K->2689K(314560K), 0.0111799 secs] 314480K->34868K(1013632K), 0.0113629 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:42:37.354+0800: 145.818: [GC (Allocation Failure) 2016-04-26T17:42:37.354+0800: 145.818: [DefNew: 282305K->2695K(314560K), 0.0102415 secs] 314484K->34874K(1013632K), 0.0104181 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:43:13.549+0800: 181.641: [GC (Allocation Failure) 2016-04-26T17:43:13.549+0800: 181.641: [DefNew: 282311K->2893K(314560K), 0.0134864 secs] 314490K->35072K(1013632K), 0.0136970 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:43:50.453+0800: 218.129: [GC (Allocation Failure) 2016-04-26T17:43:50.453+0800: 218.129: [DefNew: 282509K->2708K(314560K), 0.0160308 secs] 314688K->34887K(1013632K), 0.0162459 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2016-04-26T17:44:27.317+0800: 254.604: [GC (Allocation Failure) 2016-04-26T17:44:27.317+0800: 254.605: [DefNew: 282324K->2715K(314560K), 0.0128055 secs] 314503K->34894K(1013632K), 0.0130086 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:45:03.944+0800: 291.040: [GC (Allocation Failure) 2016-04-26T17:45:03.944+0800: 291.040: [DefNew: 282331K->2724K(314560K), 0.0109078 secs] 314510K->34903K(1013632K), 0.0110838 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:45:41.069+0800: 327.816: [GC (Allocation Failure) 2016-04-26T17:45:41.069+0800: 327.816: [DefNew: 282340K->2732K(314560K), 0.0128223 secs] 314519K->34912K(1013632K), 0.0130151 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:46:18.108+0800: 364.538: [GC (Allocation Failure) 2016-04-26T17:46:18.108+0800: 364.538: [DefNew: 282348K->2738K(314560K), 0.0123320 secs] 314528K->34917K(1013632K), 0.0125273 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:46:54.492+0800: 400.975: [GC (Allocation Failure) 2016-04-26T17:46:54.492+0800: 400.975: [DefNew: 282354K->2747K(314560K), 0.0126284 secs] 314533K->34926K(1013632K), 0.0128229 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:47:36.805+0800: 443.099: [GC (Allocation Failure) 2016-04-26T17:47:36.805+0800: 443.099: [DefNew: 282363K->2755K(314560K), 0.0137048 secs] 314542K->34934K(1013632K), 0.0139138 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:48:13.500+0800: 480.054: [GC (Allocation Failure) 2016-04-26T17:48:13.500+0800: 480.054: [DefNew: 282371K->2764K(314560K), 0.0134777 secs] 314550K->34943K(1013632K), 0.0137571 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:48:50.568+0800: 517.242: [GC (Allocation Failure) 2016-04-26T17:48:50.568+0800: 517.242: [DefNew: 282380K->2769K(314560K), 0.0121474 secs] 314559K->34948K(1013632K), 0.0125373 secs] [Times: user=0.00 sys=0.02, real=0.02 secs]
2016-04-26T17:49:28.506+0800: 555.103: [GC (Allocation Failure) 2016-04-26T17:49:28.507+0800: 555.103: [DefNew: 282385K->2777K(314560K), 0.0133794 secs] 314564K->34957K(1013632K), 0.0135732 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:50:05.869+0800: 592.355: [GC (Allocation Failure) 2016-04-26T17:50:05.869+0800: 592.355: [DefNew: 282393K->1713K(314560K), 0.0129486 secs] 314573K->34965K(1013632K), 0.0131366 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:50:43.336+0800: 629.663: [GC (Allocation Failure) 2016-04-26T17:50:43.336+0800: 629.663: [DefNew: 281329K->1688K(314560K), 0.0107519 secs] 314581K->34971K(1013632K), 0.0109511 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:51:20.603+0800: 666.805: [GC (Allocation Failure) 2016-04-26T17:51:20.604+0800: 666.806: [DefNew: 281304K->2043K(314560K), 0.0125329 secs] 314587K->35333K(1013632K), 0.0127661 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:51:57.981+0800: 704.195: [GC (Allocation Failure) 2016-04-26T17:51:57.981+0800: 704.195: [DefNew: 281659K->1688K(314560K), 0.0083589 secs] 314949K->34985K(1013632K), 0.0085578 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:52:35.234+0800: 741.413: [GC (Allocation Failure) 2016-04-26T17:52:35.234+0800: 741.413: [DefNew: 281304K->1686K(314560K), 0.0080466 secs] 314601K->34990K(1013632K), 0.0082145 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:53:11.734+0800: 778.046: [GC (Allocation Failure) 2016-04-26T17:53:11.734+0800: 778.046: [DefNew: 281302K->1689K(314560K), 0.0106016 secs] 314606K->35000K(1013632K), 0.0108028 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:53:48.679+0800: 814.992: [GC (Allocation Failure) 2016-04-26T17:53:48.679+0800: 814.992: [DefNew: 281305K->1689K(314560K), 0.0085276 secs] 314616K->35007K(1013632K), 0.0086721 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T17:54:24.734+0800: 851.078: [GC (Allocation Failure) 2016-04-26T17:54:24.735+0800: 851.078: [DefNew: 281305K->1685K(314560K), 0.0130525 secs] 314623K->35012K(1013632K), 0.0132933 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:55:01.614+0800: 888.090: [GC (Allocation Failure) 2016-04-26T17:55:01.614+0800: 888.090: [DefNew: 281301K->1686K(314560K), 0.0084343 secs] 314628K->35021K(1013632K), 0.0086352 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:55:38.728+0800: 925.129: [GC (Allocation Failure) 2016-04-26T17:55:38.728+0800: 925.129: [DefNew: 281302K->1684K(314560K), 0.0081086 secs] 314637K->35026K(1013632K), 0.0082835 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:56:15.257+0800: 961.859: [GC (Allocation Failure) 2016-04-26T17:56:15.257+0800: 961.859: [DefNew: 281300K->1683K(314560K), 0.0097599 secs] 314642K->35032K(1013632K), 0.0099144 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:56:51.417+0800: 998.123: [GC (Allocation Failure) 2016-04-26T17:56:51.417+0800: 998.123: [DefNew: 281299K->1681K(314560K), 0.0095046 secs] 314648K->35038K(1013632K), 0.0096970 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T17:57:27.838+0800: 1034.605: [GC (Allocation Failure) 2016-04-26T17:57:27.838+0800: 1034.605: [DefNew: 281297K->1681K(314560K), 0.0104843 secs] 314654K->35045K(1013632K), 0.0106871 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:58:04.008+0800: 1070.815: [GC (Allocation Failure) 2016-04-26T17:58:04.008+0800: 1070.815: [DefNew: 281297K->1680K(314560K), 0.0087810 secs] 314661K->35051K(1013632K), 0.0089486 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T17:58:40.916+0800: 1107.629: [GC (Allocation Failure) 2016-04-26T17:58:40.916+0800: 1107.629: [DefNew: 281296K->1679K(314560K), 0.0123532 secs] 314667K->35058K(1013632K), 0.0125745 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T17:59:17.297+0800: 1143.843: [GC (Allocation Failure) 2016-04-26T17:59:17.297+0800: 1143.843: [DefNew: 281295K->1682K(314560K), 0.0099356 secs] 314674K->35069K(1013632K), 0.0101206 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T17:59:54.265+0800: 1180.679: [GC (Allocation Failure) 2016-04-26T17:59:54.265+0800: 1180.680: [DefNew: 281298K->1678K(314560K), 0.0089453 secs] 314685K->35071K(1013632K), 0.0091503 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:00:31.910+0800: 1218.043: [GC (Allocation Failure) 2016-04-26T18:00:31.910+0800: 1218.043: [DefNew: 281294K->1677K(314560K), 0.0082765 secs] 314687K->35078K(1013632K), 0.0084452 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:01:07.853+0800: 1254.048: [GC (Allocation Failure) 2016-04-26T18:01:07.854+0800: 1254.048: [DefNew: 281293K->1678K(314560K), 0.0089447 secs] 314694K->35085K(1013632K), 0.0091405 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:01:44.497+0800: 1290.474: [GC (Allocation Failure) 2016-04-26T18:01:44.497+0800: 1290.474: [DefNew: 281294K->1678K(314560K), 0.0076934 secs] 314701K->35091K(1013632K), 0.0078666 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:02:27.863+0800: 1333.673: [GC (Allocation Failure) 2016-04-26T18:02:27.863+0800: 1333.674: [DefNew: 281294K->1674K(314560K), 0.0084192 secs] 314707K->35098K(1013632K), 0.0085597 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:03:04.588+0800: 1370.424: [GC (Allocation Failure) 2016-04-26T18:03:04.589+0800: 1370.424: [DefNew: 281290K->1677K(314560K), 0.0086385 secs] 314714K->35107K(1013632K), 0.0087911 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:03:41.646+0800: 1407.396: [GC (Allocation Failure) 2016-04-26T18:03:41.646+0800: 1407.396: [DefNew: 281293K->1674K(314560K), 0.0079049 secs] 314723K->35111K(1013632K), 0.0080862 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:04:18.618+0800: 1444.458: [GC (Allocation Failure) 2016-04-26T18:04:18.618+0800: 1444.458: [DefNew: 281290K->1674K(314560K), 0.0089752 secs] 314727K->35118K(1013632K), 0.0091841 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:04:55.359+0800: 1481.281: [GC (Allocation Failure) 2016-04-26T18:04:55.359+0800: 1481.281: [DefNew: 281290K->1674K(314560K), 0.0091816 secs] 314734K->35124K(1013632K), 0.0093663 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:05:31.696+0800: 1517.690: [GC (Allocation Failure) 2016-04-26T18:05:31.696+0800: 1517.690: [DefNew: 281290K->1680K(314560K), 0.0097599 secs] 314740K->35136K(1013632K), 0.0099613 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:06:08.293+0800: 1554.357: [GC (Allocation Failure) 2016-04-26T18:06:08.293+0800: 1554.357: [DefNew: 281296K->1681K(314560K), 0.0102206 secs] 314752K->35144K(1013632K), 0.0104262 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:06:44.886+0800: 1590.882: [GC (Allocation Failure) 2016-04-26T18:06:44.886+0800: 1590.882: [DefNew: 281297K->1680K(314560K), 0.0090525 secs] 314760K->35150K(1013632K), 0.0092456 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:07:20.852+0800: 1626.984: [GC (Allocation Failure) 2016-04-26T18:07:20.852+0800: 1626.984: [DefNew: 281296K->1682K(314560K), 0.0095817 secs] 314766K->35158K(1013632K), 0.0097445 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:07:57.290+0800: 1663.543: [GC (Allocation Failure) 2016-04-26T18:07:57.290+0800: 1663.543: [DefNew: 281298K->1682K(314560K), 0.0076180 secs] 314774K->35164K(1013632K), 0.0077856 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:08:34.599+0800: 1700.805: [GC (Allocation Failure) 2016-04-26T18:08:34.599+0800: 1700.805: [DefNew: 281298K->1680K(314560K), 0.0136244 secs] 314780K->35170K(1013632K), 0.0138406 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:09:10.999+0800: 1737.438: [GC (Allocation Failure) 2016-04-26T18:09:10.999+0800: 1737.438: [DefNew: 281296K->1681K(314560K), 0.0086762 secs] 314786K->35177K(1013632K), 0.0088701 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:09:47.695+0800: 1774.086: [GC (Allocation Failure) 2016-04-26T18:09:47.695+0800: 1774.086: [DefNew: 281297K->1679K(314560K), 0.0079063 secs] 314793K->35182K(1013632K), 0.0080798 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:10:24.630+0800: 1811.127: [GC (Allocation Failure) 2016-04-26T18:10:24.630+0800: 1811.127: [DefNew: 281295K->1680K(314560K), 0.0100454 secs] 314798K->35189K(1013632K), 0.0101801 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:11:00.972+0800: 1847.581: [GC (Allocation Failure) 2016-04-26T18:11:00.972+0800: 1847.581: [DefNew: 281296K->1679K(314560K), 0.0083639 secs] 314805K->35195K(1013632K), 0.0085452 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:11:37.280+0800: 1883.810: [GC (Allocation Failure) 2016-04-26T18:11:37.280+0800: 1883.810: [DefNew: 281295K->1679K(314560K), 0.0084648 secs] 314811K->35202K(1013632K), 0.0086475 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:12:13.694+0800: 1920.386: [GC (Allocation Failure) 2016-04-26T18:12:13.694+0800: 1920.386: [DefNew: 281295K->1677K(314560K), 0.0080820 secs] 314818K->35207K(1013632K), 0.0082119 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:12:49.894+0800: 1956.802: [GC (Allocation Failure) 2016-04-26T18:12:49.894+0800: 1956.803: [DefNew: 281293K->1677K(314560K), 0.0146823 secs] 314823K->35213K(1013632K), 0.0149620 secs] [Times: user=0.00 sys=0.02, real=0.01 secs]
2016-04-26T18:13:26.430+0800: 1993.317: [GC (Allocation Failure) 2016-04-26T18:13:26.430+0800: 1993.317: [DefNew: 281293K->1677K(314560K), 0.0088413 secs] 314829K->35220K(1013632K), 0.0090009 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:14:03.233+0800: 2030.321: [GC (Allocation Failure) 2016-04-26T18:14:03.233+0800: 2030.321: [DefNew: 281293K->1676K(314560K), 0.0078476 secs] 314836K->35225K(1013632K), 0.0079887 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:14:40.063+0800: 2067.102: [GC (Allocation Failure) 2016-04-26T18:14:40.063+0800: 2067.102: [DefNew: 281292K->1677K(314560K), 0.0092786 secs] 314841K->35232K(1013632K), 0.0094601 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:15:16.165+0800: 2103.376: [GC (Allocation Failure) 2016-04-26T18:15:16.165+0800: 2103.376: [DefNew: 281293K->2106K(314560K), 0.0135981 secs] 314848K->35675K(1013632K), 0.0138093 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:15:52.922+0800: 2139.884: [GC (Allocation Failure) 2016-04-26T18:15:52.922+0800: 2139.884: [DefNew: 281722K->1669K(314560K), 0.0123189 secs] 315291K->35244K(1013632K), 0.0125281 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:16:30.496+0800: 2177.184: [GC (Allocation Failure) 2016-04-26T18:16:30.496+0800: 2177.184: [DefNew: 281285K->1669K(314560K), 0.0089190 secs] 314860K->35251K(1013632K), 0.0090788 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:17:06.767+0800: 2213.365: [GC (Allocation Failure) 2016-04-26T18:17:06.767+0800: 2213.365: [DefNew: 281285K->1826K(314560K), 0.0079870 secs] 314867K->35415K(1013632K), 0.0081401 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:17:43.991+0800: 2250.422: [GC (Allocation Failure) 2016-04-26T18:17:43.991+0800: 2250.422: [DefNew: 281442K->1669K(314560K), 0.0115439 secs] 315031K->35264K(1013632K), 0.0117428 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:18:20.528+0800: 2286.733: [GC (Allocation Failure) 2016-04-26T18:18:20.528+0800: 2286.733: [DefNew: 281285K->1668K(314560K), 0.0082332 secs] 314880K->35270K(1013632K), 0.0084053 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:18:57.215+0800: 2323.271: [GC (Allocation Failure) 2016-04-26T18:18:57.215+0800: 2323.271: [DefNew: 281284K->1671K(314560K), 0.0077934 secs] 314886K->35277K(1013632K), 0.0079697 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:19:34.151+0800: 2359.977: [GC (Allocation Failure) 2016-04-26T18:19:34.151+0800: 2359.977: [DefNew: 281287K->1669K(314560K), 0.0078809 secs] 314893K->35283K(1013632K), 0.0080410 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:20:11.106+0800: 2396.706: [GC (Allocation Failure) 2016-04-26T18:20:11.106+0800: 2396.706: [DefNew: 281285K->1673K(314560K), 0.0092928 secs] 314899K->35293K(1013632K), 0.0094864 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:20:47.129+0800: 2432.801: [GC (Allocation Failure) 2016-04-26T18:20:47.129+0800: 2432.801: [DefNew: 281289K->1672K(314560K), 0.0089699 secs] 314909K->35298K(1013632K), 0.0091397 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:21:23.028+0800: 2468.633: [GC (Allocation Failure) 2016-04-26T18:21:23.028+0800: 2468.633: [DefNew: 281288K->1671K(314560K), 0.0124404 secs] 314914K->35303K(1013632K), 0.0128810 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:21:59.745+0800: 2505.506: [GC (Allocation Failure) 2016-04-26T18:21:59.745+0800: 2505.506: [DefNew: 281287K->1671K(314560K), 0.0080019 secs] 314919K->35309K(1013632K), 0.0082167 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:22:36.639+0800: 2542.570: [GC (Allocation Failure) 2016-04-26T18:22:36.639+0800: 2542.570: [DefNew: 281287K->1670K(314560K), 0.0082147 secs] 314925K->35315K(1013632K), 0.0083938 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:23:12.390+0800: 2578.421: [GC (Allocation Failure) 2016-04-26T18:23:12.390+0800: 2578.421: [DefNew: 281286K->2207K(314560K), 0.0114121 secs] 314931K->35857K(1013632K), 0.0116115 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:23:48.693+0800: 2615.071: [GC (Allocation Failure) 2016-04-26T18:23:48.693+0800: 2615.071: [DefNew: 281823K->1798K(314560K), 0.0104047 secs] 315473K->35455K(1013632K), 0.0105935 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:24:25.698+0800: 2652.227: [GC (Allocation Failure) 2016-04-26T18:24:25.698+0800: 2652.228: [DefNew: 281414K->1795K(314560K), 0.0102968 secs] 315071K->35459K(1013632K), 0.0104854 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:25:01.801+0800: 2688.457: [GC (Allocation Failure) 2016-04-26T18:25:01.801+0800: 2688.458: [DefNew: 281411K->2126K(314560K), 0.0090964 secs] 315075K->35795K(1013632K), 0.0093529 secs] [Times: user=0.00 sys=0.02, real=0.01 secs]
2016-04-26T18:25:39.055+0800: 2725.732: [GC (Allocation Failure) 2016-04-26T18:25:39.055+0800: 2725.732: [DefNew: 281742K->1799K(314560K), 0.0085785 secs] 315411K->35474K(1013632K), 0.0087168 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:26:27.232+0800: 2773.853: [GC (Allocation Failure) 2016-04-26T18:26:27.232+0800: 2773.853: [DefNew: 281415K->1800K(314560K), 0.0091224 secs] 315090K->35482K(1013632K), 0.0094059 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:27:05.583+0800: 2812.453: [GC (Allocation Failure) 2016-04-26T18:27:05.584+0800: 2812.453: [DefNew: 281416K->1801K(314560K), 0.0083206 secs] 315098K->35489K(1013632K), 0.0084969 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:27:43.412+0800: 2850.382: [GC (Allocation Failure) 2016-04-26T18:27:43.412+0800: 2850.382: [DefNew: 281417K->1801K(314560K), 0.0095576 secs] 315105K->35495K(1013632K), 0.0097616 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:28:20.067+0800: 2887.191: [GC (Allocation Failure) 2016-04-26T18:28:20.067+0800: 2887.191: [DefNew: 281417K->1799K(314560K), 0.0121244 secs] 315111K->35501K(1013632K), 0.0123463 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:28:55.431+0800: 2923.064: [GC (Allocation Failure) 2016-04-26T18:28:55.431+0800: 2923.064: [DefNew: 281415K->1798K(314560K), 0.0088000 secs] 315117K->35506K(1013632K), 0.0089771 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:29:30.761+0800: 2958.335: [GC (Allocation Failure) 2016-04-26T18:29:30.761+0800: 2958.335: [DefNew: 281414K->1798K(314560K), 0.0113492 secs] 315122K->35512K(1013632K), 0.0115526 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:30:05.795+0800: 2993.528: [GC (Allocation Failure) 2016-04-26T18:30:05.795+0800: 2993.528: [DefNew: 281414K->1797K(314560K), 0.0112305 secs] 315128K->35518K(1013632K), 0.0114227 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:30:41.648+0800: 3029.322: [GC (Allocation Failure) 2016-04-26T18:30:41.648+0800: 3029.322: [DefNew: 281413K->1797K(314560K), 0.0174930 secs] 315134K->35524K(1013632K), 0.0177282 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:31:34.467+0800: 3082.237: [GC (Allocation Failure) 2016-04-26T18:31:34.467+0800: 3082.237: [DefNew: 281413K->1800K(314560K), 0.0095434 secs] 315140K->35534K(1013632K), 0.0097186 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:32:14.021+0800: 3121.810: [GC (Allocation Failure) 2016-04-26T18:32:14.021+0800: 3121.810: [DefNew: 281416K->1800K(314560K), 0.0096049 secs] 315150K->35540K(1013632K), 0.0097856 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:33:07.874+0800: 3175.537: [GC (Allocation Failure) 2016-04-26T18:33:07.874+0800: 3175.537: [DefNew: 281416K->1697K(314560K), 0.0135850 secs] 315156K->35550K(1013632K), 0.0138194 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:33:47.621+0800: 3215.029: [GC (Allocation Failure) 2016-04-26T18:33:47.621+0800: 3215.029: [DefNew: 281313K->1694K(314560K), 0.0080256 secs] 315166K->35566K(1013632K), 0.0081854 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:34:24.160+0800: 3251.370: [GC (Allocation Failure) 2016-04-26T18:34:24.160+0800: 3251.371: [DefNew: 281310K->1691K(314560K), 0.0086257 secs] 315182K->35570K(1013632K), 0.0087810 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:34:59.507+0800: 3286.713: [GC (Allocation Failure) 2016-04-26T18:34:59.507+0800: 3286.713: [DefNew: 281307K->1690K(314560K), 0.0082689 secs] 315186K->35576K(1013632K), 0.0084583 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:35:34.974+0800: 3322.008: [GC (Allocation Failure) 2016-04-26T18:35:34.974+0800: 3322.008: [DefNew: 281306K->1690K(314560K), 0.0134400 secs] 315192K->35582K(1013632K), 0.0136509 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:36:11.007+0800: 3357.907: [GC (Allocation Failure) 2016-04-26T18:36:11.007+0800: 3357.907: [DefNew: 281306K->1689K(314560K), 0.0118568 secs] 315198K->35590K(1013632K), 0.0120744 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:36:48.227+0800: 3394.956: [GC (Allocation Failure) 2016-04-26T18:36:48.227+0800: 3394.956: [DefNew: 281305K->1688K(314560K), 0.0083156 secs] 315206K->35596K(1013632K), 0.0084603 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
2016-04-26T18:37:24.551+0800: 3430.991: [GC (Allocation Failure) 2016-04-26T18:37:24.551+0800: 3430.991: [DefNew: 281304K->1688K(314560K), 0.0091559 secs] 315212K->35602K(1013632K), 0.0093369 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:38:01.307+0800: 3467.736: [GC (Allocation Failure) 2016-04-26T18:38:01.307+0800: 3467.736: [DefNew: 281304K->1685K(314560K), 0.0085799 secs] 315218K->35609K(1013632K), 0.0087592 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:38:37.590+0800: 3503.935: [GC (Allocation Failure) 2016-04-26T18:38:37.590+0800: 3503.935: [DefNew: 281301K->1685K(314560K), 0.0103468 secs] 315225K->35616K(1013632K), 0.0105709 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:39:14.026+0800: 3540.397: [GC (Allocation Failure) 2016-04-26T18:39:14.026+0800: 3540.397: [DefNew: 281301K->1685K(314560K), 0.0119247 secs] 315232K->35622K(1013632K), 0.0121515 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:39:50.866+0800: 3577.377: [GC (Allocation Failure) 2016-04-26T18:39:50.866+0800: 3577.377: [DefNew: 281301K->1687K(314560K), 0.0134316 secs] 315238K->35630K(1013632K), 0.0136453 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:40:28.204+0800: 3614.598: [GC (Allocation Failure) 2016-04-26T18:40:28.204+0800: 3614.598: [DefNew: 281303K->1687K(314560K), 0.0093311 secs] 315246K->35635K(1013632K), 0.0095685 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:41:04.400+0800: 3650.792: [GC (Allocation Failure) 2016-04-26T18:41:04.400+0800: 3650.792: [DefNew: 281303K->1684K(314560K), 0.0114953 secs] 315251K->35642K(1013632K), 0.0116892 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:41:41.313+0800: 3687.569: [GC (Allocation Failure) 2016-04-26T18:41:41.313+0800: 3687.569: [DefNew: 281300K->1683K(314560K), 0.0078890 secs] 315258K->35648K(1013632K), 0.0080298 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:42:16.550+0800: 3722.774: [GC (Allocation Failure) 2016-04-26T18:42:16.550+0800: 3722.774: [DefNew: 281299K->1679K(314560K), 0.0124507 secs] 315264K->35654K(1013632K), 0.0126969 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:42:52.776+0800: 3759.034: [GC (Allocation Failure) 2016-04-26T18:42:52.776+0800: 3759.034: [DefNew: 281295K->1673K(314560K), 0.0105041 secs] 315270K->35661K(1013632K), 0.0106768 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:43:28.412+0800: 3794.638: [GC (Allocation Failure) 2016-04-26T18:43:28.412+0800: 3794.638: [DefNew: 281289K->1674K(314560K), 0.0103795 secs] 315277K->35669K(1013632K), 0.0105525 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:44:04.536+0800: 3830.831: [GC (Allocation Failure) 2016-04-26T18:44:04.536+0800: 3830.831: [DefNew: 281290K->1674K(314560K), 0.0079088 secs] 315285K->35675K(1013632K), 0.0080555 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:44:40.190+0800: 3866.449: [GC (Allocation Failure) 2016-04-26T18:44:40.190+0800: 3866.449: [DefNew: 281290K->1673K(314560K), 0.0095096 secs] 315291K->35681K(1013632K), 0.0096861 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:45:16.528+0800: 3902.765: [GC (Allocation Failure) 2016-04-26T18:45:16.528+0800: 3902.765: [DefNew: 281289K->1673K(314560K), 0.0130461 secs] 315297K->35687K(1013632K), 0.0132673 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:45:52.574+0800: 3939.001: [GC (Allocation Failure) 2016-04-26T18:45:52.574+0800: 3939.001: [DefNew: 281289K->1680K(314560K), 0.0101569 secs] 315303K->35700K(1013632K), 0.0103066 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:46:28.413+0800: 3975.223: [GC (Allocation Failure) 2016-04-26T18:46:28.413+0800: 3975.223: [DefNew: 281296K->1679K(314560K), 0.0080063 secs] 315316K->35706K(1013632K), 0.0081820 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:47:04.190+0800: 4011.234: [GC (Allocation Failure) 2016-04-26T18:47:04.190+0800: 4011.234: [DefNew: 281295K->1678K(314560K), 0.0102572 secs] 315322K->35712K(1013632K), 0.0104334 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:47:40.161+0800: 4047.120: [GC (Allocation Failure) 2016-04-26T18:47:40.161+0800: 4047.120: [DefNew: 281294K->1677K(314560K), 0.0109581 secs] 315328K->35718K(1013632K), 0.0111609 secs] [Times: user=0.01 sys=0.00, real=0.02 secs]
2016-04-26T18:48:15.352+0800: 4082.428: [GC (Allocation Failure) 2016-04-26T18:48:15.352+0800: 4082.428: [DefNew: 281293K->1677K(314560K), 0.0087567 secs] 315334K->35724K(1013632K), 0.0089472 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:48:51.254+0800: 4118.407: [GC (Allocation Failure) 2016-04-26T18:48:51.254+0800: 4118.407: [DefNew: 281293K->1677K(314560K), 0.0115241 secs] 315340K->35731K(1013632K), 0.0117504 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:49:26.547+0800: 4153.705: [GC (Allocation Failure) 2016-04-26T18:49:26.547+0800: 4153.705: [DefNew: 281293K->1967K(314560K), 0.0095867 secs] 315347K->36027K(1013632K), 0.0097683 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:50:02.043+0800: 4189.206: [GC (Allocation Failure) 2016-04-26T18:50:02.043+0800: 4189.206: [DefNew: 281583K->1676K(314560K), 0.0096753 secs] 315643K->35742K(1013632K), 0.0098599 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:50:37.663+0800: 4224.505: [GC (Allocation Failure) 2016-04-26T18:50:37.663+0800: 4224.505: [DefNew: 281292K->1675K(314560K), 0.0077731 secs] 315358K->35749K(1013632K), 0.0079524 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:51:13.058+0800: 4259.780: [GC (Allocation Failure) 2016-04-26T18:51:13.058+0800: 4259.780: [DefNew: 281291K->1676K(314560K), 0.0086972 secs] 315365K->35756K(1013632K), 0.0088771 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:51:48.677+0800: 4295.269: [GC (Allocation Failure) 2016-04-26T18:51:48.677+0800: 4295.269: [DefNew: 281292K->1931K(314560K), 0.0099264 secs] 315372K->36016K(1013632K), 0.0101337 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2016-04-26T18:52:23.956+0800: 4330.493: [GC (Allocation Failure) 2016-04-26T18:52:23.956+0800: 4330.493: [DefNew: 281547K->1673K(314560K), 0.0079896 secs] 315632K->35766K(1013632K), 0.0081558 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:52:59.459+0800: 4366.025: [GC (Allocation Failure) 2016-04-26T18:52:59.459+0800: 4366.025: [DefNew: 281289K->1942K(314560K), 0.0099881 secs] 315382K->36041K(1013632K), 0.0101999 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:53:35.248+0800: 4401.571: [GC (Allocation Failure) 2016-04-26T18:53:35.248+0800: 4401.571: [DefNew: 281558K->1675K(314560K), 0.0080323 secs] 315657K->35779K(1013632K), 0.0081949 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:54:11.231+0800: 4437.434: [GC (Allocation Failure) 2016-04-26T18:54:11.231+0800: 4437.434: [DefNew: 281291K->1672K(314560K), 0.0107709 secs] 315395K->35785K(1013632K), 0.0109682 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T18:54:47.360+0800: 4473.741: [GC (Allocation Failure) 2016-04-26T18:54:47.360+0800: 4473.741: [DefNew: 281288K->1675K(314560K), 0.0077845 secs] 315401K->35796K(1013632K), 0.0079611 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:55:22.937+0800: 4509.361: [GC (Allocation Failure) 2016-04-26T18:55:22.937+0800: 4509.361: [DefNew: 281291K->1675K(314560K), 0.0082094 secs] 315412K->35802K(1013632K), 0.0083421 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2016-04-26T18:55:58.273+0800: 4544.708: [GC (Allocation Failure) 2016-04-26T18:55:58.273+0800: 4544.708: [DefNew: 281291K->1674K(314560K), 0.0100806 secs] 315418K->35807K(1013632K), 0.0102764 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:56:33.184+0800: 4579.506: [GC (Allocation Failure) 2016-04-26T18:56:33.184+0800: 4579.507: [DefNew: 281290K->1674K(314560K), 0.0111419 secs] 315423K->35814K(1013632K), 0.0113590 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:57:09.091+0800: 4615.482: [GC (Allocation Failure) 2016-04-26T18:57:09.091+0800: 4615.482: [DefNew: 281290K->1672K(314560K), 0.0139571 secs] 315430K->35818K(1013632K), 0.0141755 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2016-04-26T18:57:45.582+0800: 4651.989: [GC (Allocation Failure) 2016-04-26T18:57:45.582+0800: 4651.989: [DefNew: 281288K->1672K(314560K), 0.0089576 secs] 315434K->35824K(1013632K), 0.0091738 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:58:21.593+0800: 4688.001: [GC (Allocation Failure) 2016-04-26T18:58:21.593+0800: 4688.002: [DefNew: 281288K->1676K(314560K), 0.0158752 secs] 315440K->35835K(1013632K), 0.0160361 secs] [Times: user=0.00 sys=0.00, real=0.02 secs]
2016-04-26T18:58:56.204+0800: 4722.662: [GC (Allocation Failure) 2016-04-26T18:58:56.204+0800: 4722.662: [DefNew: 281292K->1675K(314560K), 0.0076781 secs] 315451K->35841K(1013632K), 0.0078622 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T18:59:31.386+0800: 4757.760: [GC (Allocation Failure) 2016-04-26T18:59:31.386+0800: 4757.760: [DefNew: 281291K->1678K(314560K), 0.0079359 secs] 315457K->35849K(1013632K), 0.0080991 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T19:00:07.775+0800: 4794.332: [GC (Allocation Failure) 2016-04-26T19:00:07.775+0800: 4794.332: [DefNew: 281294K->1676K(314560K), 0.0087637 secs] 315465K->35854K(1013632K), 0.0089416 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2016-04-26T19:00:44.243+0800: 4830.820: [GC (Allocation Failure) 2016-04-26T19:00:44.243+0800: 4830.820: [DefNew: 281292K->1674K(314560K), 0.0098652 secs] 315470K->35859K(1013632K), 0.0101359 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T19:01:19.025+0800: 4865.773: [GC (Allocation Failure) 2016-04-26T19:01:19.025+0800: 4865.773: [DefNew: 281290K->1686K(314560K), 0.0077619 secs] 315475K->35877K(1013632K), 0.0079340 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2016-04-26T19:01:47.622+0800: 4894.541: [GC (Allocation Failure) 2016-04-26T19:01:47.622+0800: 4894.541: [DefNew: 281302K->2540K(314560K), 0.0189695 secs] 315493K->36737K(1013632K), 0.0192055 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
Heap
def new generation total 314560K, used 268425K [0x04600000, 0x19b50000, 0x19b50000)
eden space 279616K, 95% used [0x04600000, 0x149a73e0, 0x15710000)
from space 34944K, 7% used [0x17930000, 0x17bab258, 0x19b50000)
to space 34944K, 0% used [0x15710000, 0x15710000, 0x17930000)
tenured generation total 699072K, used 34196K [0x19b50000, 0x44600000, 0x44600000)
the space 699072K, 4% used [0x19b50000, 0x1bcb5300, 0x1bcb5400, 0x44600000)
Metaspace used 21496K, capacity 21761K, committed 21888K, reserved 21888K
发件人: Vladimir Ozerov [mailto:vozerov@gridgain.com]
发送时间: 2016年4月25日 20:21
收件人: user@ignite.apache.org
主题: Re: Ignite cache data size problem.
Hi Kevin,
I performed several experiments. Essentially, I put 1M entries of the class you provided with fields initialized as follows:
for (int i = 0; i < 1_000_000; i++) {
UniqueField field = new UniqueField();
field.setDate(new Date());
field.setGuid(UUID.randomUUID().toString());
field.setMsg(String.valueOf(i));
field.setNum(BigDecimal.valueOf(ThreadLocalRandom.current().nextDouble()));
field.setOId(String.valueOf(i));
field.setOrgId(String.valueOf(i));
cache.put(i, field);
}
My results are:
1) Onheap, no indexes - about 400Mb is required to store 1M objects, or ~20Gb for 47M objects.
2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
3) Offheap, with indexes - about 400Mb offheap memory is required, or ~20Gb for all your objects.
Could you please provide more information on the error you receive? Also, you could try load entries in a batches of a well-defined size (say, 1M), and see what happens to the system. I expect you should see similar numbers.
Vladimir.
On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zh...@huawei.com>> wrote:
BTW, I created 4 + 3 nodes on two servers.
each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
kind regards,
Kevin
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite cache data size problem.
Posted by ght230 <gh...@163.com>.
I want to know how to see the size of memory used, using ignitevisorcmd?
for example, for the result about "Offheap, with indexes - about 400Mb
offheap memory is required, or ~20Gb for all your objects."
How to confirm the offheap memory is 400Mb, from "Non-heap memory
initialized", "Non-heap memory used", "Non-heap memory committed" or
"Non-heap memory maximum"?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p5008.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite cache data size problem.
Posted by vkulichenko <va...@gmail.com>.
Hi Kevin,
Ignite Visor [1] provides 'gc' command that performs GC on all nodes. You
can also use any other tool, e.g. VisualVM.
[1] https://apacheignite.readme.io/docs/command-line-interface
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4680.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite cache data size problem.
Posted by "kevin.zheng" <zh...@huawei.com>.
Hi Binti,
thank you for your help.
May I ask how to perform GC on each node?
using code? or command inside CLI?
best regards,
Kevin
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4635.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite cache data size problem.
Posted by bintisepaha <bi...@tudor.com>.
Kevin, we are facing similar issues. And the reason is not ignite using up
the heap, its the data you load from the database that takes up so much
memory until it makes it to the cache. We see a lot of heap usge after start
up and then if we perform a GC the memory on each node drops by at least 4
GB.
The way we are loading the data from the database is only running the
loadCache on the final node (we call it Initiator) and others are called
joiners. the joiners do not go to the database and only initiator does. But
initiator calls ignite.compute on the server nodes so that data is loaded
parallely like we want. the joiners are started first and have no data
loading code and then initiator is started later and it distributes work to
other joiner nodes and itself. hence DB queries are performed only once, but
in parallel.
Let me know if this makes sense.
Thanks,
Binti
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4625.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite cache data size problem.
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Kevin,
I would say that your assumption - more nodes, faster data load - makes
perfect sense. N nodes will have N threads in total, performing the load.
The problem is that current JDBC POJO store implementation makes this
process not very efficient, because each node iterates over the whole data
set. More efficient approach is to split the whole data set into several
parts (e.g. by attribute's value) and then let nodes iterate only over a
part of data, taking in count affinity function. This will be the most
efficient approach, I believe.
We will improve that in further Ignite versions.
Vladimir.
On Wed, Apr 27, 2016 at 1:52 PM, Zhengqingzheng <zh...@huawei.com>
wrote:
> Thank you Vladimir,
>
> I will try to use the second recommendation first, and let you know the
> result asap.
>
>
>
> I thought more nodes will result in fast data loading. Because I assume
> that each node can load different data simultaneously. Clearly, it is my
> misunderstanding about increasing number of nodes.
>
>
>
> Best regards,
>
> Kevin
>
>
>
> *发件人:* Vladimir Ozerov [mailto:vozerov@gridgain.com]
> *发送时间:* 2016年4月27日 18:03
> *收件人:* user@ignite.apache.org
> *主题:* Re: Ignite cache data size problem.
>
>
>
> Hi Kevin,
>
>
>
> My considerations:
>
> 1) I see that amount of allocated heap is gradually increases over time.
> Can you confirm that you use the configuration with OFFHEAP which you shown
> several posts earlier?
>
> 2) Why do you have several nodes per host? Our recommended approach is to
> have a single node per machine.
>
> 3) The main things: you invoke loadCache() method. This method will
> execute the same query - "SELECT *" as you mentioned earlier - on all
> nodes. It means that your cluster will have to perform full-scan N times,
> where N - number of nodes.
>
>
>
> I looked closely at our store API and I believe it is not suited for such
> cases (millions rows, lots of nodes in topology) well. We will think of
> better API to handle such scenarios.
>
>
>
> My recommendations for now:
>
>
>
> 1) Start no more than 1 node per host. This way you will decrease amount
> of scans from 16 to 2-3, which should make perf much better. Let each node
> take as much memory as possible.
>
>
>
> 2) If this approach still doesn't show good enough numbers, consider
> switching to *IgniteDataStreamer *instead -
> https://apacheignite.readme.io/docs/data-streamers
>
> It was designed specifically for efficient bulk data load by employing
> batching and affinity co-location techniques. You can do that in the
> following way:
>
> - Create *IgniteDataStreamer *for your cache;
>
> - Scan the table using JDBC, Hibernate of any other framework you have;
>
> - For each returned row, create appropriate cache key and value object;
>
> - Put key-value pair to the streamer: * IgniteDataStreamer.addData()*;
>
> - When scan is finished, close the streamer: IgniteDataStreamer.close().
>
> This should be done only on one node.
>
>
>
> I believe the second approach should show much better numbers.
>
>
>
> Vladimir.
>
>
>
>
>
> On Wed, Apr 27, 2016 at 6:06 AM, Zhengqingzheng <zh...@huawei.com>
> wrote:
>
> Hi Vladimir,
>
> Sorry to reply so late. The loadCache process took 8 hours to load all
> the data (this time ,there is no exception occurred, but memory consumption
> up to 56gb, 80% of all the heap that I have defined, which include 10
> nodes, each node allocate 7gb heap).
>
> I give you all the log files which were located at work/log/ folder.
> Please see the attachment.
>
>
>
>
>
> *发件人:* Vladimir Ozerov [mailto:vozerov@gridgain.com]
> *发送时间:* 2016年4月25日 20:21
> *收件人:* user@ignite.apache.org
> *主题:* Re: Ignite cache data size problem.
>
>
>
> Hi Kevin,
>
>
>
> I performed several experiments. Essentially, I put 1M entries of the
> class you provided with fields initialized as follows:
>
>
>
> *for *(*int *i = 0; i < 1_000_000; i++) {
> UniqueField field = *new *UniqueField();
>
> field.setDate(*new *Date());
> field.setGuid(UUID.*randomUUID*().toString());
> field.setMsg(String.*valueOf*(i));
> field.setNum(BigDecimal.*valueOf*(ThreadLocalRandom.*current*().nextDouble()));
> field.setOId(String.*valueOf*(i));
> field.setOrgId(String.*valueOf*(i));
>
> cache.put(i, field);
> }
>
>
>
> My results are:
>
> 1) Onheap, no indexes - about 400Mb is required to store 1M objects, or
> ~20Gb for 47M objects.
>
> 2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
>
> 3) Offheap, with indexes - about 400Mb offheap memory is required, or
> ~20Gb for all your objects.
>
>
>
> Could you please provide more information on the error you receive? Also,
> you could try load entries in a batches of a well-defined size (say, 1M),
> and see what happens to the system. I expect you should see similar numbers.
>
>
>
> Vladimir.
>
>
>
> On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zh...@huawei.com>
> wrote:
>
> BTW, I created 4 + 3 nodes on two servers.
> each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
>
> kind regards,
> Kevin
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
>
>
re: Ignite cache data size problem.
Posted by Zhengqingzheng <zh...@huawei.com>.
Thank you Vladimir,
I will try to use the second recommendation first, and let you know the result asap.
I thought more nodes will result in fast data loading. Because I assume that each node can load different data simultaneously. Clearly, it is my misunderstanding about increasing number of nodes.
Best regards,
Kevin
发件人: Vladimir Ozerov [mailto:vozerov@gridgain.com]
发送时间: 2016年4月27日 18:03
收件人: user@ignite.apache.org
主题: Re: Ignite cache data size problem.
Hi Kevin,
My considerations:
1) I see that amount of allocated heap is gradually increases over time. Can you confirm that you use the configuration with OFFHEAP which you shown several posts earlier?
2) Why do you have several nodes per host? Our recommended approach is to have a single node per machine.
3) The main things: you invoke loadCache() method. This method will execute the same query - "SELECT *" as you mentioned earlier - on all nodes. It means that your cluster will have to perform full-scan N times, where N - number of nodes.
I looked closely at our store API and I believe it is not suited for such cases (millions rows, lots of nodes in topology) well. We will think of better API to handle such scenarios.
My recommendations for now:
1) Start no more than 1 node per host. This way you will decrease amount of scans from 16 to 2-3, which should make perf much better. Let each node take as much memory as possible.
2) If this approach still doesn't show good enough numbers, consider switching to IgniteDataStreamer instead - https://apacheignite.readme.io/docs/data-streamers
It was designed specifically for efficient bulk data load by employing batching and affinity co-location techniques. You can do that in the following way:
- Create IgniteDataStreamer for your cache;
- Scan the table using JDBC, Hibernate of any other framework you have;
- For each returned row, create appropriate cache key and value object;
- Put key-value pair to the streamer: IgniteDataStreamer.addData();
- When scan is finished, close the streamer: IgniteDataStreamer.close().
This should be done only on one node.
I believe the second approach should show much better numbers.
Vladimir.
On Wed, Apr 27, 2016 at 6:06 AM, Zhengqingzheng <zh...@huawei.com>> wrote:
Hi Vladimir,
Sorry to reply so late. The loadCache process took 8 hours to load all the data (this time ,there is no exception occurred, but memory consumption up to 56gb, 80% of all the heap that I have defined, which include 10 nodes, each node allocate 7gb heap).
I give you all the log files which were located at work/log/ folder. Please see the attachment.
发件人: Vladimir Ozerov [mailto:vozerov@gridgain.com<ma...@gridgain.com>]
发送时间: 2016年4月25日 20:21
收件人: user@ignite.apache.org<ma...@ignite.apache.org>
主题: Re: Ignite cache data size problem.
Hi Kevin,
I performed several experiments. Essentially, I put 1M entries of the class you provided with fields initialized as follows:
for (int i = 0; i < 1_000_000; i++) {
UniqueField field = new UniqueField();
field.setDate(new Date());
field.setGuid(UUID.randomUUID().toString());
field.setMsg(String.valueOf(i));
field.setNum(BigDecimal.valueOf(ThreadLocalRandom.current().nextDouble()));
field.setOId(String.valueOf(i));
field.setOrgId(String.valueOf(i));
cache.put(i, field);
}
My results are:
1) Onheap, no indexes - about 400Mb is required to store 1M objects, or ~20Gb for 47M objects.
2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
3) Offheap, with indexes - about 400Mb offheap memory is required, or ~20Gb for all your objects.
Could you please provide more information on the error you receive? Also, you could try load entries in a batches of a well-defined size (say, 1M), and see what happens to the system. I expect you should see similar numbers.
Vladimir.
On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zh...@huawei.com>> wrote:
BTW, I created 4 + 3 nodes on two servers.
each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
kind regards,
Kevin
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite cache data size problem.
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Kevin,
My considerations:
1) I see that amount of allocated heap is gradually increases over time.
Can you confirm that you use the configuration with OFFHEAP which you shown
several posts earlier?
2) Why do you have several nodes per host? Our recommended approach is to
have a single node per machine.
3) The main things: you invoke loadCache() method. This method will execute
the same query - "SELECT *" as you mentioned earlier - on all nodes. It
means that your cluster will have to perform full-scan N times, where N -
number of nodes.
I looked closely at our store API and I believe it is not suited for such
cases (millions rows, lots of nodes in topology) well. We will think of
better API to handle such scenarios.
My recommendations for now:
1) Start no more than 1 node per host. This way you will decrease amount of
scans from 16 to 2-3, which should make perf much better. Let each node
take as much memory as possible.
2) If this approach still doesn't show good enough numbers, consider
switching to *IgniteDataStreamer *instead -
https://apacheignite.readme.io/docs/data-streamers
It was designed specifically for efficient bulk data load by employing
batching and affinity co-location techniques. You can do that in the
following way:
- Create *IgniteDataStreamer *for your cache;
- Scan the table using JDBC, Hibernate of any other framework you have;
- For each returned row, create appropriate cache key and value object;
- Put key-value pair to the streamer: *IgniteDataStreamer.addData()*;
- When scan is finished, close the streamer: IgniteDataStreamer.close().
This should be done only on one node.
I believe the second approach should show much better numbers.
Vladimir.
On Wed, Apr 27, 2016 at 6:06 AM, Zhengqingzheng <zh...@huawei.com>
wrote:
> Hi Vladimir,
>
> Sorry to reply so late. The loadCache process took 8 hours to load all
> the data (this time ,there is no exception occurred, but memory consumption
> up to 56gb, 80% of all the heap that I have defined, which include 10
> nodes, each node allocate 7gb heap).
>
> I give you all the log files which were located at work/log/ folder.
> Please see the attachment.
>
>
>
>
>
> *发件人:* Vladimir Ozerov [mailto:vozerov@gridgain.com]
> *发送时间:* 2016年4月25日 20:21
> *收件人:* user@ignite.apache.org
> *主题:* Re: Ignite cache data size problem.
>
>
>
> Hi Kevin,
>
>
>
> I performed several experiments. Essentially, I put 1M entries of the
> class you provided with fields initialized as follows:
>
>
>
> *for *(*int *i = 0; i < 1_000_000; i++) {
> UniqueField field = *new *UniqueField();
>
> field.setDate(*new *Date());
> field.setGuid(UUID.*randomUUID*().toString());
> field.setMsg(String.*valueOf*(i));
> field.setNum(BigDecimal.*valueOf*(ThreadLocalRandom.*current*().nextDouble()));
> field.setOId(String.*valueOf*(i));
> field.setOrgId(String.*valueOf*(i));
>
> cache.put(i, field);
> }
>
>
>
> My results are:
>
> 1) Onheap, no indexes - about 400Mb is required to store 1M objects, or
> ~20Gb for 47M objects.
>
> 2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
>
> 3) Offheap, with indexes - about 400Mb offheap memory is required, or
> ~20Gb for all your objects.
>
>
>
> Could you please provide more information on the error you receive? Also,
> you could try load entries in a batches of a well-defined size (say, 1M),
> and see what happens to the system. I expect you should see similar numbers.
>
>
>
> Vladimir.
>
>
>
> On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zh...@huawei.com>
> wrote:
>
> BTW, I created 4 + 3 nodes on two servers.
> each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
>
> kind regards,
> Kevin
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
re: Ignite cache data size problem.
Posted by Zhengqingzheng <zh...@huawei.com>.
Hi Vladimir,
Sorry to reply so late. The loadCache process took 8 hours to load all the data (this time ,there is no exception occurred, but memory consumption up to 56gb, 80% of all the heap that I have defined, which include 10 nodes, each node allocate 7gb heap).
I give you all the log files which were located at work/log/ folder. Please see the attachment.
发件人: Vladimir Ozerov [mailto:vozerov@gridgain.com]
发送时间: 2016年4月25日 20:21
收件人: user@ignite.apache.org
主题: Re: Ignite cache data size problem.
Hi Kevin,
I performed several experiments. Essentially, I put 1M entries of the class you provided with fields initialized as follows:
for (int i = 0; i < 1_000_000; i++) {
UniqueField field = new UniqueField();
field.setDate(new Date());
field.setGuid(UUID.randomUUID().toString());
field.setMsg(String.valueOf(i));
field.setNum(BigDecimal.valueOf(ThreadLocalRandom.current().nextDouble()));
field.setOId(String.valueOf(i));
field.setOrgId(String.valueOf(i));
cache.put(i, field);
}
My results are:
1) Onheap, no indexes - about 400Mb is required to store 1M objects, or ~20Gb for 47M objects.
2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
3) Offheap, with indexes - about 400Mb offheap memory is required, or ~20Gb for all your objects.
Could you please provide more information on the error you receive? Also, you could try load entries in a batches of a well-defined size (say, 1M), and see what happens to the system. I expect you should see similar numbers.
Vladimir.
On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zh...@huawei.com>> wrote:
BTW, I created 4 + 3 nodes on two servers.
each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
kind regards,
Kevin
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite cache data size problem.
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Kevin,
I performed several experiments. Essentially, I put 1M entries of the class
you provided with fields initialized as follows:
for (int i = 0; i < 1_000_000; i++) {
UniqueField field = new UniqueField();
field.setDate(new Date());
field.setGuid(UUID.randomUUID().toString());
field.setMsg(String.valueOf(i));
field.setNum(BigDecimal.valueOf(ThreadLocalRandom.current().nextDouble()));
field.setOId(String.valueOf(i));
field.setOrgId(String.valueOf(i));
cache.put(i, field);
}
My results are:
1) Onheap, no indexes - about 400Mb is required to store 1M objects, or
~20Gb for 47M objects.
2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
3) Offheap, with indexes - about 400Mb offheap memory is required, or ~20Gb
for all your objects.
Could you please provide more information on the error you receive? Also,
you could try load entries in a batches of a well-defined size (say, 1M),
and see what happens to the system. I expect you should see similar numbers.
Vladimir.
On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zh...@huawei.com>
wrote:
> BTW, I created 4 + 3 nodes on two servers.
> each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
>
> kind regards,
> Kevin
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Ignite cache data size problem.
Posted by "kevin.zheng" <zh...@huawei.com>.
BTW, I created 4 + 3 nodes on two servers.
each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
kind regards,
Kevin
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite cache data size problem.
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi,
It looks like you have relatively small entries (somewhere about 60-70
bytes per key-value pair). Ignite also has some intrinsic overhead, which
could be more than actual data in this case. However, I surely would not
expect that it will not fit into 80GB.
Could you please share your key and value model classes and your XML
configuration to investigate it further?
Vladimir.
On Fri, Apr 22, 2016 at 2:02 PM, Zhengqingzheng <zh...@huawei.com>
wrote:
> Hi there,
>
> I am trying to load a table with 47Million records, in which the data size
> is less than 3gb. However, When I load into the memory ( two vm with 48+32
> = 80gb), it is still crushed due to not enough memory space? This problem
> occurred when I instantiated 6 + 4 nodes.
>
> Why the cache model need so much space ( 3g vs 80g)?
>
> Any idea to explain this issue?
>
>
>
> Kind regards,
>
> Kevin
>
>
>