You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Gonzalo Aguilar (JIRA)" <ji...@apache.org> on 2015/05/23 10:35:17 UTC
[jira] [Commented] (THRIFT-2284) java.lang.OutOfMemoryError: Java
heap space
[ https://issues.apache.org/jira/browse/THRIFT-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557219#comment-14557219 ]
Gonzalo Aguilar commented on THRIFT-2284:
-----------------------------------------
I have the same problem. I start the server do a request. And do nothing. After few hours:
23:01:23,695 ERROR [org.apache.thrift.server.TThreadedSelectorServer] (Thread-90) run() exiting due to uncaught error: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) [rt.jar:1.7.0_80]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:331) [rt.jar:1.7.0_80]
at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:371) [libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203) [libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:590) [libthrift-0.9.2.jar:0.9.2]
at org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:545) [libthrift-0.9.2.jar:0.9.2]
23:01:23,804 INFO [com.seglan.tvm.thrift.mc.mobile.ThriftDaemon] (Thread-88) Thrift server is finished
And this is the offending code:
TNonblockingServerSocket serverTransport = null;
InetSocketAddress address = new InetSocketAddress(serverURI.getHost(), serverURI.getPort());
log.info("Running " + address.getHostString() + " port " + address.getPort());
try {
serverTransport = new TNonblockingServerSocket(address);
log.info("Server is created as asynchronous server non blocking");
} catch (TTransportException e) {
log.warn("Cannot create server: " + e.getMessage());
return;
}
/*
* TThreadedSelectorServer allows you to have multiple threads for network I/O. It maintains 2 thread pools,
* one for handling network I/O, and one for handling request processing.
* TThreadedSelectorServer performs better than THsHaServer when the network io is the bottleneck.
*/
TThreadedSelectorServer.Args serverArgs = new TThreadedSelectorServer.Args(serverTransport);
/*
* TFramedTransport is a buffered TTransport that ensures a fully read message every time
* by preceding messages with a 4-byte frame size.
*/
serverArgs.transportFactory(new TFramedTransport.Factory());
/*
* We want to transcode everything to binary format for performance reasons
*/
serverArgs.protocolFactory(new TBinaryProtocol.Factory());
/*
* This is the processor that will call handler
*/
serverArgs.processor(this.getServerProcessor(this.getServerHandler()));
/*
* We want to receive messages with 4 threads
*/
serverArgs.selectorThreads(4);
/*
* There will be 8 threads for serving messages
*/
serverArgs.workerThreads(8);
/*
* Fix heap problem
* https://issues.apache.org/jira/browse/THRIFT-2284
*/
serverArgs.maxReadBufferBytes = Integer.MAX_VALUE;
server = new TThreadedSelectorServer(serverArgs);
log.info("Server is running...");
server.serve();
> java.lang.OutOfMemoryError: Java heap space
> -------------------------------------------
>
> Key: THRIFT-2284
> URL: https://issues.apache.org/jira/browse/THRIFT-2284
> Project: Thrift
> Issue Type: Bug
> Components: Java - Library
> Affects Versions: 0.7, 0.9.1
> Environment: Thrift Server start up by Resin Server at Linux system.
> Reporter: hiyoucai
> Priority: Critical
> Fix For: 0.9.2
>
>
> 【1】The first version application Environment:
> {code}
> # cat /proc/version
> Linux version 3.5.0-23-generic (buildd@komainu) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 UTC 2013
> # java -version
> java version "1.6.0_45"
> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
> #thrift version is 0.7
> {code}
> Java Server code like following:use TNonblockingServer type server.
> {code}
> public void start() {
> try {
> logger.info("TNonblockingServer start ....");
> TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
> TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(19875);
> TNonblockingServer.Args tnbArgs = new TNonblockingServer.Args(tnbSocketTransport);
> tnbArgs.processor(tprocessor);
> tnbArgs.transportFactory(new TFramedTransport.Factory());
> tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
> TServer server = new TNonblockingServer(tnbArgs);
> logger.info("TNonblockingServer start OK1....");
> server.serve();
> logger.info("TNonblockingServer start OK2....");
> } catch (Exception e) {
> logger.info("TNonblockingServer start ERROR!!!");
> e.printStackTrace();
> }
> }
> {code}
> Problem description:
> the thrift server interface invoked per day is much higher.as following:
> # wc -l thrift.log.20131209
> 55598276 thrift.log
> In general, the thrift server will be crashed down about 16 pm.
> I much restart thrift server again an again,
> it is very terrible,my boss will kill me(oh.oh).
> so i use tool to monitor the thrift server.
> linux top command result is that the RES item is 9.2G
> Why use so many memory?
> so crazy.
> So I use Jmap command to print the jvm info that as following:
> Object Histogram:
> {code}
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 2493990 191419432 char[]
> 2: 1371792 43897344 java.lang.String
> 3: 525011 42306448 int[]
> 4: 356604 17116992 java.nio.HeapByteBuffer
> 5: 299290 14365920 java.nio.HeapCharBuffer
> 6: 70742 10203032 * ConstMethodKlass
> 7: 70742 9629584 * MethodKlass
> 8: 342788 8226912 java.lang.StringBuffer
> 9: 341661 8199864 java.lang.StringBuilder
> 10: 6644 7149072 * ConstantPoolKlass
> 11: 133484 6407232 org.springframework.aop.framework.ReflectiveMethodInvocation
> 12: 101052 6096736 * SymbolKlass
> 13: 144401 5776040 java.util.HashMap$KeyIterator
> 14: 6644 5039192 * InstanceKlassKlass
> 15: 160734 4881928 java.lang.Object[]
> 16: 149160 4773120 java.util.AbstractList$Itr
> 17: 59294 4269168 org.apache.log4j.spi.LoggingEvent
> 18: 5359 4171640 * ConstantPoolCacheKlass
> 19: 164812 3955488 java.util.LinkedList$Entry
> 20: 109117 3491744 java.util.HashMap$Entry
> 21: 59069 3307864 org.apache.thrift.protocol.TBinaryProtocol
> {code}
> Just a moment, thrift server throw an exception as following:
> {code}
> 2013-12-09 16:01:32,181 ERROR [,Thread-44] - [org.apache.thrift.server.TNonblockingServer] - run() exiting due to uncaught error
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at org.apache.thrift.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:491)
> at org.apache.thrift.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359)
> at org.apache.thrift.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304)
> at org.apache.thrift.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243)
> {code}
> 【2】The seconde version application Environment:
> because The first version application throw an
> java.lang.OutOfMemoryError: Java heap space that i don't know how to do.
> so the first step i want to do is that upgrade my thrift version..
> the version 0.7 upgrade to the version 0.9.1
> of course, I have changed my java code as following:
> use the TThreadedSelectorServer type server.
> {code}
> try {
> logger.info("TNonblockingServer start ....");
> TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
> TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(9876);
> TThreadedSelectorServer.Args tnbArgs = new TThreadedSelectorServer.Args(tnbSocketTransport);
> tnbArgs.processor(tprocessor);
> tnbArgs.transportFactory(new TFramedTransport.Factory(256));
> tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
> TServer server = new TThreadedSelectorServer(tnbArgs);
> logger.info("TNonblockingServer start OK1....");
> server.serve();
> logger.info("TNonblockingServer start OK2....");
> } catch (Exception e) {
> logger.info("TNonblockingServer start ERROR!!!");
> e.printStackTrace();
> }
> {code}
> now it work well one day ,but i use linux top command find that the java process use much memory, i worry about that the thrift server will run out of memory again.what should i do ?
> {code}
> Cpu(s): 1.7%us, 1.0%sy, 0.0%ni, 97.1%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 16374956k total, 16205088k used, 169868k free, 70912k buffers
> Swap: 3999740k total, 4660k used, 3995080k free, 10265608k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 60104 www-data 20 0 12.4g 4.5g 11m S 71 29.0 154:50.91 java
> 52938 root 20 0 264m 37m 1168 S 2 0.2 43:10.60 rsyslogd
> 903 root 20 0 17468 1384 944 R 1 0.0 0:00.02 top
> 53674 root 20 0 191m 23m 4740 S 1 0.1 71:11.18 admin-agent-msg
> 487 root 20 0 0 0 0 S 0 0.0 1:05.70 kworker/13:1
> 547 root 20 0 0 0 0 S 0 0.0 34:44.21 kworker/0:2
> {code}
> jmap info as following:
> Object Histogram:
> {code}
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 140522 1205495664 byte[]
> 2: 707542 58720520 char[]
> 3: 393972 12607104 java.lang.String
> 4: 143339 12535960 int[]
> 5: 71035 10236744 * ConstMethodKlass
> 6: 71035 9669432 * MethodKlass
> 7: 6674 7173400 * ConstantPoolKlass
> 8: 101411 6121416 * SymbolKlass
> {code}
> if you have any good idean,please share it to me,thanks.
> address:zhuhai city of china
> QQ:1010859834
> mail:hiyoucai@126.com
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)