You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Jens Geyer (JIRA)" <ji...@apache.org> on 2014/04/27 18:36:15 UTC
[jira] [Updated] (THRIFT-2284) java.lang.OutOfMemoryError: Java
heap space
[ https://issues.apache.org/jira/browse/THRIFT-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jens Geyer updated THRIFT-2284:
-------------------------------
Description:
【1】The first version application Environment:
{code}
# cat /proc/version
Linux version 3.5.0-23-generic (buildd@komainu) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 UTC 2013
# java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
#thrift version is 0.7
{code}
Java Server code like following:use TNonblockingServer type server.
{code}
public void start() {
try {
logger.info("TNonblockingServer start ....");
TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(19875);
TNonblockingServer.Args tnbArgs = new TNonblockingServer.Args(tnbSocketTransport);
tnbArgs.processor(tprocessor);
tnbArgs.transportFactory(new TFramedTransport.Factory());
tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
TServer server = new TNonblockingServer(tnbArgs);
logger.info("TNonblockingServer start OK1....");
server.serve();
logger.info("TNonblockingServer start OK2....");
} catch (Exception e) {
logger.info("TNonblockingServer start ERROR!!!");
e.printStackTrace();
}
}
{code}
Problem description:
the thrift server interface invoked per day is much higher.as following:
# wc -l thrift.log.20131209
55598276 thrift.log
In general, the thrift server will be crashed down about 16 pm.
I much restart thrift server again an again,
it is very terrible,my boss will kill me(oh.oh).
so i use tool to monitor the thrift server.
linux top command result is that the RES item is 9.2G
Why use so many memory?
so crazy.
So I use Jmap command to print the jvm info that as following:
Object Histogram:
{code}
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 2493990 191419432 char[]
2: 1371792 43897344 java.lang.String
3: 525011 42306448 int[]
4: 356604 17116992 java.nio.HeapByteBuffer
5: 299290 14365920 java.nio.HeapCharBuffer
6: 70742 10203032 * ConstMethodKlass
7: 70742 9629584 * MethodKlass
8: 342788 8226912 java.lang.StringBuffer
9: 341661 8199864 java.lang.StringBuilder
10: 6644 7149072 * ConstantPoolKlass
11: 133484 6407232 org.springframework.aop.framework.ReflectiveMethodInvocation
12: 101052 6096736 * SymbolKlass
13: 144401 5776040 java.util.HashMap$KeyIterator
14: 6644 5039192 * InstanceKlassKlass
15: 160734 4881928 java.lang.Object[]
16: 149160 4773120 java.util.AbstractList$Itr
17: 59294 4269168 org.apache.log4j.spi.LoggingEvent
18: 5359 4171640 * ConstantPoolCacheKlass
19: 164812 3955488 java.util.LinkedList$Entry
20: 109117 3491744 java.util.HashMap$Entry
21: 59069 3307864 org.apache.thrift.protocol.TBinaryProtocol
{code}
Just a moment, thrift server throw an exception as following:
{code}
2013-12-09 16:01:32,181 ERROR [,Thread-44] - [org.apache.thrift.server.TNonblockingServer] - run() exiting due to uncaught error
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.apache.thrift.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:491)
at org.apache.thrift.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359)
at org.apache.thrift.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304)
at org.apache.thrift.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243)
{code}
【2】The seconde version application Environment:
because The first version application throw an
java.lang.OutOfMemoryError: Java heap space that i don't know how to do.
so the first step i want to do is that upgrade my thrift version..
the version 0.7 upgrade to the version 0.9.1
of course, I have changed my java code as following:
use the TThreadedSelectorServer type server.
{code}
try {
logger.info("TNonblockingServer start ....");
TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(9876);
TThreadedSelectorServer.Args tnbArgs = new TThreadedSelectorServer.Args(tnbSocketTransport);
tnbArgs.processor(tprocessor);
tnbArgs.transportFactory(new TFramedTransport.Factory(256));
tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
TServer server = new TThreadedSelectorServer(tnbArgs);
logger.info("TNonblockingServer start OK1....");
server.serve();
logger.info("TNonblockingServer start OK2....");
} catch (Exception e) {
logger.info("TNonblockingServer start ERROR!!!");
e.printStackTrace();
}
{code}
now it work well one day ,but i use linux top command find that the java process use much memory, i worry about that the thrift server will run out of memory again.what should i do ?
{code}
Cpu(s): 1.7%us, 1.0%sy, 0.0%ni, 97.1%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 16374956k total, 16205088k used, 169868k free, 70912k buffers
Swap: 3999740k total, 4660k used, 3995080k free, 10265608k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
60104 www-data 20 0 12.4g 4.5g 11m S 71 29.0 154:50.91 java
52938 root 20 0 264m 37m 1168 S 2 0.2 43:10.60 rsyslogd
903 root 20 0 17468 1384 944 R 1 0.0 0:00.02 top
53674 root 20 0 191m 23m 4740 S 1 0.1 71:11.18 admin-agent-msg
487 root 20 0 0 0 0 S 0 0.0 1:05.70 kworker/13:1
547 root 20 0 0 0 0 S 0 0.0 34:44.21 kworker/0:2
{code}
jmap info as following:
Object Histogram:
{code}
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 140522 1205495664 byte[]
2: 707542 58720520 char[]
3: 393972 12607104 java.lang.String
4: 143339 12535960 int[]
5: 71035 10236744 * ConstMethodKlass
6: 71035 9669432 * MethodKlass
7: 6674 7173400 * ConstantPoolKlass
8: 101411 6121416 * SymbolKlass
{code}
if you have any good idean,please share it to me,thanks.
address:zhuhai city of china
QQ:1010859834
mail:hiyoucai@126.com
was:
【1】The first version application Environment:
# cat /proc/version
Linux version 3.5.0-23-generic (buildd@komainu) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 UTC 2013
# java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
#thrift version is 0.7
Java Server code like following:use TNonblockingServer type server.
public void start() {
try {
logger.info("TNonblockingServer start ....");
TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(19875);
TNonblockingServer.Args tnbArgs = new TNonblockingServer.Args(tnbSocketTransport);
tnbArgs.processor(tprocessor);
tnbArgs.transportFactory(new TFramedTransport.Factory());
tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
TServer server = new TNonblockingServer(tnbArgs);
logger.info("TNonblockingServer start OK1....");
server.serve();
logger.info("TNonblockingServer start OK2....");
} catch (Exception e) {
logger.info("TNonblockingServer start ERROR!!!");
e.printStackTrace();
}
}
Problem description:
the thrift server interface invoked per day is much higher.as following:
# wc -l thrift.log.20131209
55598276 thrift.log
In general, the thrift server will be crashed down about 16 pm.
I much restart thrift server again an again,
it is very terrible,my boss will kill me(oh.oh).
so i use tool to monitor the thrift server.
linux top command result is that the RES item is 9.2G
Why use so many memory?
so crazy.
So I use Jmap command to print the jvm info that as following:
Object Histogram:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 2493990 191419432 char[]
2: 1371792 43897344 java.lang.String
3: 525011 42306448 int[]
4: 356604 17116992 java.nio.HeapByteBuffer
5: 299290 14365920 java.nio.HeapCharBuffer
6: 70742 10203032 * ConstMethodKlass
7: 70742 9629584 * MethodKlass
8: 342788 8226912 java.lang.StringBuffer
9: 341661 8199864 java.lang.StringBuilder
10: 6644 7149072 * ConstantPoolKlass
11: 133484 6407232 org.springframework.aop.framework.ReflectiveMethodInvocation
12: 101052 6096736 * SymbolKlass
13: 144401 5776040 java.util.HashMap$KeyIterator
14: 6644 5039192 * InstanceKlassKlass
15: 160734 4881928 java.lang.Object[]
16: 149160 4773120 java.util.AbstractList$Itr
17: 59294 4269168 org.apache.log4j.spi.LoggingEvent
18: 5359 4171640 * ConstantPoolCacheKlass
19: 164812 3955488 java.util.LinkedList$Entry
20: 109117 3491744 java.util.HashMap$Entry
21: 59069 3307864 org.apache.thrift.protocol.TBinaryProtocol
Just a moment, thrift server throw an exception as following:
2013-12-09 16:01:32,181 ERROR [,Thread-44] - [org.apache.thrift.server.TNonblockingServer] - run() exiting due to uncaught error
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.apache.thrift.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:491)
at org.apache.thrift.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359)
at org.apache.thrift.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304)
at org.apache.thrift.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243)
【2】The seconde version application Environment:
because The first version application throw an
java.lang.OutOfMemoryError: Java heap space that i don't know how to do.
so the first step i want to do is that upgrade my thrift version..
the version 0.7 upgrade to the version 0.9.1
of course, I have changed my java code as following:
use the TThreadedSelectorServer type server.
try {
logger.info("TNonblockingServer start ....");
TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(9876);
TThreadedSelectorServer.Args tnbArgs = new TThreadedSelectorServer.Args(tnbSocketTransport);
tnbArgs.processor(tprocessor);
tnbArgs.transportFactory(new TFramedTransport.Factory(256));
tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
TServer server = new TThreadedSelectorServer(tnbArgs);
logger.info("TNonblockingServer start OK1....");
server.serve();
logger.info("TNonblockingServer start OK2....");
} catch (Exception e) {
logger.info("TNonblockingServer start ERROR!!!");
e.printStackTrace();
}
now it work well one day ,but i use linux top command find that the java process use much memory, i worry about that the thrift server will run out of memory again.what should i do ?
Cpu(s): 1.7%us, 1.0%sy, 0.0%ni, 97.1%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 16374956k total, 16205088k used, 169868k free, 70912k buffers
Swap: 3999740k total, 4660k used, 3995080k free, 10265608k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
60104 www-data 20 0 12.4g 4.5g 11m S 71 29.0 154:50.91 java
52938 root 20 0 264m 37m 1168 S 2 0.2 43:10.60 rsyslogd
903 root 20 0 17468 1384 944 R 1 0.0 0:00.02 top
53674 root 20 0 191m 23m 4740 S 1 0.1 71:11.18 admin-agent-msg
487 root 20 0 0 0 0 S 0 0.0 1:05.70 kworker/13:1
547 root 20 0 0 0 0 S 0 0.0 34:44.21 kworker/0:2
jmap info as following:
Object Histogram:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 140522 1205495664 byte[]
2: 707542 58720520 char[]
3: 393972 12607104 java.lang.String
4: 143339 12535960 int[]
5: 71035 10236744 * ConstMethodKlass
6: 71035 9669432 * MethodKlass
7: 6674 7173400 * ConstantPoolKlass
8: 101411 6121416 * SymbolKlass
if you have any good idean,please share it to me,thanks.
address:zhuhai city of china
QQ:1010859834
mail:hiyoucai@126.com
> java.lang.OutOfMemoryError: Java heap space
> -------------------------------------------
>
> Key: THRIFT-2284
> URL: https://issues.apache.org/jira/browse/THRIFT-2284
> Project: Thrift
> Issue Type: Bug
> Components: Java - Library
> Affects Versions: 0.7, 0.9.1
> Environment: Thrift Server start up by Resin Server at Linux system.
> Reporter: hiyoucai
> Priority: Critical
>
> 【1】The first version application Environment:
> {code}
> # cat /proc/version
> Linux version 3.5.0-23-generic (buildd@komainu) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 UTC 2013
> # java -version
> java version "1.6.0_45"
> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
> #thrift version is 0.7
> {code}
> Java Server code like following:use TNonblockingServer type server.
> {code}
> public void start() {
> try {
> logger.info("TNonblockingServer start ....");
> TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
> TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(19875);
> TNonblockingServer.Args tnbArgs = new TNonblockingServer.Args(tnbSocketTransport);
> tnbArgs.processor(tprocessor);
> tnbArgs.transportFactory(new TFramedTransport.Factory());
> tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
> TServer server = new TNonblockingServer(tnbArgs);
> logger.info("TNonblockingServer start OK1....");
> server.serve();
> logger.info("TNonblockingServer start OK2....");
> } catch (Exception e) {
> logger.info("TNonblockingServer start ERROR!!!");
> e.printStackTrace();
> }
> }
> {code}
> Problem description:
> the thrift server interface invoked per day is much higher.as following:
> # wc -l thrift.log.20131209
> 55598276 thrift.log
> In general, the thrift server will be crashed down about 16 pm.
> I much restart thrift server again an again,
> it is very terrible,my boss will kill me(oh.oh).
> so i use tool to monitor the thrift server.
> linux top command result is that the RES item is 9.2G
> Why use so many memory?
> so crazy.
> So I use Jmap command to print the jvm info that as following:
> Object Histogram:
> {code}
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 2493990 191419432 char[]
> 2: 1371792 43897344 java.lang.String
> 3: 525011 42306448 int[]
> 4: 356604 17116992 java.nio.HeapByteBuffer
> 5: 299290 14365920 java.nio.HeapCharBuffer
> 6: 70742 10203032 * ConstMethodKlass
> 7: 70742 9629584 * MethodKlass
> 8: 342788 8226912 java.lang.StringBuffer
> 9: 341661 8199864 java.lang.StringBuilder
> 10: 6644 7149072 * ConstantPoolKlass
> 11: 133484 6407232 org.springframework.aop.framework.ReflectiveMethodInvocation
> 12: 101052 6096736 * SymbolKlass
> 13: 144401 5776040 java.util.HashMap$KeyIterator
> 14: 6644 5039192 * InstanceKlassKlass
> 15: 160734 4881928 java.lang.Object[]
> 16: 149160 4773120 java.util.AbstractList$Itr
> 17: 59294 4269168 org.apache.log4j.spi.LoggingEvent
> 18: 5359 4171640 * ConstantPoolCacheKlass
> 19: 164812 3955488 java.util.LinkedList$Entry
> 20: 109117 3491744 java.util.HashMap$Entry
> 21: 59069 3307864 org.apache.thrift.protocol.TBinaryProtocol
> {code}
> Just a moment, thrift server throw an exception as following:
> {code}
> 2013-12-09 16:01:32,181 ERROR [,Thread-44] - [org.apache.thrift.server.TNonblockingServer] - run() exiting due to uncaught error
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at org.apache.thrift.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:491)
> at org.apache.thrift.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359)
> at org.apache.thrift.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304)
> at org.apache.thrift.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243)
> {code}
> 【2】The seconde version application Environment:
> because The first version application throw an
> java.lang.OutOfMemoryError: Java heap space that i don't know how to do.
> so the first step i want to do is that upgrade my thrift version..
> the version 0.7 upgrade to the version 0.9.1
> of course, I have changed my java code as following:
> use the TThreadedSelectorServer type server.
> {code}
> try {
> logger.info("TNonblockingServer start ....");
> TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl);
> TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(9876);
> TThreadedSelectorServer.Args tnbArgs = new TThreadedSelectorServer.Args(tnbSocketTransport);
> tnbArgs.processor(tprocessor);
> tnbArgs.transportFactory(new TFramedTransport.Factory(256));
> tnbArgs.protocolFactory(new TBinaryProtocol.Factory());
> TServer server = new TThreadedSelectorServer(tnbArgs);
> logger.info("TNonblockingServer start OK1....");
> server.serve();
> logger.info("TNonblockingServer start OK2....");
> } catch (Exception e) {
> logger.info("TNonblockingServer start ERROR!!!");
> e.printStackTrace();
> }
> {code}
> now it work well one day ,but i use linux top command find that the java process use much memory, i worry about that the thrift server will run out of memory again.what should i do ?
> {code}
> Cpu(s): 1.7%us, 1.0%sy, 0.0%ni, 97.1%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
> Mem: 16374956k total, 16205088k used, 169868k free, 70912k buffers
> Swap: 3999740k total, 4660k used, 3995080k free, 10265608k cached
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 60104 www-data 20 0 12.4g 4.5g 11m S 71 29.0 154:50.91 java
> 52938 root 20 0 264m 37m 1168 S 2 0.2 43:10.60 rsyslogd
> 903 root 20 0 17468 1384 944 R 1 0.0 0:00.02 top
> 53674 root 20 0 191m 23m 4740 S 1 0.1 71:11.18 admin-agent-msg
> 487 root 20 0 0 0 0 S 0 0.0 1:05.70 kworker/13:1
> 547 root 20 0 0 0 0 S 0 0.0 34:44.21 kworker/0:2
> {code}
> jmap info as following:
> Object Histogram:
> {code}
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 140522 1205495664 byte[]
> 2: 707542 58720520 char[]
> 3: 393972 12607104 java.lang.String
> 4: 143339 12535960 int[]
> 5: 71035 10236744 * ConstMethodKlass
> 6: 71035 9669432 * MethodKlass
> 7: 6674 7173400 * ConstantPoolKlass
> 8: 101411 6121416 * SymbolKlass
> {code}
> if you have any good idean,please share it to me,thanks.
> address:zhuhai city of china
> QQ:1010859834
> mail:hiyoucai@126.com
--
This message was sent by Atlassian JIRA
(v6.2#6252)