You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@skywalking.apache.org by bingjava <bi...@163.com> on 2020/04/21 06:54:58 UTC
skywalking Oapserver has a bug of OOM
Hi,All
The oapserver starts up less than 10 minutes, and the oom will occur. The memory object cannot be recycled。
Our generation environment is：
os：center os 7
jvm：jdk1.8
store：es6
Jvm config:
-Xmx10000M -Xms10000M -XX:MaxMetaspaceSize=512M -XX:MetaspaceSize=512M -XX:+UseG1GC -XX:MaxGCPauseMillis=200
[admin@b01_skywalking_service_pro_v_hk bin]$ jmap -histo:live 8955|more
num #instances #bytes class name
----------------------------------------------
1: 51829445 5950648600 [C
2: 51455355 2881499880 org.apache.skywalking.apm.network.register.v2.Endpoint
3: 51829083 1243897992 java.lang.String
4: 23340 320117896 [Ljava.lang.Object;
5: 502227 24106896 org.apache.skywalking.apm.network.register.v2.EndpointMappingElement
6: 186129 8934192 com.google.common.cache.LocalCache$StrongAccessEntry
7: 20448 5932408 [B
8: 91391 5117896 org.apache.skywalking.oap.server.core.register.EndpointInventory
9: 126969 4063008 java.util.HashMap$Node
10: 187623 3001968 java.lang.Integer
11: 186129 2978064 com.google.common.cache.LocalCache$StrongValueReference
12: 15944 2134000 [Ljava.util.HashMap$Node;
13: 3120 2046720 io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
14: 59041 1416984 org.apache.skywalking.oap.server.core.analysis.metrics.IntKeyLongValue
15: 10722 1189296 java.lang.Class
16: 184 990592 [Lio.netty.buffer.PoolSubpage;
17: 23800 952000 io.netty.buffer.UnpooledSlicedByteBuf
18: 26937 861984 java.util.concurrent.ConcurrentHashMap$Node
19: 9296 818048 io.netty.buffer.PooledUnsafeDirectByteBuf
20: 7719 744344 [I
21: 13564 651072 org.apache.skywalking.oap.server.core.analysis.metrics.IntKeyLongValueHashMap
22: 23800 571200 io.grpc.netty.NettyReadableBuffer
23: 6799 435136 java.nio.DirectByteBuffer
24: 15649 375576 javassist.bytecode.Utf8Info
25: 15544 373056 java.util.ArrayList
[admin@b01_skywalking_service_pro_v_hk bin]$ jstat -gcutil 8955 2000
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 0.00 0.00 100.00 97.16 93.64 86 9.428 4 19.310 28.739
0.00 0.00 0.00 100.00 97.16 93.64 86 9.428 4 19.310 28.739
0.00 0.00 0.00 99.98 96.97 93.35 86 9.428 4 35.545 44.974
0.00 0.00 0.00 99.98 96.97 93.35 86 9.428 4 35.545 44.974
0.00 100.00 84.62 99.97 96.97 93.35 88 9.784 4 35.545 45.329
0.00 0.00 0.00 99.97 96.98 93.35 91 11.077 5 35.545 46.622
0.00 100.00 94.44 99.98 96.98 93.35 93 11.087 5 52.030 63.116
0.00 0.00 0.00 100.00 96.83 93.64 94 12.600 6 52.030 64.629
0.00 0.00 0.00 99.97 96.83 93.64 94 12.600 6 68.554 81.154
0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
Re: skywalking Oapserver has a bug of OOM
Posted by Sheng Wu <wu...@gmail.com>.
I think you sent both OOM mails, right?
Both of them are same, Too many endpoints in the cache, due to you are
having parameter in the URI, and the parameter is a countless ID, which
caused countless endpoints and burns the cache.
In the 8.0, we removed that cache, so, basically, we could get rid of this,
but the release would be 1 month later.
At your current versions, consider to find out which URIs have this issue,
and fix them.
Sheng Wu 吴晟
Twitter, wusheng1108
bingjava <bi...@163.com> 于2020年4月21日周二 下午3:02写道：
> Hi,All
> The oapserver starts up less than 10 minutes, and the oom will occur.
> The memory object cannot be recycled。
> Our generation environment is：
> os：center os 7
> jvm：jdk1.8
> store：es6
> Jvm config:
> -Xmx10000M -Xms10000M -XX:MaxMetaspaceSize=512M -XX:MetaspaceSize=512M
> -XX:+UseG1GC -XX:MaxGCPauseMillis=200
>
>
> [admin@b01_skywalking_service_pro_v_hk bin]$ jmap -histo:live 8955|more
> num #instances #bytes class name
> ----------------------------------------------
> 1: 51829445 5950648600 [C
> 2: 51455355 2881499880
> org.apache.skywalking.apm.network.register.v2.Endpoint
> 3: 51829083 1243897992 java.lang.String
> 4: 23340 320117896 [Ljava.lang.Object;
> 5: 502227 24106896
> org.apache.skywalking.apm.network.register.v2.EndpointMappingElement
> 6: 186129 8934192 com.google.common.cache.LocalCache$StrongAccessEntry
> 7: 20448 5932408 [B
> 8: 91391 5117896
> org.apache.skywalking.oap.server.core.register.EndpointInventory
> 9: 126969 4063008 java.util.HashMap$Node
> 10: 187623 3001968 java.lang.Integer
> 11: 186129 2978064 com.google.common.cache.LocalCache$StrongValueReference
> 12: 15944 2134000 [Ljava.util.HashMap$Node;
> 13: 3120 2046720
> io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
> 14: 59041 1416984
> org.apache.skywalking.oap.server.core.analysis.metrics.IntKeyLongValue
> 15: 10722 1189296 java.lang.Class
> 16: 184 990592 [Lio.netty.buffer.PoolSubpage;
> 17: 23800 952000 io.netty.buffer.UnpooledSlicedByteBuf
> 18: 26937 861984 java.util.concurrent.ConcurrentHashMap$Node
> 19: 9296 818048 io.netty.buffer.PooledUnsafeDirectByteBuf
> 20: 7719 744344 [I
> 21: 13564 651072
> org.apache.skywalking.oap.server.core.analysis.metrics.IntKeyLongValueHashMap
> 22: 23800 571200 io.grpc.netty.NettyReadableBuffer
> 23: 6799 435136 java.nio.DirectByteBuffer
> 24: 15649 375576 javassist.bytecode.Utf8Info
> 25: 15544 373056 java.util.ArrayList
>
>
> [admin@b01_skywalking_service_pro_v_hk bin]$ jstat -gcutil 8955 2000
> S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
> 0.00 0.00 0.00 100.00 97.16 93.64 86 9.428 4 19.310 28.739
> 0.00 0.00 0.00 100.00 97.16 93.64 86 9.428 4 19.310 28.739
> 0.00 0.00 0.00 99.98 96.97 93.35 86 9.428 4 35.545 44.974
> 0.00 0.00 0.00 99.98 96.97 93.35 86 9.428 4 35.545 44.974
> 0.00 100.00 84.62 99.97 96.97 93.35 88 9.784 4 35.545 45.329
> 0.00 0.00 0.00 99.97 96.98 93.35 91 11.077 5 35.545 46.622
> 0.00 100.00 94.44 99.98 96.98 93.35 93 11.087 5 52.030 63.116
> 0.00 0.00 0.00 100.00 96.83 93.64 94 12.600 6 52.030 64.629
> 0.00 0.00 0.00 99.97 96.83 93.64 94 12.600 6 68.554 81.154
> 0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
> 0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
> 0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753