You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@skywalking.apache.org by bingjava <bi...@163.com> on 2020/04/21 10:32:11 UTC
回复：skywalking Oapserver has a bug of OOM
Our API is restful and Dynamic ID will be included in the path, and QPS can reach more than 50000，So we have to wait until version 8 is released！
原始邮件
发件人: bingjava<bi...@163.com>
收件人: dev<de...@skywalking.apache.org>
抄送: Sheng Wu<wu...@gmail.com>
发送时间: 2020年4月21日(周二) 14:54
主题: skywalking Oapserver has a bug of OOM
Hi,All
The oapserver starts up less than 10 minutes, and the oom will occur. The memory object cannot be recycled。
Our generation environment is：
os：center os 7
jvm：jdk1.8
store：es6
Jvm config:
-Xmx10000M -Xms10000M -XX:MaxMetaspaceSize=512M -XX:MetaspaceSize=512M -XX:+UseG1GC -XX:MaxGCPauseMillis=200
[admin@b01_skywalking_service_pro_v_hk bin]$ jmap -histo:live 8955|more
num #instances #bytes class name
----------------------------------------------
1: 51829445 5950648600 [C
2: 51455355 2881499880 org.apache.skywalking.apm.network.register.v2.Endpoint
3: 51829083 1243897992 java.lang.String
4: 23340 320117896 [Ljava.lang.Object;
5: 502227 24106896 org.apache.skywalking.apm.network.register.v2.EndpointMappingElement
6: 186129 8934192 com.google.common.cache.LocalCache$StrongAccessEntry
7: 20448 5932408 [B
8: 91391 5117896 org.apache.skywalking.oap.server.core.register.EndpointInventory
9: 126969 4063008 java.util.HashMap$Node
10: 187623 3001968 java.lang.Integer
11: 186129 2978064 com.google.common.cache.LocalCache$StrongValueReference
12: 15944 2134000 [Ljava.util.HashMap$Node;
13: 3120 2046720 io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
14: 59041 1416984 org.apache.skywalking.oap.server.core.analysis.metrics.IntKeyLongValue
15: 10722 1189296 java.lang.Class
16: 184 990592 [Lio.netty.buffer.PoolSubpage;
17: 23800 952000 io.netty.buffer.UnpooledSlicedByteBuf
18: 26937 861984 java.util.concurrent.ConcurrentHashMap$Node
19: 9296 818048 io.netty.buffer.PooledUnsafeDirectByteBuf
20: 7719 744344 [I
21: 13564 651072 org.apache.skywalking.oap.server.core.analysis.metrics.IntKeyLongValueHashMap
22: 23800 571200 io.grpc.netty.NettyReadableBuffer
23: 6799 435136 java.nio.DirectByteBuffer
24: 15649 375576 javassist.bytecode.Utf8Info
25: 15544 373056 java.util.ArrayList
[admin@b01_skywalking_service_pro_v_hk bin]$ jstat -gcutil 8955 2000
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 0.00 0.00 100.00 97.16 93.64 86 9.428 4 19.310 28.739
0.00 0.00 0.00 100.00 97.16 93.64 86 9.428 4 19.310 28.739
0.00 0.00 0.00 99.98 96.97 93.35 86 9.428 4 35.545 44.974
0.00 0.00 0.00 99.98 96.97 93.35 86 9.428 4 35.545 44.974
0.00 100.00 84.62 99.97 96.97 93.35 88 9.784 4 35.545 45.329
0.00 0.00 0.00 99.97 96.98 93.35 91 11.077 5 35.545 46.622
0.00 100.00 94.44 99.98 96.98 93.35 93 11.087 5 52.030 63.116
0.00 0.00 0.00 100.00 96.83 93.64 94 12.600 6 52.030 64.629
0.00 0.00 0.00 99.97 96.83 93.64 94 12.600 6 68.554 81.154
0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
Re: skywalking Oapserver has a bug of OOM
Posted by Sheng Wu <wu...@gmail.com>.
If you don't want to change SkyWalking source codes, then yes.
But to be honest, that is still a fail safe, from my understanding, you
still need to format /path/123, /path/234 into /path/{parameter} by using a
custom/private plugin for your own system.
Because if not, even agent and backend wouldn't crash, the endpoint
statistic result would be pointless, as you have countless endpoints, and
no aggregation happens.
Sheng Wu 吴晟
Twitter, wusheng1108
bingjava <bi...@163.com> 于2020年4月21日周二 下午6:32写道：
> Our API is restful and Dynamic ID will be included in the path, and QPS
> can reach more than 50000，So we have to wait until version 8 is released！
>
>
>
>
> 原始邮件
> 发件人: bingjava<bi...@163.com>
> 收件人: dev<de...@skywalking.apache.org>
> 抄送: Sheng Wu<wu...@gmail.com>
> 发送时间: 2020年4月21日(周二) 14:54
> 主题: skywalking Oapserver has a bug of OOM
>
>
> Hi,All
> The oapserver starts up less than 10 minutes, and the oom will occur.
> The memory object cannot be recycled。
> Our generation environment is：
> os：center os 7
> jvm：jdk1.8
> store：es6
> Jvm config:
> -Xmx10000M -Xms10000M -XX:MaxMetaspaceSize=512M -XX:MetaspaceSize=512M
> -XX:+UseG1GC -XX:MaxGCPauseMillis=200
>
>
> [admin@b01_skywalking_service_pro_v_hk bin]$ jmap -histo:live 8955|more
> num #instances #bytes class name
> ----------------------------------------------
> 1: 51829445 5950648600 [C
> 2: 51455355 2881499880
> org.apache.skywalking.apm.network.register.v2.Endpoint
> 3: 51829083 1243897992 java.lang.String
> 4: 23340 320117896 [Ljava.lang.Object;
> 5: 502227 24106896
> org.apache.skywalking.apm.network.register.v2.EndpointMappingElement
> 6: 186129 8934192 com.google.common.cache.LocalCache$StrongAccessEntry
> 7: 20448 5932408 [B
> 8: 91391 5117896
> org.apache.skywalking.oap.server.core.register.EndpointInventory
> 9: 126969 4063008 java.util.HashMap$Node
> 10: 187623 3001968 java.lang.Integer
> 11: 186129 2978064 com.google.common.cache.LocalCache$StrongValueReference
> 12: 15944 2134000 [Ljava.util.HashMap$Node;
> 13: 3120 2046720
> io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue
> 14: 59041 1416984
> org.apache.skywalking.oap.server.core.analysis.metrics.IntKeyLongValue
> 15: 10722 1189296 java.lang.Class
> 16: 184 990592 [Lio.netty.buffer.PoolSubpage;
> 17: 23800 952000 io.netty.buffer.UnpooledSlicedByteBuf
> 18: 26937 861984 java.util.concurrent.ConcurrentHashMap$Node
> 19: 9296 818048 io.netty.buffer.PooledUnsafeDirectByteBuf
> 20: 7719 744344 [I
> 21: 13564 651072
> org.apache.skywalking.oap.server.core.analysis.metrics.IntKeyLongValueHashMap
> 22: 23800 571200 io.grpc.netty.NettyReadableBuffer
> 23: 6799 435136 java.nio.DirectByteBuffer
> 24: 15649 375576 javassist.bytecode.Utf8Info
> 25: 15544 373056 java.util.ArrayList
>
>
> [admin@b01_skywalking_service_pro_v_hk bin]$ jstat -gcutil 8955 2000
> S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
> 0.00 0.00 0.00 100.00 97.16 93.64 86 9.428 4 19.310 28.739
> 0.00 0.00 0.00 100.00 97.16 93.64 86 9.428 4 19.310 28.739
> 0.00 0.00 0.00 99.98 96.97 93.35 86 9.428 4 35.545 44.974
> 0.00 0.00 0.00 99.98 96.97 93.35 86 9.428 4 35.545 44.974
> 0.00 100.00 84.62 99.97 96.97 93.35 88 9.784 4 35.545 45.329
> 0.00 0.00 0.00 99.97 96.98 93.35 91 11.077 5 35.545 46.622
> 0.00 100.00 94.44 99.98 96.98 93.35 93 11.087 5 52.030 63.116
> 0.00 0.00 0.00 100.00 96.83 93.64 94 12.600 6 52.030 64.629
> 0.00 0.00 0.00 99.97 96.83 93.64 94 12.600 6 68.554 81.154
> 0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
> 0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753
> 0.00 0.00 0.00 99.97 96.83 93.64 98 13.199 7 68.554 81.753