You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/12/18 10:53:18 UTC
[GitHub] [skywalking] leego86 opened a new issue #6030: APM for ES7 oap print a lot of errors
leego86 opened a new issue #6030:
URL: https://github.com/apache/skywalking/issues/6030
### Bug
- Which version of SkyWalking, OS, and JRE?
**version** APM for ES7 8.2 & 8.3
**OS** CentOS Linux release 7.3.1611 (Core)
**JRE** 1.8.0_171
- Which company or project?
LPPZ
- What happened?
The Rocketbot can display some data,but log file is printing errors
1. oap.log is printing a lot of errors, and the file is grouing bigger and bigger
```
Dec 18, 2020 6:24:10 PM io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable@7d968be6
io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte sequence
at io.grpc.Status.asRuntimeException(Status.java:524)
at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:218)
at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:118)
at io.grpc.MethodDescriptor.parseRequest(MethodDescriptor.java:296)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:782)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8.
at com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:141)
at com.google.protobuf.Utf8$DecodeUtil.handleTwoBytes(Utf8.java:1909)
at com.google.protobuf.Utf8$DecodeUtil.access$700(Utf8.java:1883)
at com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8(Utf8.java:1411)
at com.google.protobuf.Utf8.decodeUtf8(Utf8.java:340)
at com.google.protobuf.CodedInputStream$ArrayDecoder.readStringRequireUtf8(CodedInputStream.java:809)
at org.apache.skywalking.apm.network.language.agent.v3.SegmentObject.<init>(SegmentObject.java:65)
at org.apache.skywalking.apm.network.language.agent.v3.SegmentObject.<init>(SegmentObject.java:16)
at org.apache.skywalking.apm.network.language.agent.v3.SegmentObject$1.parsePartialFrom(SegmentObject.java:1656)
at org.apache.skywalking.apm.network.language.agent.v3.SegmentObject$1.parsePartialFrom(SegmentObject.java:1650)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:86)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parseFrom(ProtoLiteUtils.java:223)
at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:215)
... 10 more
```
2. skywalking-oap-server.log print error too,but the frequency is much lower ,like every 12s print some errors
```
2020-12-18 18:06:14,205 - org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker - 192 [pool-12-thread-1] ERROR [] - Can't split instance id into 2 parts, .1_
org.apache.skywalking.oap.server.core.UnexpectedException: Can't split instance id into 2 parts, .1_
at org.apache.skywalking.oap.server.core.analysis.IDManager$ServiceInstanceID.analysisId(IDManager.java:121) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.alarm.provider.NotifyHandler.notify(NotifyHandler.java:78) ~[server-alarm-plugin-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.alarm.AlarmEntrance.forward(AlarmEntrance.java:39) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:41) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:30) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.lambda$nextWorker$2(MetricsPersistentWorker.java:199) ~[server-core-8.3.0.jar:8.3.0]
at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_171]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.nextWorker(MetricsPersistentWorker.java:199) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.flushDataToStorage(MetricsPersistentWorker.java:182) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.prepareBatch(MetricsPersistentWorker.java:141) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.PersistenceWorker.buildBatchRequests(PersistenceWorker.java:74) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$extractDataAndSave$2(PersistenceTimer.java:107) ~[server-core-8.3.0.jar:8.3.0]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_171]
at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:102) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.3.0.jar:8.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2020-12-18 18:06:14,212 - org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker - 192 [pool-12-thread-1] ERROR [] - Can't split instance id into 2 parts, .1_
org.apache.skywalking.oap.server.core.UnexpectedException: Can't split instance id into 2 parts, .1_
at org.apache.skywalking.oap.server.core.analysis.IDManager$ServiceInstanceID.analysisId(IDManager.java:121) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.alarm.provider.NotifyHandler.notify(NotifyHandler.java:78) ~[server-alarm-plugin-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.alarm.AlarmEntrance.forward(AlarmEntrance.java:39) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:41) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.AlarmNotifyWorker.in(AlarmNotifyWorker.java:30) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.lambda$nextWorker$2(MetricsPersistentWorker.java:199) ~[server-core-8.3.0.jar:8.3.0]
at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_171]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.nextWorker(MetricsPersistentWorker.java:199) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.flushDataToStorage(MetricsPersistentWorker.java:182) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.MetricsPersistentWorker.prepareBatch(MetricsPersistentWorker.java:141) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.analysis.worker.PersistenceWorker.buildBatchRequests(PersistenceWorker.java:74) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$extractDataAndSave$2(PersistenceTimer.java:107) ~[server-core-8.3.0.jar:8.3.0]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[?:1.8.0_171]
at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.extractDataAndSave(PersistenceTimer.java:102) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.oap.server.core.storage.PersistenceTimer.lambda$start$0(PersistenceTimer.java:78) ~[server-core-8.3.0.jar:8.3.0]
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33) [apm-util-8.3.0.jar:8.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
```
I want to known that
1. how to solve the problems
2. how can I config **oap.log** rolling to prevent disk from becoming full
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] leego86 commented on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
leego86 commented on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-748408732
It means I should set charset to UTF-8?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] zj452008181 commented on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
zj452008181 commented on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-1007873602
i got the same errors in oap 8.9,my es version is 7.2,agent is 8.8.0,did you solve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] zj452008181 commented on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
zj452008181 commented on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-1007873602
i got the same errors in oap 8.9,my es version is 7.2,agent is 8.8.0,did you solve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-748021898
I think we have expired data, such as <8 release database.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 edited a comment on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
kezhenxu94 edited a comment on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-748417416
> It means I should set charset to UTF-8?
@leego86 "Protocol message had invalid UTF-8." indicates that there are other services than SkyWalking agents that try to send non-ProtoBuffer message to the host:port, make sure to give a dedicated port for the OAP, a quick diagnose method is to stop all your agent services, and see whether the logs still exist, if it does, you should find out who is sending the invalid messages.
If there is proxy between the OAP and agents, should also check the proxy settings
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-748417416
> It means I should set charset to UTF-8?
@leego86 "Protocol message had invalid UTF-8." indicates that there are other services than SkyWalking agents that try to send non-ProtoBuffer message to the host:port, make sure to give a dedicated port for the OAP, a quick diagnose method is to stop all your agent services, and see whether the logs still exist, if it does, you should find out who is sending the invalid messages.
If there is proxy between the OAP and agents, check the proxy settings
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-748416466
There are 2 issues in your logs. One old version data is incompatible, which is clear, you are sharing database. UTF8 is the thing logs say.
I can't make sure what is your real case. You have to try by yourself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-748407258
I can't set what your service did, I just highlight this from your logs. SkyWalking is depending on UTF-8 in many places.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #6030:
URL: https://github.com/apache/skywalking/issues/6030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] leego86 commented on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
leego86 commented on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-748406139
I only have 3 springboot services use of agent now,And have not config of charset any where
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-748022578
> Protocol message had invalid UTF-8.
I think you are not using other chatset?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 edited a comment on issue #6030: APM for ES7 oap print a lot of errors
Posted by GitBox <gi...@apache.org>.
kezhenxu94 edited a comment on issue #6030:
URL: https://github.com/apache/skywalking/issues/6030#issuecomment-748417416
> It means I should set charset to UTF-8?
@leego86 "Invalid protobuf byte sequence" and "Protocol message had invalid UTF-8." indicates that there are other services than SkyWalking agents that try to send non-ProtoBuffer message to the host:port, make sure to give a dedicated port for the OAP, a quick diagnose method is to stop all your agent services, and see whether the logs still exist, if it does, you should find out who is sending the invalid messages.
If there is proxy between the OAP and agents, should also check the proxy settings
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org