You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/07/22 08:28:13 UTC
[GitHub] [skywalking] whfjam opened a new issue #5148: es6.3.2 in skywalking7 meet with deadlock
whfjam opened a new issue #5148:
URL: https://github.com/apache/skywalking/issues/5148
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [1] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
My es's version is 6.8.0 (if i should set it to 6.3.2)
My sk7 benkand has got the thread deadlock serveral times，the thread stacktrace is like below:
----------------------------------------------------------------------------------------------------------------------------
"grpcServerPool-1-thread-15" #65 prio=5 os_prio=0 tid=0x00000000007e7800 nid=0x4c91 waiting for monitor entry [0x00007f09e05f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:286)
- waiting to lock <0x00000000a401dae8> (a org.elasticsearch.action.bulk.BulkProcessor)
at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:271)
at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:267)
at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:253)
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.BatchProcessEsDAO.asynchronous(BatchProcessEsDAO.java:56)
at org.apache.skywalking.oap.server.core.analysis.worker.RecordPersistentWorker.in(RecordPersistentWorker.java:52)
at org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor.in(RecordStreamProcessor.java:51)
at org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher.dispatch(SegmentDispatcher.java:44)
at org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher.dispatch(SegmentDispatcher.java:25)
at org.apache.skywalking.oap.server.core.analysis.DispatcherManager.forward(DispatcherManager.java:59)
at org.apache.skywalking.oap.server.core.source.SourceReceiverImpl.receive(SourceReceiverImpl.java:35)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.listener.segment.SegmentSpanListener.build(SegmentSpanListener.java:142)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2$$Lambda$315/436973509.accept(Unknown Source)
at java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.notifyListenerToBuild(SegmentParseV2.java:260)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.parse(SegmentParseV2.java:165)
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2$Producer.send(SegmentParseV2.java:327)
at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:62)
at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:53)
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:251)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:782)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000000a435cbb0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
-----------------------------------------------------------------------------------------------------------------------------------
I found that the method "org.elasticsearch.action.bulk.BulkProcessor.internalAdd" is a synchronized one,and I also found that in
es6.8.5 it likes a bug ,in the page "https://www.elastic.co/guide/en/elasticsearch/reference/6.8/release-notes-6.8.5.html" it said "Prevent deadlock in BulkProcessor by using separate schedulers #48697 (issue: #47599) "
So do you have any ideas? Thks a lot!!
- Which company or project?
- What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #5148: es6.3.2 in skywalking7 meet with deadlock
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #5148:
URL: https://github.com/apache/skywalking/issues/5148#issuecomment-662329005
I think it is better you upgrade the es client to the version you need. If we upgrade, then 6.8.0 is required.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #5148: es6.3.2 in skywalking7 meet with deadlock
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #5148:
URL: https://github.com/apache/skywalking/issues/5148#issuecomment-662808254
If it isn't, you have to fix by yourself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #5148: es6.3.2 in skywalking7 meet with deadlock
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #5148:
URL: https://github.com/apache/skywalking/issues/5148
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] whfjam commented on issue #5148: es6.3.2 in skywalking7 meet with deadlock
Posted by GitBox <gi...@apache.org>.
whfjam commented on issue #5148:
URL: https://github.com/apache/skywalking/issues/5148#issuecomment-662805449
>
>
> I think it is better you upgrade the es client to the version you need. If we upgrade, then 6.8.0 is required.
i have updated my es client to 6.8.0 ，but it was not compatible with sk8，and it showed out several errors when i was compiling， just like：
/G:/eSign/skywalking8/oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java:[66,53] can not find symbol
[main] ERROR org.apache.maven.cli.MavenCli - symbol: class DeleteIndexResponse
[main] ERROR org.apache.maven.cli.MavenCli - location: package org.elasticsearch.action.admin.indices.delete
[main] ERROR org.apache.maven.cli.MavenCli -
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org