You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/02/17 09:41:30 UTC
[GitHub] [skywalking] justmine66 opened a new issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
justmine66 opened a new issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ ] Question or discussion
- [√ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
Use ALS solution，istio integrates skywalking, bug has error.
istio version 1.4.4
___
### Bug
- Which version of SkyWalking, OS and JRE?
SkyWalking oap log：
2020-02-17 09:37:53,227 - org.apache.skywalking.oap.server.receiver.envoy.als.K8sALSServiceMeshHTTPAnalysis -20802651 [load-pod-0] INFO [] - Load 40 pods in 92ms
Feb 17, 2020 9:38:05 AM io.grpc.netty.NettyServerTransport notifyTerminated
INFO: Transport failed
io.netty.handler.codec.http2.Http2Exception: Stream 19 does not exist
at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2020-02-17 09:28:34,625 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker -20244049 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 583;
org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 583;
- Which company or project?
- What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587321792
> We are using this in the production env. There is no issue. I don't know what is your issue.
skywalking is deployed in the way: https://github.com/apache/skywalking-kubernetes, but has error:
Caused by: org.elasticsearch.client.ResponseException: method [GET], host [http://skywalking-elasticsearch-client:9200], URI [/service_instance_inventory/type/2_a24abba3-5aec-43a4-87ea-9d4b020cf244_0_0], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"No shard available for [get [service_instance_inventory][type][2_a24abba3-5aec-43a4-87ea-9d4b020cf244_0_0]: routing [null]]"}],"type":"no_shard_available_action_exception","reason":"No shard available for [get [service_instance_inventory][type][2_a24abba3-5aec-43a4-87ea-9d4b020cf244_0_0]: routing [null]]"},"status":503}
this solution(ttps://github.com/apache/skywalking-kubernetes), you tested?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587324901
> 503 means ES crashed.
That's why there's no data. ES is also automatically deployed by this solution, (K8s all states are green)there is bug...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587294141
> We are using this in the production env. There is no issue. I don't know what is your issue.
skywalking is deployed in the way: https://github.com/apache/skywalking-kubernetes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587322488
503 means ES crashed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 edited a comment on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 edited a comment on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587294141
> We are using this in the production env. There is no issue. I don't know what is your issue.
in your production env, skywalking is deployed in the way: https://github.com/apache/skywalking-kubernetes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 removed a comment on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 removed a comment on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587232360
i see, Envoy attatched istio, i find other more effective error:
2020-02-18 01:26:04,456 - org.apache.skywalking.oap.server.receiver.envoy.als.K8sALSServiceMeshHTTPAnalysis -38576405 [load-pod-0] INFO [] - Load 40 pods in 80ms
2020-02-18 01:26:04,736 - org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient -38576685 [grpc-default-executor-139] ERROR [] - INTERNAL: Received unexpected EOS on DATA frame from server.
io.grpc.StatusRuntimeException: INTERNAL: Received unexpected EOS on DATA frame from server.
Is there a requirement for envoy specific version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-586911187
> I read two error
>
> 1. K8sALSServiceMeshHTTPAnalysis, it may be caused by your API server could not response in time.
> 2. `org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 583;`. Your endpoint is too long, I don't know why.
What is the istio version and env of this scheme tested before?
I'm good at simulation testing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng closed issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-586904242
I read two error
1. K8sALSServiceMeshHTTPAnalysis, it may be caused by your API server could not response in time.
1. `org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 583;`. Your endpoint is too long, I don't know why.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587232786
> There is nothing about istio. It is related in envoy and k8s api server. We are using it inside the company.
From the log, skywalking received the metrics, but failed to parse.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587232360
i see, Envoy attatched istio, i find other more effective error:
2020-02-18 01:26:04,456 - org.apache.skywalking.oap.server.receiver.envoy.als.K8sALSServiceMeshHTTPAnalysis -38576405 [load-pod-0] INFO [] - Load 40 pods in 80ms
2020-02-18 01:26:04,736 - org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient -38576685 [grpc-default-executor-139] ERROR [] - INTERNAL: Received unexpected EOS on DATA frame from server.
io.grpc.StatusRuntimeException: INTERNAL: Received unexpected EOS on DATA frame from server.
Is there a requirement for envoy specific version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587232638
> There is nothing about istio. It is related in envoy and k8s api server. We are using it inside the company.
i see, Envoy attatched istio, i find other more effective error:
2020-02-18 01:26:04,456 - org.apache.skywalking.oap.server.receiver.envoy.als.K8sALSServiceMeshHTTPAnalysis -38576405 [load-pod-0] INFO [] - Load 40 pods in 80ms
2020-02-18 01:26:04,736 - org.apache.skywalking.oap.server.core.remote.client.GRPCRemoteClient -38576685 [grpc-default-executor-139] ERROR [] - INTERNAL: Received unexpected EOS on DATA frame from server.
io.grpc.StatusRuntimeException: INTERNAL: Received unexpected EOS on DATA frame from server.
Is there a requirement for envoy specific version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587235012
This is the ALS proto we are using, please cross check with your envoy version.
https://github.com/apache/skywalking/blob/master/oap-server/server-receiver-plugin/receiver-proto/src/main/proto/envoy/data/accesslog/v2/accesslog.proto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587292817
We are using this in the production env. There is no issue. I don't know what is your issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] wu-sheng commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-586914652
There is nothing about istio. It is related in envoy and k8s api server. We are using it inside the company.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] justmine66 commented on issue #4376:
K8sALSServiceMeshHTTPAnalysis Transport failed
Posted by GitBox <gi...@apache.org>.
justmine66 commented on issue #4376: K8sALSServiceMeshHTTPAnalysis Transport failed
URL: https://github.com/apache/skywalking/issues/4376#issuecomment-587291006
> This is the ALS proto we are using, please cross check with your envoy version.
>
> https://github.com/apache/skywalking/blob/master/oap-server/server-receiver-plugin/receiver-proto/src/main/proto/envoy/data/accesslog/v2/accesslog.proto
my envoy version also is v2.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services