You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/10/22 08:33:23 UTC
[GitHub] [skywalking] wu-sheng opened a new issue, #9832: [Bug] [Service Mesh] Fix TCP analysis
wu-sheng opened a new issue, #9832:
URL: https://github.com/apache/skywalking/issues/9832
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.
### Apache SkyWalking Component
OAP server (apache/skywalking)
### What happened
We noticed an Envoy bug reported through https://github.com/envoyproxy/envoy/issues/23579. Generally, this is caused by ALS TCP didn't work as we expected, but only reported when TCP closed.
So, the topology can't show TCP until TCP is closed, and metrics are wrong as the summary got statistics in the time point connection closed.
### What you expected to happen
We expect the TCP ALS could report periodically, which should be fixed through https://github.com/envoyproxy/envoy/pull/23630 by @wbpcode (Thank you)
### How to reproduce
Just observe TCP traffic in Istio mesh
### Anything else
SkyWalking backend should do several things to adopt this fix,
- A new TCP-specific mesh object(ref https://github.com/apache/skywalking-data-collect-protocol/blob/3dc973f859e4dab9521bac7f5d2ede2c4f57c524/service-mesh-probe/service-mesh.proto#L36) should be added.
- Add TCP-specific sources(service, instance, service relation, and instance relation), which are only generated from TCP-specific mesh object
- Metrics in the TCP ALS are counters from the TCP connection established(confirmed by @wbpcode), so we need to build an incrementing mechanism compared to the last report(same pod), and then push the data into the sources for OAL engine. @mrproliu Could you confirm, the ALS route in Satellite is service instance oriented? So one service pod's ALS could be sent to a certain OAP backend.
I hope we could fix this in 9.3.0, of course, this only means the changes on SkyWalking side should be done. Envoy has its roadmap, and we should mention this fix(Envoy fix) in docs and change logs to indicate the fixed version.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1323120875
Moving this to 9.4.0. Envoy side enhancement has been merged, but we still don't have a proper Istio side fix to adopt the new added Envoy TCP ALS mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1290741694
TCP receiver would use connection ID and node ID to calculate the rate/increasement.
This means at least we need to receive ALS TCP twice. Then we could downsampling it to increased value per minute.
Satellite would adjust according to these IDs too, make sure the route target OAP predictable.
ALS connection would be broken and rewrite randomly, if it is occasionally, we just lost a few statistics, generally, we are still good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] toffentoffen commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
toffentoffen commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1290704495
I've also wondered about computing rates, and asked how observed values around metrics, for instance bytes_send, will be calculated. Incrementally, or per duration.
https://github.com/envoyproxy/envoy/pull/23630#discussion_r1003081917
And https://github.com/envoyproxy/envoy/pull/23630#discussion_r1003090727
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #9832: [Bug] [Service Mesh] Fix TCP analysis
URL: https://github.com/apache/skywalking/issues/9832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1287697293
@kezhenxu94 We need to follow that Envoy discussion, it seems a connection ID should be able to use to calculate rate/increasement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1288385478
Like as usual, counter increase would restart, and statistics begin from 2nd round.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1288392080
> Like as usual, counter increase would restart, and statistics begin from 2nd round.
In that case, we may also need something like connection ID.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1294810842
Step one has been done. We are waiting for the envoy side ready to fix others accordingly. @wbpcode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1287698485
> I have confirmed, the satellite route ALS message is not fixed. I could do some updates on the satellite, but one thing I need to be confirmed. How can I get the service instance in the ALS Identifier(https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/accesslog/v3/als.proto#envoy-v3-api-msg-service-accesslog-v3-streamaccesslogsmessage-identifier)?
@kezhenxu94 I think we could use node ID?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] mrproliu commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
mrproliu commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1287691562
I have confirmed, the satellite route ALS message is not fixed. I could do some updates on the satellite, but one thing I need to be confirmed. How can I get the service instance in the ALS Identifier(https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/accesslog/v3/als.proto#envoy-v3-api-msg-service-accesslog-v3-streamaccesslogsmessage-identifier)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1288378145
> * Metrics in the TCP ALS are counters from the TCP connection established(confirmed by @wbpcode), so we need to build an incrementing mechanism compared to the last report(same pod), and then push the data into the sources for OAL engine.
What will be reported if a new connection is established, I think it doesn't contain the previous connections' metrics values, right? If that's the case, using node ID is not enough to build the incrementing mechanism, you have to know the connection ID as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #9832: [Bug] [Service Mesh] Fix TCP analysis
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #9832:
URL: https://github.com/apache/skywalking/issues/9832#issuecomment-1288434246
Yes, that is being discussed on envoy PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org