You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by GitBox <gi...@apache.org> on 2020/10/07 17:39:56 UTC
[GitHub] [thrift] fishy opened a new pull request #2256: THRIFT-5240: Disable go server side connectivity check by default
fishy opened a new pull request #2256:
URL: https://github.com/apache/thrift/pull/2256
Client: go
This is a follow up to 4db7a0af13ac9614e3e9758d42b2791040f4dc7e.
Due to the go runtime bug [1], the 1ms ticker used by this feature could
cause excessive cpu usage. So until the bug is fixed in go it's better
to leave this behavior disabled by default. The instructions in the
README file has been updated to how to enable this feature if needed.
[1] https://github.com/golang/go/issues/27707
<!-- Explain the changes in the pull request below: -->
<!-- We recommend you review the checklist/tips before submitting a pull request. -->
- [x] Did you create an [Apache Jira](https://issues.apache.org/jira/projects/THRIFT/issues/) ticket? (not required for trivial changes)
- [x] If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
- [x] Did you squash your changes to a single commit? (not required, but preferred)
- [x] Did you do your best to avoid breaking changes? If one was needed, did you label the Jira ticket with "Breaking-Change"?
- [ ] If your change does not involve any code, include `[skip ci]` anywhere in the commit message to free up build resources.
<!--
The Contributing Guide at:
https://github.com/apache/thrift/blob/master/CONTRIBUTING.md
has more details and tips for committing properly.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] fishy merged pull request #2256: THRIFT-5240: Tweak the default go server connectivity check interval
Posted by GitBox <gi...@apache.org>.
fishy merged pull request #2256:
URL: https://github.com/apache/thrift/pull/2256
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] fishy commented on pull request #2256: THRIFT-5240: Disable go server side connectivity check by default
Posted by GitBox <gi...@apache.org>.
fishy commented on pull request #2256:
URL: https://github.com/apache/thrift/pull/2256#issuecomment-705140409
2ms it's ~13% and 3ms it's ~9%. Also all these tests are done on go1.15.2 on linux/amd64.
I think around 10% cpu might still be too high as the default value. I think if we decided to not disable it by default, then the default value should be something like 5ms or 10ms, so that it doesn't auto add too much overhead, and still being useful for the majority of cases. For services with very low latency endpoints (like yours), they just need to change the default value, either:
1. Disable it (set it to 0) if they don't want those excessive cpu usage
or
2. Set it to a smaller number (1ms/2ms/3ms) to balance between the extra cpu usage and able to abandon requests early when it's no longer needed.
I think even for a 5ms p50 service, the 5ms default value can still be helpful for the long tails?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] dcelasun commented on pull request #2256: THRIFT-5240: Disable go server side connectivity check by default
Posted by GitBox <gi...@apache.org>.
dcelasun commented on pull request #2256:
URL: https://github.com/apache/thrift/pull/2256#issuecomment-705135142
I'm not so sure about a 5ms default. We have a fairly loaded Go 1.15 server (so with the ticker optimization) with 5ms/8ms p50/p75 latency respectively and a fixed 5ms increase seems a bit too much.
Would there be enough cpu load reduction with a 2-3ms ticker?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] fishy edited a comment on pull request #2256: THRIFT-5240: Disable go server side connectivity check by default
Posted by GitBox <gi...@apache.org>.
fishy edited a comment on pull request #2256:
URL: https://github.com/apache/thrift/pull/2256#issuecomment-705101786
Actually dig deeper to the go runtime bug, it looks like the excessive cpu only happens when the ticker interval is short (1ms). using the example code given by that bug, using 1ms it causes ~20% cpu, but changing that to 5ms makes the cpu drop to ~6%, and 10ms makes it drop to ~3%.
So maybe 5ms is a better default than 0? @dcelasun what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] fishy commented on pull request #2256: THRIFT-5240: Disable go server side connectivity check by default
Posted by GitBox <gi...@apache.org>.
fishy commented on pull request #2256:
URL: https://github.com/apache/thrift/pull/2256#issuecomment-705101786
Actually dig deeper to the go runtime bug, it looks like the excessive cpu only happens when the ticker interval is short (1ms). using the example code given by that bug, using 1ms it causes ~20% cpu, but changing that to 5ms makes the cpu drop to ~5%, and 10ms makes it drop to ~3%.
So maybe 5ms is a better default than 0? @dcelasun what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] dcelasun commented on pull request #2256: THRIFT-5240: Disable go server side connectivity check by default
Posted by GitBox <gi...@apache.org>.
dcelasun commented on pull request #2256:
URL: https://github.com/apache/thrift/pull/2256#issuecomment-705142473
> I think even for a 5ms p50 service, the 5ms default value can still be helpful for the long tails?
Yeah that's probably true. Let's go with 5ms then :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] fishy merged pull request #2256: THRIFT-5240: Tweak the default go server connectivity check interval
Posted by GitBox <gi...@apache.org>.
fishy merged pull request #2256:
URL: https://github.com/apache/thrift/pull/2256
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org