You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "mochengqian (via GitHub)" <gi...@apache.org> on 2026/06/07 11:13:30 UTC
[PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
mochengqian opened a new pull request, #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965
## Summary
Closes #956.
Every LLM request resolves a `cooldownKey` at least once (more on the retry/fallback path) through `endpointCredentialHash`. That function ran `fmt.Sprintf("%x", sum)` and allocated a fresh 64-byte hex string that escapes to the heap. The hex string is pure overhead — `credentialHash` is only ever used as a comparable component of the `cooldownKey` map key, never displayed or logged.
This PR uses the raw `[32]byte` sha256 output directly as the key component. A `[32]byte` array is comparable and valid in a struct map key, so the hex encoding and its heap allocation are removed while the cooldown identity semantics stay byte-for-byte identical.
`sha256` itself stays per-call (nanoseconds for short keys); the allocation was the dominant cost. Caching the hash on the endpoint was explicitly **not** done — it would pollute the model struct and complicate snapshot cloning for a ~200ns win, as the issue notes.
## Change
- `cooldownKey.credentialHash`: `string` → `[32]byte`
- `endpointCredentialHash(...)`: returns `[32]byte` (zero value for no-credential endpoints), dropping `fmt.Sprintf`
`cooldownKey` is unexported, so this is a purely internal change with no upstream-visible API impact.
## Benchmark
`BenchmarkCooldown_EndpointInCooldown` (added in this PR), 100 LLM endpoints each with an API key, on the hot `endpointInCooldown` path:
| | ns/op | B/op | allocs/op |
|---|---|---|---|
| before | 300.0 | 168 | 6 |
| after | 163.2 | 40 | 3 |
3 fewer allocs/op and ~46% faster, exceeding the issue's "at least one fewer alloc/op" target.
## Test plan
- [x] `go test -race ./pkg/filter/llm/proxy/...` — existing cooldown tests pass unchanged with the `[32]byte` zero value
- [x] `go vet ./pkg/filter/llm/proxy/` clean
- [x] `go build ./...` clean
- [x] New `BenchmarkCooldown_EndpointInCooldown` documents the before/after alloc delta
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "mochengqian (via GitHub)" <gi...@apache.org>.
mochengqian commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3372361221
##########
pkg/filter/llm/proxy/filter_test.go:
##########
@@ -412,3 +412,26 @@ func testLLMEndpoint(id string, port int) *model.Endpoint {
},
}
}
+
+func BenchmarkCooldown_EndpointInCooldown(b *testing.B) {
+ const clusterName = "llm-cooldown-bench"
+ const endpointCount = 100
+ store := newCooldownStore()
+ executor := &RequestExecutor{
+ clusterName: clusterName,
+ cooldowns: store,
+ }
+ endpoints := make([]*model.Endpoint, endpointCount)
+ for i := range endpoints {
+ endpoint := testLLMEndpoint(fmt.Sprintf("ep-%d", i), 19000+i)
+ endpoint.LLMMeta.APIKey = fmt.Sprintf("api-key-%d", i)
+ endpoints[i] = endpoint
Review Comment:
Good catch — fixed in f2aa49be. The benchmark now sets each endpoint's `HealthCheckInterval` to 24h (via a `cooldownTTLMillis` const), so the cooldown TTL far exceeds any realistic `-benchtime`. Every iteration stays on the intended in-cooldown hot path and can't fall through to the delete+log path mid-run. Verified stable at `-benchtime=2s` (~13.5M iters): 164.8 ns/op, 40 B/op, 3 allocs/op.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#issuecomment-4642327149
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-go-pixiu&pullRequest=965) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go-pixiu&pullRequest=965&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go-pixiu&pullRequest=965&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go-pixiu&pullRequest=965)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "mochengqian (via GitHub)" <gi...@apache.org>.
mochengqian commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3392965045
##########
pkg/filter/llm/proxy/filter.go:
##########
@@ -582,12 +582,11 @@ func newCooldownKey(clusterName string, endpoint *model.Endpoint) cooldownKey {
}
}
-func endpointCredentialHash(endpoint *model.Endpoint) string {
Review Comment:
意义是减少热路径 allocation。
原589: sum := sha256.Sum256([]byte(endpoint.LLMMeta.APIKey))
原590: return fmt.Sprintf("%x", sum)
这里64 字节 hex string,通常会分配,要是换成[32]byte,可直接作为map key.
原来的 hex string 只是为了让 hash 可比较,但 Go 的 [32]byte 本来就可比较,也能放进 struct map key。所以 hex 编码这一步不就是纯开销了吗。删掉它可以少一次格式化和 heap allocation。
您要是觉得没意义我就把PR关了 @AlexStocks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "mochengqian (via GitHub)" <gi...@apache.org>.
mochengqian commented on PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#issuecomment-4700384329
已有benchmark:
`BenchmarkCooldown_EndpointInCooldown` (added in this PR), 100 LLM endpoints each with an API key, on the hot `endpointInCooldown` path:
| | ns/op | B/op | allocs/op |
|---|---|---|---|
| before | 300.0 | 168 | 6 |
| after | 163.2 | 40 | 3 |
补充benchmark @AlexStocks :
验证长时间稳定性 (`-benchtime=10s`)输出:
BenchmarkCooldown_EndpointInCooldown-10 71128392 166.6 ns/op 40 B/op 3 allocs/op
结果:71M+迭代运行稳定
验证多核并发 (`-cpu=1,2,4,8`)输出:
BenchmarkCooldown_EndpointInCooldown 6498583 187.3 ns/op 40 B/op 3 allocs/op
BenchmarkCooldown_EndpointInCooldown-2 7174485 167.5 ns/op 40 B/op 3 allocs/op
BenchmarkCooldown_EndpointInCooldown-4 7150099 167.9 ns/op 40 B/op 3 allocs/op
BenchmarkCooldown_EndpointInCooldown-8 7156354 167.5 ns/op 40 B/op 3 allocs/op
结果:2-8核心表现一致
验证内存:
命令: $ go tool pprof -top -alloc_objects mem.out ,输出:
12026039 62.36% SocketAddress.GetAddress
7143532 37.04% fmt.Sprintf
结果:本PR消除的hex编码部分的fmt.Sprintf得到的优化,已验证
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "AlexStocks (via GitHub)" <gi...@apache.org>.
AlexStocks commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3395713975
##########
pkg/filter/llm/proxy/filter.go:
##########
@@ -582,12 +582,11 @@ func newCooldownKey(clusterName string, endpoint *model.Endpoint) cooldownKey {
}
}
-func endpointCredentialHash(endpoint *model.Endpoint) string {
Review Comment:
最好给一个 benchmark 测试结果
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3371159144
##########
pkg/filter/llm/proxy/filter_test.go:
##########
@@ -412,3 +412,26 @@ func testLLMEndpoint(id string, port int) *model.Endpoint {
},
}
}
+
+func BenchmarkCooldown_EndpointInCooldown(b *testing.B) {
+ const clusterName = "llm-cooldown-bench"
+ const endpointCount = 100
+ store := newCooldownStore()
+ executor := &RequestExecutor{
+ clusterName: clusterName,
+ cooldowns: store,
+ }
+ endpoints := make([]*model.Endpoint, endpointCount)
+ for i := range endpoints {
+ endpoint := testLLMEndpoint(fmt.Sprintf("ep-%d", i), 19000+i)
+ endpoint.LLMMeta.APIKey = fmt.Sprintf("api-key-%d", i)
+ endpoints[i] = endpoint
Review Comment:
This benchmark relies on the endpoint cooldown TTL not expiring during the run; with the current 60s `HealthCheckInterval`, running with a long `-benchtime` (or under a slow environment) could cause entries to expire mid-benchmark and shift the measured path (including deletions/logging), making results unstable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "AlexStocks (via GitHub)" <gi...@apache.org>.
AlexStocks merged PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#issuecomment-4645992269
## [Codecov](https://app.codecov.io/gh/apache/dubbo-go-pixiu/pull/965?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 26.14%. Comparing base ([`d2e421d`](https://app.codecov.io/gh/apache/dubbo-go-pixiu/commit/d2e421d99ee5770798ad82f3f1efe3fcf496035f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ba01675`](https://app.codecov.io/gh/apache/dubbo-go-pixiu/commit/ba016750e0fd382245ca48307d43470bfbd2dedc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #965 +/- ##
===========================================
- Coverage 26.14% 26.14% -0.01%
===========================================
Files 275 275
Lines 21981 21980 -1
===========================================
- Hits 5748 5747 -1
Misses 15623 15623
Partials 610 610
```
| [Flag](https://app.codecov.io/gh/apache/dubbo-go-pixiu/pull/965/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [unittests](https://app.codecov.io/gh/apache/dubbo-go-pixiu/pull/965/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.14% <100.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/dubbo-go-pixiu/pull/965?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#issuecomment-4647807213
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-go-pixiu&pullRequest=965) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go-pixiu&pullRequest=965&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go-pixiu&pullRequest=965&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go-pixiu&pullRequest=965)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]
Posted by "AlexStocks (via GitHub)" <gi...@apache.org>.
AlexStocks commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3392669870
##########
pkg/filter/llm/proxy/filter.go:
##########
@@ -582,12 +582,11 @@ func newCooldownKey(clusterName string, endpoint *model.Endpoint) cooldownKey {
}
}
-func endpointCredentialHash(endpoint *model.Endpoint) string {
Review Comment:
没看明白这里改动的意义在哪
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org