You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "mochengqian (via GitHub)" <gi...@apache.org> on 2026/06/07 11:13:30 UTC

[PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

mochengqian opened a new pull request, #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965

   ## Summary
   
   Closes #956.
   
   Every LLM request resolves a `cooldownKey` at least once (more on the retry/fallback path) through `endpointCredentialHash`. That function ran `fmt.Sprintf("%x", sum)` and allocated a fresh 64-byte hex string that escapes to the heap. The hex string is pure overhead — `credentialHash` is only ever used as a comparable component of the `cooldownKey` map key, never displayed or logged.
   
   This PR uses the raw `[32]byte` sha256 output directly as the key component. A `[32]byte` array is comparable and valid in a struct map key, so the hex encoding and its heap allocation are removed while the cooldown identity semantics stay byte-for-byte identical.
   
   `sha256` itself stays per-call (nanoseconds for short keys); the allocation was the dominant cost. Caching the hash on the endpoint was explicitly **not** done — it would pollute the model struct and complicate snapshot cloning for a ~200ns win, as the issue notes.
   
   ## Change
   
   - `cooldownKey.credentialHash`: `string` → `[32]byte`
   - `endpointCredentialHash(...)`: returns `[32]byte` (zero value for no-credential endpoints), dropping `fmt.Sprintf`
   
   `cooldownKey` is unexported, so this is a purely internal change with no upstream-visible API impact.
   
   ## Benchmark
   
   `BenchmarkCooldown_EndpointInCooldown` (added in this PR), 100 LLM endpoints each with an API key, on the hot `endpointInCooldown` path:
   
   | | ns/op | B/op | allocs/op |
   |---|---|---|---|
   | before | 300.0 | 168 | 6 |
   | after | 163.2 | 40 | 3 |
   
   3 fewer allocs/op and ~46% faster, exceeding the issue's "at least one fewer alloc/op" target.
   
   ## Test plan
   
   - [x] `go test -race ./pkg/filter/llm/proxy/...` — existing cooldown tests pass unchanged with the `[32]byte` zero value
   - [x] `go vet ./pkg/filter/llm/proxy/` clean
   - [x] `go build ./...` clean
   - [x] New `BenchmarkCooldown_EndpointInCooldown` documents the before/after alloc delta


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "mochengqian (via GitHub)" <gi...@apache.org>.
mochengqian commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3372361221


##########
pkg/filter/llm/proxy/filter_test.go:
##########
@@ -412,3 +412,26 @@ func testLLMEndpoint(id string, port int) *model.Endpoint {
 		},
 	}
 }
+
+func BenchmarkCooldown_EndpointInCooldown(b *testing.B) {
+	const clusterName = "llm-cooldown-bench"
+	const endpointCount = 100
+	store := newCooldownStore()
+	executor := &RequestExecutor{
+		clusterName: clusterName,
+		cooldowns:   store,
+	}
+	endpoints := make([]*model.Endpoint, endpointCount)
+	for i := range endpoints {
+		endpoint := testLLMEndpoint(fmt.Sprintf("ep-%d", i), 19000+i)
+		endpoint.LLMMeta.APIKey = fmt.Sprintf("api-key-%d", i)
+		endpoints[i] = endpoint

Review Comment:
   Good catch — fixed in f2aa49be. The benchmark now sets each endpoint's `HealthCheckInterval` to 24h (via a `cooldownTTLMillis` const), so the cooldown TTL far exceeds any realistic `-benchtime`. Every iteration stays on the intended in-cooldown hot path and can't fall through to the delete+log path mid-run. Verified stable at `-benchtime=2s` (~13.5M iters): 164.8 ns/op, 40 B/op, 3 allocs/op.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#issuecomment-4642327149

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_dubbo-go-pixiu&pullRequest=965) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go-pixiu&pullRequest=965&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go-pixiu&pullRequest=965&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go-pixiu&pullRequest=965)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "mochengqian (via GitHub)" <gi...@apache.org>.
mochengqian commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3392965045


##########
pkg/filter/llm/proxy/filter.go:
##########
@@ -582,12 +582,11 @@ func newCooldownKey(clusterName string, endpoint *model.Endpoint) cooldownKey {
 	}
 }
 
-func endpointCredentialHash(endpoint *model.Endpoint) string {

Review Comment:
   意义是减少热路径 allocation。
   原589: sum := sha256.Sum256([]byte(endpoint.LLMMeta.APIKey))
   原590: return fmt.Sprintf("%x", sum)
   这里64 字节 hex string,通常会分配,要是换成[32]byte,可直接作为map key.
   原来的 hex string 只是为了让 hash 可比较,但 Go 的 [32]byte 本来就可比较,也能放进 struct map key。所以 hex 编码这一步不就是纯开销了吗。删掉它可以少一次格式化和 heap allocation。
   您要是觉得没意义我就把PR关了 @AlexStocks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "mochengqian (via GitHub)" <gi...@apache.org>.
mochengqian commented on PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#issuecomment-4700384329

   已有benchmark:
   `BenchmarkCooldown_EndpointInCooldown` (added in this PR), 100 LLM endpoints each with an API key, on the hot `endpointInCooldown` path:
   
   | | ns/op | B/op | allocs/op |
   |---|---|---|---|
   | before | 300.0 | 168 | 6 |
   | after | 163.2 | 40 | 3 |
   
   补充benchmark @AlexStocks :
   验证长时间稳定性 (`-benchtime=10s`)输出:
   BenchmarkCooldown_EndpointInCooldown-10   71128392   166.6 ns/op   40 B/op   3 allocs/op
   结果:71M+迭代运行稳定
   
   验证多核并发 (`-cpu=1,2,4,8`)输出:
   BenchmarkCooldown_EndpointInCooldown     6498583   187.3 ns/op   40 B/op   3 allocs/op
   BenchmarkCooldown_EndpointInCooldown-2   7174485   167.5 ns/op   40 B/op   3 allocs/op
   BenchmarkCooldown_EndpointInCooldown-4   7150099   167.9 ns/op   40 B/op   3 allocs/op
   BenchmarkCooldown_EndpointInCooldown-8   7156354   167.5 ns/op   40 B/op   3 allocs/op
   结果:2-8核心表现一致
   
   验证内存:
   命令: $ go tool pprof -top -alloc_objects mem.out ,输出:
    
    12026039 62.36%  SocketAddress.GetAddress
      7143532 37.04%  fmt.Sprintf  
   
   结果:本PR消除的hex编码部分的fmt.Sprintf得到的优化,已验证


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "AlexStocks (via GitHub)" <gi...@apache.org>.
AlexStocks commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3395713975


##########
pkg/filter/llm/proxy/filter.go:
##########
@@ -582,12 +582,11 @@ func newCooldownKey(clusterName string, endpoint *model.Endpoint) cooldownKey {
 	}
 }
 
-func endpointCredentialHash(endpoint *model.Endpoint) string {

Review Comment:
   最好给一个 benchmark 测试结果



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3371159144


##########
pkg/filter/llm/proxy/filter_test.go:
##########
@@ -412,3 +412,26 @@ func testLLMEndpoint(id string, port int) *model.Endpoint {
 		},
 	}
 }
+
+func BenchmarkCooldown_EndpointInCooldown(b *testing.B) {
+	const clusterName = "llm-cooldown-bench"
+	const endpointCount = 100
+	store := newCooldownStore()
+	executor := &RequestExecutor{
+		clusterName: clusterName,
+		cooldowns:   store,
+	}
+	endpoints := make([]*model.Endpoint, endpointCount)
+	for i := range endpoints {
+		endpoint := testLLMEndpoint(fmt.Sprintf("ep-%d", i), 19000+i)
+		endpoint.LLMMeta.APIKey = fmt.Sprintf("api-key-%d", i)
+		endpoints[i] = endpoint

Review Comment:
   This benchmark relies on the endpoint cooldown TTL not expiring during the run; with the current 60s `HealthCheckInterval`, running with a long `-benchtime` (or under a slow environment) could cause entries to expire mid-benchmark and shift the measured path (including deletions/logging), making results unstable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "AlexStocks (via GitHub)" <gi...@apache.org>.
AlexStocks merged PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#issuecomment-4645992269

   ## [Codecov](https://app.codecov.io/gh/apache/dubbo-go-pixiu/pull/965?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 26.14%. Comparing base ([`d2e421d`](https://app.codecov.io/gh/apache/dubbo-go-pixiu/commit/d2e421d99ee5770798ad82f3f1efe3fcf496035f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ba01675`](https://app.codecov.io/gh/apache/dubbo-go-pixiu/commit/ba016750e0fd382245ca48307d43470bfbd2dedc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff             @@
   ##           develop     #965      +/-   ##
   ===========================================
   - Coverage    26.14%   26.14%   -0.01%     
   ===========================================
     Files          275      275              
     Lines        21981    21980       -1     
   ===========================================
   - Hits          5748     5747       -1     
     Misses       15623    15623              
     Partials       610      610              
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/dubbo-go-pixiu/pull/965/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [unittests](https://app.codecov.io/gh/apache/dubbo-go-pixiu/pull/965/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.14% <100.00%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/dubbo-go-pixiu/pull/965?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#issuecomment-4647807213

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_dubbo-go-pixiu&pullRequest=965) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go-pixiu&pullRequest=965&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go-pixiu&pullRequest=965&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go-pixiu&pullRequest=965&metric=new_duplicated_lines_density&view=list)  
     
   <!-- sqra-placement-anchor -->
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go-pixiu&pullRequest=965)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org


Re: [PR] perf(llm): drop per-request hex alloc in cooldown credential hash [dubbo-go-pixiu]

Posted by "AlexStocks (via GitHub)" <gi...@apache.org>.
AlexStocks commented on code in PR #965:
URL: https://github.com/apache/dubbo-go-pixiu/pull/965#discussion_r3392669870


##########
pkg/filter/llm/proxy/filter.go:
##########
@@ -582,12 +582,11 @@ func newCooldownKey(clusterName string, endpoint *model.Endpoint) cooldownKey {
 	}
 }
 
-func endpointCredentialHash(endpoint *model.Endpoint) string {

Review Comment:
   没看明白这里改动的意义在哪



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org