You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "AsperforMias (via GitHub)" <gi...@apache.org> on 2026/08/09 09:36:48 UTC
[PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
AsperforMias opened a new pull request, #3625:
URL: https://github.com/apache/dubbo-go/pull/3625
### Description
Fixes #3615
**Problem.** With application-level service discovery, a single transient `MetadataService` fetch failure during a provider restart could leave the consumer directory permanently empty (`No provider available`) until the consumer was restarted. Root cause chain, confirmed by a deterministic docker reproduction (details in [this comment](https://github.com/apache/dubbo-go/issues/3615#issuecomment-5226228151)):
1. `ServiceInstancesChangedListenerImpl.OnEvent` skipped instances whose metadata fetch failed, overwrote the service URLs with the (possibly empty) result, and returned success. No retry queue, no error propagation — and since Nacos only pushes on instance-list *changes*, no later event ever retried the fetch.
2. Even when a later push did re-fetch metadata successfully, `RegistryDirectory` refused to rebuild the invoker: the graceful shutdown of the provider had left a **closing tombstone** on the instance key (default TTL 30s), and a genuine same-address restart was indistinguishable from a stale pre-shutdown registry snapshot.
**Fix (two layers, both required — verified by e2e):**
- **Metadata retry** (`service_instances_changed_listener_impl.go`): failed revisions are tracked in `unresolvedRevisions` and re-resolved by replaying the latest instance snapshot through the existing build path, using one shared timer per listener with capped exponential backoff (1s→30s, 25% jitter, **unlimited attempts** — a retry cap would re-introduce the permanent failure this fixes). Retries stop naturally when instances leave the snapshot, when the last subscriber is removed, or on registry `Destroy()` (`service_discovery_registry.go`). Metadata RPCs no longer run under the listener state mutex (build is serialized by a separate `buildMu`; the state mutex only guards field access), so a slow/dead provider cannot block event processing.
- **Tombstone restart discrimination** (`directory.go`): the closing tombstone now records the closing invoker's export `timestamp`. A re-add carrying the *same* timestamp is a stale pre-shutdown snapshot and stays vetoed (existing graceful-shutdown semantics unchanged); a *different* timestamp proves a genuine restart, so the tombstone is cleared and the invoker rebuilt immediately. URLs without a timestamp (older providers) keep the conservative veto.
**Known limitation:** export `timestamp` is second-precision in dubbo-go (`server/action.go`), so a stop+restart+export+register completing within the same second would still be vetoed (degrading to pre-fix behavior until the next registry event). This is practically unreachable — graceful stop alone takes seconds. If it ever matters, the principled follow-up is discriminating by metadata `revision` or aligning the timestamp to milliseconds (Java Dubbo uses `System.currentTimeMillis()`), which would be a separate, framework-wide discussion.
**Out of scope (tracked separately):** application-level `AddListener` failure is likewise never retried — same recovery-gap family, different layer; see #3624. Observability for unresolved revisions (e.g. a gauge) belongs to #3356.
**Verification:**
- New unit tests (`metadata_retry_test.go`, `directory_test.go`): recovery without any further registry event; retries stop when the instance is removed; only the latest revision is retried; one shared timer across repeated events; timer cancelled on subscriber detach and resumed on re-attach; tombstone vetoes stale re-adds but allows genuine restarts.
- `go test ./registry/...` and the listener/directory tests with `-race` pass; broader regression over `registry/metadata/cluster/server/client/common` passes.
- E2E (docker compose: Nacos 2.5.1 + single Triple provider + consumer, application-level discovery, local metadata): with traffic to the provider blocked during its restart, the consumer failed metadata fetches as before, but **self-healed seconds after unblocking without a consumer restart**, followed by a 30s steady-state check with all calls succeeding. The same scenario on v3.3.2 stays at `No provider available` permanently.
### Checklist
- [x] I confirm the target branch is `develop`
- [x] Code has passed local testing
- [x] I have added tests that prove my fix is effective or that my feature works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "Alanxtl (via GitHub)" <gi...@apache.org>.
Alanxtl commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5351829750
pls fix conflict
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5235767980
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5231046952
## [Codecov](https://app.codecov.io/gh/apache/dubbo-go/pull/3625?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `90.00000%` with `12 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 53.95%. Comparing base ([`60d1c2a`](https://app.codecov.io/gh/apache/dubbo-go/commit/60d1c2a949f0ee0da4be3fe09fb79d295491e040?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a2e3561`](https://app.codecov.io/gh/apache/dubbo-go/commit/a2e3561617743b310b7dc1afa15b34b21def75e6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 900 commits behind head on develop.
| [Files with missing lines](https://app.codecov.io/gh/apache/dubbo-go/pull/3625?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...scovery/service\_instances\_changed\_listener\_impl.go](https://app.codecov.io/gh/apache/dubbo-go/pull/3625?src=pr&el=tree&filepath=registry%2Fservicediscovery%2Fservice_instances_changed_listener_impl.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cmVnaXN0cnkvc2VydmljZWRpc2NvdmVyeS9zZXJ2aWNlX2luc3RhbmNlc19jaGFuZ2VkX2xpc3RlbmVyX2ltcGwuZ28=) | 92.00% | [7 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/dubbo-go/pull/3625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...try/servicediscovery/service\_discovery\_registry.go](https://app.codecov.io/gh/apache/dubbo-go/pull/3625?src=pr&el=tree&filepath=registry%2Fservicediscovery%2Fservice_discovery_registry.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cmVnaXN0cnkvc2VydmljZWRpc2NvdmVyeS9zZXJ2aWNlX2Rpc2NvdmVyeV9yZWdpc3RyeS5nbw==) | 40.00% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/dubbo-go/pull/3625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [registry/directory/directory.go](https://app.codecov.io/gh/apache/dubbo-go/pull/3625?src=pr&el=tree&filepath=registry%2Fdirectory%2Fdirectory.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cmVnaXN0cnkvZGlyZWN0b3J5L2RpcmVjdG9yeS5nbw==) | 93.33% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/dubbo-go/pull/3625?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## develop #3625 +/- ##
===========================================
+ Coverage 46.76% 53.95% +7.18%
===========================================
Files 295 474 +179
Lines 17172 36479 +19307
===========================================
+ Hits 8031 19681 +11650
- Misses 8287 15221 +6934
- Partials 854 1577 +723
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/dubbo-go/pull/3625?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5230945312
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "Alanxtl (via GitHub)" <gi...@apache.org>.
Alanxtl commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3746173939
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -91,19 +105,40 @@ func (lstn *ServiceInstancesChangedListenerImpl) OnEvent(e observer.Event) error
return nil
}
- lstn.mutex.Lock()
- defer lstn.mutex.Unlock()
+ logger.Infof("[Registry][ServiceDiscovery] received instance notification event, service=%s size=%d", ce.ServiceName, len(ce.Instances))
+ lstn.mutex.Lock()
lstn.allInstances[ce.ServiceName] = ce.Instances
- revisionToInstances := make(map[string][]registry.ServiceInstance, len(lstn.revisionToMetadata))
- newRevisionToMetadata := make(map[string]*info.MetadataInfo, len(lstn.revisionToMetadata))
+ lstn.mutex.Unlock()
+
+ lstn.refreshServiceURLs()
+ return nil
+}
+
+// refreshServiceURLs rebuilds service URLs from the latest instance snapshot and
+// notifies subscribers. The build is serialized by buildMu, but lstn.mutex is
+// only held while reading or committing in-memory state: metadata RPCs run in
+// between without it, so a slow or unreachable provider cannot block event
+// processing or retry scheduling.
+func (lstn *ServiceInstancesChangedListenerImpl) refreshServiceURLs() {
Review Comment:
这个函数返回一个ok是不是更好
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -91,19 +105,40 @@ func (lstn *ServiceInstancesChangedListenerImpl) OnEvent(e observer.Event) error
return nil
}
- lstn.mutex.Lock()
- defer lstn.mutex.Unlock()
+ logger.Infof("[Registry][ServiceDiscovery] received instance notification event, service=%s size=%d", ce.ServiceName, len(ce.Instances))
Review Comment:
改成debug等级吧
##########
registry/directory/directory.go:
##########
@@ -645,24 +649,30 @@ func (dir *RegistryDirectory) markClosingTombstone(instanceKey string, invoker p
if invoker != nil && invoker.GetURL() != nil {
tombstone.ServiceKey = invoker.GetURL().ServiceKey()
tombstone.Address = invoker.GetURL().Location
+ tombstone.Timestamp = invoker.GetURL().GetParam(constant.TimestampKey, "")
}
dir.closingTombstones.Store(instanceKey, tombstone)
}
func (dir *RegistryDirectory) hasActiveClosingTombstone(instanceKey string) bool {
+ _, ok := dir.activeClosingTombstone(instanceKey)
+ return ok
+}
+
+func (dir *RegistryDirectory) activeClosingTombstone(instanceKey string) (closingTombstone, bool) {
if instanceKey == "" {
- return false
+ return closingTombstone{}, false
}
tombstoneValue, ok := dir.closingTombstones.Load(instanceKey)
if !ok {
- return false
+ return closingTombstone{}, false
}
tombstone := tombstoneValue.(closingTombstone)
if time.Now().After(tombstone.ExpireAt) {
dir.closingTombstones.Delete(instanceKey)
- return false
+ return closingTombstone{}, false
}
- return true
+ return tombstone, true
}
Review Comment:
hasActiveClosingTombstone现在也没有调用者 而且还是个内部函数 直接删了不就行了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5230841033
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "Alanxtl (via GitHub)" <gi...@apache.org>.
Alanxtl merged PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5354299336
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AsperforMias (via GitHub)" <gi...@apache.org>.
AsperforMias commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3746282370
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -91,19 +105,40 @@ func (lstn *ServiceInstancesChangedListenerImpl) OnEvent(e observer.Event) error
return nil
}
- lstn.mutex.Lock()
- defer lstn.mutex.Unlock()
+ logger.Infof("[Registry][ServiceDiscovery] received instance notification event, service=%s size=%d", ce.ServiceName, len(ce.Instances))
+ lstn.mutex.Lock()
lstn.allInstances[ce.ServiceName] = ce.Instances
- revisionToInstances := make(map[string][]registry.ServiceInstance, len(lstn.revisionToMetadata))
- newRevisionToMetadata := make(map[string]*info.MetadataInfo, len(lstn.revisionToMetadata))
+ lstn.mutex.Unlock()
+
+ lstn.refreshServiceURLs()
+ return nil
+}
+
+// refreshServiceURLs rebuilds service URLs from the latest instance snapshot and
+// notifies subscribers. The build is serialized by buildMu, but lstn.mutex is
+// only held while reading or committing in-memory state: metadata RPCs run in
+// between without it, so a slow or unreachable provider cannot block event
+// processing or retry scheduling.
+func (lstn *ServiceInstancesChangedListenerImpl) refreshServiceURLs() {
Review Comment:
然后要改的话,相对应的,`require.NoError`断言这边也要一起改(在测试那边)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AsperforMias (via GitHub)" <gi...@apache.org>.
AsperforMias commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3746598171
##########
registry/servicediscovery/service_discovery_registry.go:
##########
@@ -323,6 +323,15 @@ func (s *serviceDiscoveryRegistry) IsAvailable() bool {
func (s *serviceDiscoveryRegistry) Destroy() {
s.stopMetadataTimers()
+ s.lock.Lock()
+ for _, l := range s.serviceListeners {
+ // Destroy drops listeners without RemoveListener; cancel any pending
+ // metadata retry so its timer cannot leak.
+ if impl, ok := l.(*ServiceInstancesChangedListenerImpl); ok {
+ impl.stopMetadataRetry()
+ }
+ }
+ s.lock.Unlock()
err := s.serviceDiscovery.Destroy()
if err != nil {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AlexStocks (via GitHub)" <gi...@apache.org>.
AlexStocks commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3819238019
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -343,3 +409,94 @@ func GetMetadataInfo(app string, instance registry.ServiceInstance, revision str
metaCache.Set(cacheKey, metadataInfo)
return metadataInfo, nil
}
+
+var (
+ // metadataRetryInitialDelay is the first backoff delay before retrying a failed
+ // metadata fetch. Package-level so tests can shrink it.
+ metadataRetryInitialDelay = time.Second
+ // metadataRetryMaxDelay caps the backoff. The retry count itself is
+ // intentionally unlimited: retries only target instances the registry still
+ // reports as alive, and a capped count would re-introduce the permanent
+ // empty-directory failure this mechanism fixes.
+ metadataRetryMaxDelay = 30 * time.Second
+ // metadataFetchFailureLogInterval throttles repeated fetch-failure warnings
+ // for the same revision key.
+ metadataFetchFailureLogInterval = 5 * time.Minute
+)
+
+// metadataInfoFetcher resolves MetadataInfo for a revision; a package-level
+// indirection so tests can inject transient failures.
+var metadataInfoFetcher = GetMetadataInfo
+
+// stopMetadataRetry cancels any pending metadata retry. It is called when the
+// owning registry is destroyed and drops this listener without RemoveListener,
+// so the retry timer cannot leak.
+func (lstn *ServiceInstancesChangedListenerImpl) stopMetadataRetry() {
+ lstn.mutex.Lock()
+ defer lstn.mutex.Unlock()
+ if lstn.retryTimer != nil {
+ lstn.retryTimer.Stop()
+ lstn.retryTimer = nil
+ }
+}
+
+// scheduleMetadataRetry arms the shared retry timer while unresolved revisions
+// remain. Retries replay the latest instance snapshot, so revisions whose
+// instances disappeared from the registry are dropped naturally on the next run.
+func (lstn *ServiceInstancesChangedListenerImpl) scheduleMetadataRetry() {
+ lstn.mutex.Lock()
+ defer lstn.mutex.Unlock()
+ if len(lstn.unresolvedRevisions) == 0 {
+ if lstn.retryTimer != nil {
+ lstn.retryTimer.Stop()
+ lstn.retryTimer = nil
+ }
+ lstn.retryAttempts = 0
+ return
+ }
+ if lstn.retryTimer != nil {
+ // One shared timer per listener: repeated events must not multiply retries.
+ return
+ }
Review Comment:
[P1] 当前 Head `b1235e19` 仍未修复这个公共入口:`SubscribeURL` 继续用 `url.ServiceKey() + ":" + protocol` 写入 listener,而 `UnSubscribe` 仍在 `service_discovery_registry.go:230` 用裸 `url.ServiceKey()` 删除。当前 Head 的公开入口探针在订阅后制造 unresolved revision 和 retry timer,再调用 `UnSubscribe`,结果 `listeners` 仍保留 `org.apache.dubbo.test.TestService:dubbo`,timer 也不会走“最后订阅者移除”分支;探针因此失败。仅把删除键改为与订阅端一致的 protocol-qualified key 后,同一 `-race` 探针通过。请复用同一个 key helper,并把这条 `SubscribeURL -> UnSubscribe` 回归补进正式测试。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5242369719
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AsperforMias (via GitHub)" <gi...@apache.org>.
AsperforMias commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3746278286
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -91,19 +105,40 @@ func (lstn *ServiceInstancesChangedListenerImpl) OnEvent(e observer.Event) error
return nil
}
- lstn.mutex.Lock()
- defer lstn.mutex.Unlock()
+ logger.Infof("[Registry][ServiceDiscovery] received instance notification event, service=%s size=%d", ce.ServiceName, len(ce.Instances))
+ lstn.mutex.Lock()
lstn.allInstances[ce.ServiceName] = ce.Instances
- revisionToInstances := make(map[string][]registry.ServiceInstance, len(lstn.revisionToMetadata))
- newRevisionToMetadata := make(map[string]*info.MetadataInfo, len(lstn.revisionToMetadata))
+ lstn.mutex.Unlock()
+
+ lstn.refreshServiceURLs()
+ return nil
+}
+
+// refreshServiceURLs rebuilds service URLs from the latest instance snapshot and
+// notifies subscribers. The build is serialized by buildMu, but lstn.mutex is
+// only held while reading or committing in-memory state: metadata RPCs run in
+// between without it, so a slow or unreachable provider cannot block event
+// processing or retry scheduling.
+func (lstn *ServiceInstancesChangedListenerImpl) refreshServiceURLs() {
Review Comment:
这里有个问题,因为实际上,`refreshServiceURLs()` 不返回任何东西,也就是内部算出"哪些 revision 的 metadata 没解析出来",自己存状态、自己调度重试。那这里返回的ok 要用于让 OnEvent 返回 err,这也意味着要改变 OnEvent 原有契约(之前 metadata 失败是"跳过该实例但不报错",现在加重试了)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5231022399
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AsperforMias (via GitHub)" <gi...@apache.org>.
AsperforMias commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5231089508
Note on CI: the `Integration Test` failure is in `observability/integration` ("Prometheus scrape targets are up"), which is unrelated to this change — the same check fails identically on two other PRs today ([run 31297924714](https://github.com/apache/dubbo-go/actions/runs/31297924714), [run 31301434603](https://github.com/apache/dubbo-go/actions/runs/31301434603)). In this PR's run, the sample's RPCs, traces, and metric series all verified fine; only the Prometheus scrape-target liveness check failed. All code-level checks (Unit Test, Lint, check-fmt, license, CodeQL, RISC-V build, SonarCloud) pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AsperforMias (via GitHub)" <gi...@apache.org>.
AsperforMias commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3746282370
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -91,19 +105,40 @@ func (lstn *ServiceInstancesChangedListenerImpl) OnEvent(e observer.Event) error
return nil
}
- lstn.mutex.Lock()
- defer lstn.mutex.Unlock()
+ logger.Infof("[Registry][ServiceDiscovery] received instance notification event, service=%s size=%d", ce.ServiceName, len(ce.Instances))
+ lstn.mutex.Lock()
lstn.allInstances[ce.ServiceName] = ce.Instances
- revisionToInstances := make(map[string][]registry.ServiceInstance, len(lstn.revisionToMetadata))
- newRevisionToMetadata := make(map[string]*info.MetadataInfo, len(lstn.revisionToMetadata))
+ lstn.mutex.Unlock()
+
+ lstn.refreshServiceURLs()
+ return nil
+}
+
+// refreshServiceURLs rebuilds service URLs from the latest instance snapshot and
+// notifies subscribers. The build is serialized by buildMu, but lstn.mutex is
+// only held while reading or committing in-memory state: metadata RPCs run in
+// between without it, so a slow or unreachable provider cannot block event
+// processing or retry scheduling.
+func (lstn *ServiceInstancesChangedListenerImpl) refreshServiceURLs() {
Review Comment:
然后要改的话,相对应的,require.NoError 断言这边也要一起改
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5235689549
## [](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_dubbo-go&pullRequest=3625&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_dubbo-go&pullRequest=3625&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_dubbo-go&pullRequest=3625&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_dubbo-go&pullRequest=3625)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "Alanxtl (via GitHub)" <gi...@apache.org>.
Alanxtl commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3746534115
##########
registry/servicediscovery/service_discovery_registry.go:
##########
@@ -323,6 +323,15 @@ func (s *serviceDiscoveryRegistry) IsAvailable() bool {
func (s *serviceDiscoveryRegistry) Destroy() {
s.stopMetadataTimers()
+ s.lock.Lock()
+ for _, l := range s.serviceListeners {
+ // Destroy drops listeners without RemoveListener; cancel any pending
+ // metadata retry so its timer cannot leak.
+ if impl, ok := l.(*ServiceInstancesChangedListenerImpl); ok {
+ impl.stopMetadataRetry()
+ }
+ }
+ s.lock.Unlock()
err := s.serviceDiscovery.Destroy()
if err != nil {
Review Comment:
`Destroy()` 不能保证阻止正在执行的重试重新创建 timer
如果 timer callback 已经把 `retryTimer` 置空,或 `refreshServiceURLs()` 正在进行 metadata RPC,`Destroy()` 调用 `stopMetadataRetry()` 后,刷新流程仍会执行到:
```go
lstn.scheduleMetadataRetry()
```
于是 registry 已销毁后又创建新的无限重试 timer。需要增加 `closed/destroyed` 状态或 generation 检查,并在刷新完成后再次确认 listener 未销毁。
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -343,3 +409,94 @@ func GetMetadataInfo(app string, instance registry.ServiceInstance, revision str
metaCache.Set(cacheKey, metadataInfo)
return metadataInfo, nil
}
+
+var (
+ // metadataRetryInitialDelay is the first backoff delay before retrying a failed
+ // metadata fetch. Package-level so tests can shrink it.
+ metadataRetryInitialDelay = time.Second
+ // metadataRetryMaxDelay caps the backoff. The retry count itself is
+ // intentionally unlimited: retries only target instances the registry still
+ // reports as alive, and a capped count would re-introduce the permanent
+ // empty-directory failure this mechanism fixes.
+ metadataRetryMaxDelay = 30 * time.Second
+ // metadataFetchFailureLogInterval throttles repeated fetch-failure warnings
+ // for the same revision key.
+ metadataFetchFailureLogInterval = 5 * time.Minute
+)
+
+// metadataInfoFetcher resolves MetadataInfo for a revision; a package-level
+// indirection so tests can inject transient failures.
+var metadataInfoFetcher = GetMetadataInfo
+
+// stopMetadataRetry cancels any pending metadata retry. It is called when the
+// owning registry is destroyed and drops this listener without RemoveListener,
+// so the retry timer cannot leak.
+func (lstn *ServiceInstancesChangedListenerImpl) stopMetadataRetry() {
+ lstn.mutex.Lock()
+ defer lstn.mutex.Unlock()
+ if lstn.retryTimer != nil {
+ lstn.retryTimer.Stop()
+ lstn.retryTimer = nil
+ }
+}
+
+// scheduleMetadataRetry arms the shared retry timer while unresolved revisions
+// remain. Retries replay the latest instance snapshot, so revisions whose
+// instances disappeared from the registry are dropped naturally on the next run.
+func (lstn *ServiceInstancesChangedListenerImpl) scheduleMetadataRetry() {
+ lstn.mutex.Lock()
+ defer lstn.mutex.Unlock()
+ if len(lstn.unresolvedRevisions) == 0 {
+ if lstn.retryTimer != nil {
+ lstn.retryTimer.Stop()
+ lstn.retryTimer = nil
+ }
+ lstn.retryAttempts = 0
+ return
+ }
+ if lstn.retryTimer != nil {
+ // One shared timer per listener: repeated events must not multiply retries.
+ return
+ }
Review Comment:
订阅者移除后,重试 timer 仍可能无限重建
`scheduleMetadataRetry()` 只检查 `unresolvedRevisions`,没有检查 `len(listeners)`。因此:
- 初始 `OnEvent` 在 listener 安装前失败,会先启动 timer;
- 并发订阅中被丢弃的临时 listener 可能永久重试;
- `RemoveListener` 与 timer callback 竞争时,callback 仍会在无订阅者后重新调度。
结果是持续 metadata RPC、日志输出,并通过 timer closure 持有 listener。建议调度和 callback 中都检查 listener 是否仍存在,并为被丢弃的临时 listener 增加清理路径。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AsperforMias (via GitHub)" <gi...@apache.org>.
AsperforMias commented on PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#issuecomment-5354410959
> pls fix conflict
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AsperforMias (via GitHub)" <gi...@apache.org>.
AsperforMias commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3746597017
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -343,3 +409,94 @@ func GetMetadataInfo(app string, instance registry.ServiceInstance, revision str
metaCache.Set(cacheKey, metadataInfo)
return metadataInfo, nil
}
+
+var (
+ // metadataRetryInitialDelay is the first backoff delay before retrying a failed
+ // metadata fetch. Package-level so tests can shrink it.
+ metadataRetryInitialDelay = time.Second
+ // metadataRetryMaxDelay caps the backoff. The retry count itself is
+ // intentionally unlimited: retries only target instances the registry still
+ // reports as alive, and a capped count would re-introduce the permanent
+ // empty-directory failure this mechanism fixes.
+ metadataRetryMaxDelay = 30 * time.Second
+ // metadataFetchFailureLogInterval throttles repeated fetch-failure warnings
+ // for the same revision key.
+ metadataFetchFailureLogInterval = 5 * time.Minute
+)
+
+// metadataInfoFetcher resolves MetadataInfo for a revision; a package-level
+// indirection so tests can inject transient failures.
+var metadataInfoFetcher = GetMetadataInfo
+
+// stopMetadataRetry cancels any pending metadata retry. It is called when the
+// owning registry is destroyed and drops this listener without RemoveListener,
+// so the retry timer cannot leak.
+func (lstn *ServiceInstancesChangedListenerImpl) stopMetadataRetry() {
+ lstn.mutex.Lock()
+ defer lstn.mutex.Unlock()
+ if lstn.retryTimer != nil {
+ lstn.retryTimer.Stop()
+ lstn.retryTimer = nil
+ }
+}
+
+// scheduleMetadataRetry arms the shared retry timer while unresolved revisions
+// remain. Retries replay the latest instance snapshot, so revisions whose
+// instances disappeared from the registry are dropped naturally on the next run.
+func (lstn *ServiceInstancesChangedListenerImpl) scheduleMetadataRetry() {
+ lstn.mutex.Lock()
+ defer lstn.mutex.Unlock()
+ if len(lstn.unresolvedRevisions) == 0 {
+ if lstn.retryTimer != nil {
+ lstn.retryTimer.Stop()
+ lstn.retryTimer = nil
+ }
+ lstn.retryAttempts = 0
+ return
+ }
+ if lstn.retryTimer != nil {
+ // One shared timer per listener: repeated events must not multiply retries.
+ return
+ }
Review Comment:
增加listener生命周期状态(closed)和互斥锁保护
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AsperforMias (via GitHub)" <gi...@apache.org>.
AsperforMias commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3746314586
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -91,19 +105,40 @@ func (lstn *ServiceInstancesChangedListenerImpl) OnEvent(e observer.Event) error
return nil
}
- lstn.mutex.Lock()
- defer lstn.mutex.Unlock()
+ logger.Infof("[Registry][ServiceDiscovery] received instance notification event, service=%s size=%d", ce.ServiceName, len(ce.Instances))
+ lstn.mutex.Lock()
lstn.allInstances[ce.ServiceName] = ce.Instances
- revisionToInstances := make(map[string][]registry.ServiceInstance, len(lstn.revisionToMetadata))
- newRevisionToMetadata := make(map[string]*info.MetadataInfo, len(lstn.revisionToMetadata))
+ lstn.mutex.Unlock()
+
+ lstn.refreshServiceURLs()
+ return nil
+}
+
+// refreshServiceURLs rebuilds service URLs from the latest instance snapshot and
+// notifies subscribers. The build is serialized by buildMu, but lstn.mutex is
+// only held while reading or committing in-memory state: metadata RPCs run in
+// between without it, so a slow or unreachable provider cannot block event
+// processing or retry scheduling.
+func (lstn *ServiceInstancesChangedListenerImpl) refreshServiceURLs() {
Review Comment:
让 OnEvent 返回 err也就是,全部解析成功 → nil,有未解析 revision → 返回 err,同时重试照常调度
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [PR] fix: self-heal application-level discovery after transient metadata failure (#3615) [dubbo-go]
Posted by "AlexStocks (via GitHub)" <gi...@apache.org>.
AlexStocks commented on code in PR #3625:
URL: https://github.com/apache/dubbo-go/pull/3625#discussion_r3749843553
##########
registry/servicediscovery/service_instances_changed_listener_impl.go:
##########
@@ -343,3 +409,94 @@ func GetMetadataInfo(app string, instance registry.ServiceInstance, revision str
metaCache.Set(cacheKey, metadataInfo)
return metadataInfo, nil
}
+
+var (
+ // metadataRetryInitialDelay is the first backoff delay before retrying a failed
+ // metadata fetch. Package-level so tests can shrink it.
+ metadataRetryInitialDelay = time.Second
+ // metadataRetryMaxDelay caps the backoff. The retry count itself is
+ // intentionally unlimited: retries only target instances the registry still
+ // reports as alive, and a capped count would re-introduce the permanent
+ // empty-directory failure this mechanism fixes.
+ metadataRetryMaxDelay = 30 * time.Second
+ // metadataFetchFailureLogInterval throttles repeated fetch-failure warnings
+ // for the same revision key.
+ metadataFetchFailureLogInterval = 5 * time.Minute
+)
+
+// metadataInfoFetcher resolves MetadataInfo for a revision; a package-level
+// indirection so tests can inject transient failures.
+var metadataInfoFetcher = GetMetadataInfo
+
+// stopMetadataRetry cancels any pending metadata retry. It is called when the
+// owning registry is destroyed and drops this listener without RemoveListener,
+// so the retry timer cannot leak.
+func (lstn *ServiceInstancesChangedListenerImpl) stopMetadataRetry() {
+ lstn.mutex.Lock()
+ defer lstn.mutex.Unlock()
+ if lstn.retryTimer != nil {
+ lstn.retryTimer.Stop()
+ lstn.retryTimer = nil
+ }
+}
+
+// scheduleMetadataRetry arms the shared retry timer while unresolved revisions
+// remain. Retries replay the latest instance snapshot, so revisions whose
+// instances disappeared from the registry are dropped naturally on the next run.
+func (lstn *ServiceInstancesChangedListenerImpl) scheduleMetadataRetry() {
+ lstn.mutex.Lock()
+ defer lstn.mutex.Unlock()
+ if len(lstn.unresolvedRevisions) == 0 {
+ if lstn.retryTimer != nil {
+ lstn.retryTimer.Stop()
+ lstn.retryTimer = nil
+ }
+ lstn.retryAttempts = 0
+ return
+ }
+ if lstn.retryTimer != nil {
+ // One shared timer per listener: repeated events must not multiply retries.
+ return
+ }
Review Comment:
[P1] 当前公共 `UnSubscribe` 入口仍没有真正覆盖这个生命周期出口:`SubscribeURL` 以 `ServiceKey:protocol` 保存通知器,但 `UnSubscribe` 仍用 `url.ServiceKey()` 调 `RemoveListener`,所以最后一个订阅者离开后 `len(listeners)` 仍非零,新增的 timer 取消分支不会执行。exact-Head 公开入口探针在 `UnSubscribe` 后 metadata fetch 仍从 4 增至 8。请统一两端 key(最好共用 helper),并用 `SubscribeURL -> UnSubscribe` 的回归测试验证 fetch/timer 停止,而不是只直接测试 `RemoveListener`。
##########
registry/servicediscovery/service_discovery_registry.go:
##########
@@ -323,6 +323,15 @@ func (s *serviceDiscoveryRegistry) IsAvailable() bool {
func (s *serviceDiscoveryRegistry) Destroy() {
s.stopMetadataTimers()
+ s.lock.Lock()
+ for _, l := range s.serviceListeners {
+ // Destroy drops listeners without RemoveListener; cancel any pending
+ // metadata retry so its timer cannot leak.
+ if impl, ok := l.(*ServiceInstancesChangedListenerImpl); ok {
+ impl.stopMetadataRetry()
+ }
+ }
+ s.lock.Unlock()
err := s.serviceDiscovery.Destroy()
if err != nil {
Review Comment:
[P1] `closed` 修复了已安装 listener 的在途 refresh,但 `Destroy` 仍看不到尚在初始 `GetInstances`/metadata 阶段的 `SubscribeURL`。该调用在 `Destroy` 返回后完成时仍会安装 listener、`AddListenerAndNotify` 并启动新的无限 timer;exact-Head 阻塞探针中 fetch 在 Destroy 后从 3 增至 7。这里需要 registry 级 `destroyed`/generation 状态,并在安装/附加订阅者前复检、丢弃晚到 listener;请补 `SubscribeURL` 与 `Destroy` 的确定性交错测试。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org