You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "bzp2010 (via GitHub)" <gi...@apache.org> on 2026/09/03 15:51:32 UTC
[PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
bzp2010 opened a new pull request, #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856
<!-- Please answer these questions before submitting a pull request -->
### Type of change:
<!-- Please delete options that are not relevant. -->
<!-- Select all the options from below that matches the type your PR best -->
- [ ] Bugfix
- [x] New feature provided
- [ ] Improve performance
- [ ] Backport patches
- [ ] Documentation
- [x] Refactor
- [ ] Chore
- [ ] CI/CD or Tests
### What this PR does / why we need it:
Accurately report instance status.
Errors are detected based on the status and error messages for each individual endpoint contained in the `endpoint_status` returned by the ADC, and these are logged in the newly added `status.conditions` field of the Gateway Proxy resource.
The `status.conditions` conforms to the Kubernetes definition and is not a custom one. The `DataPlaneAvailable` condition is used to log node anomalies. When an anomaly occurs, the `reason` is set to `DataPlaneInstanceUnavailable`, and the `message` contains an aggregated error message.
```yaml
# bad
status:
conditions:
- type: DataPlaneAvailable
status: "False"
observedGeneration: 1
lastTransitionTime: "2026-09-04T02:10:03Z"
reason: DataPlaneInstanceUnavailable
message: "1/2 gateway instance(s) failed to apply the last sync: http://unreachable-instance.invalid:9180: dial tcp: lookup unreachable-instance.invalid: server misbehaving"
# good
status:
conditions:
- type: DataPlaneAvailable
status: "True"
observedGeneration: 1
lastTransitionTime: "2026-09-04T02:25:11Z"
reason: DataPlaneAvailable
message: ""
```
In addition, errors at the per-endpoint level are reported as events and associated with the Gateway Proxy.
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning DataPlaneInstanceUnavailable 8s (x23 over 12m) apisix-provider http://apisix-2:9180: dial tcp 10.244.0.12:9180: connect: connection refused
Warning DataPlaneInstanceUnavailable 8s (x23 over 12m) apisix-provider https://apisix-3:9180: tls: handshake failure
```
```yaml
apiVersion: v1
kind: Event
metadata:
name: gateway-proxy-partial-instance.17f2a3c1d0e5f6a7
namespace: ns-e2e-test
count: 23
firstTimestamp: "2026-09-04T02:10:03Z"
lastTimestamp: "2026-09-04T02:22:11Z"
type: Warning
reason: DataPlaneInstanceUnavailable
message: "http://apisix-2:9180: dial tcp 10.244.0.12:9180: connect: connection refused"
involvedObject:
apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
name: gateway-proxy-partial-instance
namespace: ns-e2e-test
source:
component: apisix-provider
```
### Pre-submission checklist:
<!--
Please follow the requirements:
1. Use Draft if the PR is not ready to be reviewed
2. Test is required for the feat/fix PR, unless you have a good reason
3. Doc is required for the feat PR
4. Use a new commit to resolve review instead of `push -f`
5. Use "request review" to notify the reviewer once you have resolved the review
-->
- [x] Did you explain what problem does this PR solve? Or what new features have been added?
- [x] Have you added corresponding test cases?
- [ ] Have you modified the corresponding document?
- [x] Is this PR backward compatible? **If it is not backward compatible, please discuss on the [mailing list](https://github.com/apache/apisix-ingress-controller#community) first**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3988807339
##########
.github/workflows/apisix-e2e-test.yml:
##########
@@ -30,7 +30,7 @@ concurrency:
cancel-in-progress: true
env:
- ADC_RUST_VERSION: "0.30.2"
+ ADC_RUST_VERSION: "0.30.3"
Review Comment:
This upgrades only the Rust binary used by E2E. The shipped manager manifest still selects `ghcr.io/api7/adc:0.29.0` (`config/manager/kustomization.yaml:18-20`), and the linked Node/standalone job demonstrates that implementation fails the new per-endpoint scenario. Consequently a default deployment cannot provide the per-instance events and messages promised by this PR and the troubleshooting guide. Align the shipped ADC implementation/version with the tested one, or explicitly scope and gate the feature until the default ADC supports it.
##########
internal/provider/apisix/status.go:
##########
@@ -18,56 +18,232 @@
package apisix
import (
+ "context"
"fmt"
"strings"
+ corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
+ adctypes "github.com/apache/apisix-ingress-controller/api/adc"
+ apiv1alpha1 "github.com/apache/apisix-ingress-controller/api/v1alpha1"
apiv2 "github.com/apache/apisix-ingress-controller/api/v2"
"github.com/apache/apisix-ingress-controller/internal/controller/label"
"github.com/apache/apisix-ingress-controller/internal/controller/status"
cutils "github.com/apache/apisix-ingress-controller/internal/controller/utils"
"github.com/apache/apisix-ingress-controller/internal/types"
)
-// handleStatusUpdate updates resource conditions based on the latest sync results.
+// GatewayProxyConditionDataPlaneAvailable reports whether every APISIX instance a
+// GatewayProxy addresses took the last sync. It lands on the GatewayProxy rather than
+// a Gateway, an IngressClass, or a CRD, since GatewayProxy is what every path shares.
+const (
+ GatewayProxyConditionDataPlaneAvailable = "DataPlaneAvailable"
+
+ GatewayProxyReasonDataPlaneAvailable = "DataPlaneAvailable"
+ GatewayProxyReasonDataPlaneInstanceUnavailable = "DataPlaneInstanceUnavailable"
+)
+
+// updateStatusFromSyncResults updates every resource and GatewayProxy status this
+// round's sync results call for. results holds one entry per config sync() actually
+// reached pushConfig for this round, success (a zero-value types.ADCExecutionErrors) or
+// failure; a config sync() could not even build a SyncInput for is absent here entirely
+// and its status is left untouched this round, see sync().
//
-// It maintains a history of failed resources in d.statusUpdateMap.
+// GatewayProxy's DataPlaneAvailable condition is recomputed and written fresh every
+// round directly from this round's result, never compared against any remembered
+// history: a config with no error this round is written True, one with any error is
+// written False. That is what makes a GatewayProxy that has always been healthy actually
+// get a True the first time, and what keeps a restart from leaving a stale False stuck
+// forever: the write only ever depends on this round's actual outcome.
//
-// For resources in the current failure map (statusUpdateMap), it marks them as failed.
-// For resources that exist only in the previous failure history (i.e. not in this sync's failures),
-// it marks them as accepted (success).
-func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
- // Mark all resources in the current failure set as failed.
- for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+// Resource status can't afford the same full recompute: a config's resource set can be
+// large, and rewriting every one of them every round even when nothing changed would be
+// wasteful. So resources keep a small persisted delta in d.resourceFailures instead:
+// newly (or still) failing resources are written SyncFailed, and any resource that was
+// failing last round but isn't failing this one gets its error explicitly cleared with
+// an Accepted write.
+func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors) {
+ resourceFailures := map[types.NamespacedNameKind][]string{}
+
+ for configName, execErrs := range results {
+ var gatewayProxy types.NamespacedNameKind
+ if err := gatewayProxy.FromString(configName); err != nil {
+ d.log.Error(err, "failed to parse config name as a GatewayProxy key", "configName", configName)
+ continue
+ }
+
+ gatewayProxyMsgs, failedEndpoints := d.classifySyncResult(configName, execErrs, resourceFailures)
+ if len(gatewayProxyMsgs) > 0 {
+ d.updateStatus(gatewayProxy, failureCondition(gatewayProxy, strings.Join(gatewayProxyMsgs, "; ")))
+ d.recordFailedEndpointEvents(ctx, gatewayProxy, failedEndpoints)
+ } else {
+ d.updateStatus(gatewayProxy, successCondition(gatewayProxy))
+ }
+ }
+
+ d.applyResourceFailures(resourceFailures)
+ d.log.V(1).Info("updated status from sync results", "results", results, "resource_failures", resourceFailures)
+}
+
+// classifySyncResult splits one config's this-round result into what belongs on the
+// GatewayProxy (returned) and what belongs on specific Kubernetes resources (added into
+// resourceFailures). A FailedStatuses entry that resolves to a resource via its Event
+// goes there; everything else, no FailedStatuses at all, or a FailedStatuses entry
+// with no Event to resolve, which is what apisix-standalone's own endpoint-driven
+// rejections look like, is a GatewayProxy-level signal instead, using
+// EndpointStatuses for the message when there is one and the raw error otherwise.
+func (d *apisixProvider) classifySyncResult(
+ configName string,
+ execErrs types.ADCExecutionErrors,
+ resourceFailures map[types.NamespacedNameKind][]string,
+) (gatewayProxyMsgs []string, failedEndpoints []adctypes.EndpointStatus) {
+ unattributed := func(addrErr types.ADCExecutionServerAddrError) {
+ msg := unavailableEndpointsMessage(addrErr.EndpointStatuses)
+ if msg == "" {
+ msg = addrErr.Error()
+ }
+ gatewayProxyMsgs = append(gatewayProxyMsgs, msg)
+ failedEndpoints = append(failedEndpoints, addrErr.EndpointStatuses...)
+ }
+
+ for _, execErr := range execErrs.Errors {
+ for _, addrErr := range execErr.FailedErrors {
+ if len(addrErr.FailedStatuses) == 0 {
+ unattributed(addrErr)
+ continue
+ }
Review Comment:
`EndpointStatuses` are only examined when `FailedStatuses` is empty (or contains an un-attributable event). ADC 0.30.3 can return an all-failed standalone response containing both resource-attributed `failed` entries and failed `endpoint_status` entries; in that case this method records the resource failure but returns no GatewayProxy message/endpoints, so line 83 writes `DataPlaneAvailable=True` and emits no events while every endpoint reports failure. Process failed endpoint statuses independently from the per-resource attribution path, and add a regression test covering both fields in one result.
##########
internal/provider/apisix/status.go:
##########
@@ -18,56 +18,232 @@
package apisix
import (
+ "context"
"fmt"
"strings"
+ corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
+ adctypes "github.com/apache/apisix-ingress-controller/api/adc"
+ apiv1alpha1 "github.com/apache/apisix-ingress-controller/api/v1alpha1"
apiv2 "github.com/apache/apisix-ingress-controller/api/v2"
"github.com/apache/apisix-ingress-controller/internal/controller/label"
"github.com/apache/apisix-ingress-controller/internal/controller/status"
cutils "github.com/apache/apisix-ingress-controller/internal/controller/utils"
"github.com/apache/apisix-ingress-controller/internal/types"
)
-// handleStatusUpdate updates resource conditions based on the latest sync results.
+// GatewayProxyConditionDataPlaneAvailable reports whether every APISIX instance a
+// GatewayProxy addresses took the last sync. It lands on the GatewayProxy rather than
+// a Gateway, an IngressClass, or a CRD, since GatewayProxy is what every path shares.
+const (
+ GatewayProxyConditionDataPlaneAvailable = "DataPlaneAvailable"
+
+ GatewayProxyReasonDataPlaneAvailable = "DataPlaneAvailable"
+ GatewayProxyReasonDataPlaneInstanceUnavailable = "DataPlaneInstanceUnavailable"
+)
+
+// updateStatusFromSyncResults updates every resource and GatewayProxy status this
+// round's sync results call for. results holds one entry per config sync() actually
+// reached pushConfig for this round, success (a zero-value types.ADCExecutionErrors) or
+// failure; a config sync() could not even build a SyncInput for is absent here entirely
+// and its status is left untouched this round, see sync().
//
-// It maintains a history of failed resources in d.statusUpdateMap.
+// GatewayProxy's DataPlaneAvailable condition is recomputed and written fresh every
+// round directly from this round's result, never compared against any remembered
+// history: a config with no error this round is written True, one with any error is
+// written False. That is what makes a GatewayProxy that has always been healthy actually
+// get a True the first time, and what keeps a restart from leaving a stale False stuck
+// forever: the write only ever depends on this round's actual outcome.
//
-// For resources in the current failure map (statusUpdateMap), it marks them as failed.
-// For resources that exist only in the previous failure history (i.e. not in this sync's failures),
-// it marks them as accepted (success).
-func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
- // Mark all resources in the current failure set as failed.
- for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+// Resource status can't afford the same full recompute: a config's resource set can be
+// large, and rewriting every one of them every round even when nothing changed would be
+// wasteful. So resources keep a small persisted delta in d.resourceFailures instead:
+// newly (or still) failing resources are written SyncFailed, and any resource that was
+// failing last round but isn't failing this one gets its error explicitly cleared with
+// an Accepted write.
+func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors) {
+ resourceFailures := map[types.NamespacedNameKind][]string{}
+
+ for configName, execErrs := range results {
+ var gatewayProxy types.NamespacedNameKind
+ if err := gatewayProxy.FromString(configName); err != nil {
+ d.log.Error(err, "failed to parse config name as a GatewayProxy key", "configName", configName)
+ continue
+ }
+
+ gatewayProxyMsgs, failedEndpoints := d.classifySyncResult(configName, execErrs, resourceFailures)
+ if len(gatewayProxyMsgs) > 0 {
+ d.updateStatus(gatewayProxy, failureCondition(gatewayProxy, strings.Join(gatewayProxyMsgs, "; ")))
+ d.recordFailedEndpointEvents(ctx, gatewayProxy, failedEndpoints)
+ } else {
+ d.updateStatus(gatewayProxy, successCondition(gatewayProxy))
+ }
+ }
+
+ d.applyResourceFailures(resourceFailures)
+ d.log.V(1).Info("updated status from sync results", "results", results, "resource_failures", resourceFailures)
+}
+
+// classifySyncResult splits one config's this-round result into what belongs on the
+// GatewayProxy (returned) and what belongs on specific Kubernetes resources (added into
+// resourceFailures). A FailedStatuses entry that resolves to a resource via its Event
+// goes there; everything else, no FailedStatuses at all, or a FailedStatuses entry
+// with no Event to resolve, which is what apisix-standalone's own endpoint-driven
+// rejections look like, is a GatewayProxy-level signal instead, using
+// EndpointStatuses for the message when there is one and the raw error otherwise.
+func (d *apisixProvider) classifySyncResult(
+ configName string,
+ execErrs types.ADCExecutionErrors,
+ resourceFailures map[types.NamespacedNameKind][]string,
+) (gatewayProxyMsgs []string, failedEndpoints []adctypes.EndpointStatus) {
+ unattributed := func(addrErr types.ADCExecutionServerAddrError) {
+ msg := unavailableEndpointsMessage(addrErr.EndpointStatuses)
+ if msg == "" {
+ msg = addrErr.Error()
+ }
+ gatewayProxyMsgs = append(gatewayProxyMsgs, msg)
+ failedEndpoints = append(failedEndpoints, addrErr.EndpointStatuses...)
+ }
+
+ for _, execErr := range execErrs.Errors {
+ for _, addrErr := range execErr.FailedErrors {
+ if len(addrErr.FailedStatuses) == 0 {
+ unattributed(addrErr)
+ continue
+ }
+
+ attributedAll := true
+ for _, syncStatus := range addrErr.FailedStatuses {
+ if syncStatus.Event.ResourceType == "" {
+ // standalone: this whole addrErr carries no per-resource attribution.
+ attributedAll = false
+ break
+ }
+ labels, err := d.store.GetResourceLabel(configName, syncStatus.Event.ResourceType, syncStatus.Event.ResourceID)
+ if err != nil {
+ d.log.Error(err, "failed to get resource label",
+ "configName", configName, "resourceType", syncStatus.Event.ResourceType, "id", syncStatus.Event.ResourceID)
+ continue
+ }
+ resourceKey := types.NamespacedNameKind{
+ Name: labels[label.LabelName],
+ Namespace: labels[label.LabelNamespace],
+ Kind: labels[label.LabelKind],
+ }
+ msg := fmt.Sprintf("ServerAddr: %s, Error: %s", addrErr.ServerAddr, syncStatus.Reason)
+ resourceFailures[resourceKey] = append(resourceFailures[resourceKey], msg)
+ }
+ if !attributedAll {
+ unattributed(addrErr)
+ }
+ }
+ }
+ return gatewayProxyMsgs, failedEndpoints
+}
+
+// applyResourceFailures writes this round's newly (or still) failing resources, and
+// clears the recorded error from any resource that was failing last round but isn't in
+// newFailures now. See updateStatusFromSyncResults for why resources use this delta
+// instead of GatewayProxy's full recompute.
+func (d *apisixProvider) applyResourceFailures(newFailures map[types.NamespacedNameKind][]string) {
+ for resourceKey, msgs := range newFailures {
+ d.updateStatus(resourceKey, failureCondition(resourceKey, strings.Join(msgs, "; ")))
+ }
+ for resourceKey := range d.resourceFailures {
+ if _, stillFailing := newFailures[resourceKey]; !stillFailing {
+ d.updateStatus(resourceKey, successCondition(resourceKey))
+ }
+ }
+ d.resourceFailures = newFailures
+}
+
+// failureCondition and successCondition pick which condition a NamespacedNameKind
+// gets: GatewayProxyConditionDataPlaneAvailable for a GatewayProxy, the existing
+// Accepted/SyncFailed condition for everything else.
+func failureCondition(nnk types.NamespacedNameKind, msg string) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(false, GatewayProxyReasonDataPlaneInstanceUnavailable, msg)
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonSyncFailed, false, 0, msg)
+}
+
+func successCondition(nnk types.NamespacedNameKind) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(true, GatewayProxyReasonDataPlaneAvailable, "")
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonAccepted, true, 0, "")
+}
+
+func newGatewayProxyDataPlaneAvailableCondition(available bool, reason, msg string) metav1.Condition {
+ conditionStatus := metav1.ConditionFalse
+ if available {
+ conditionStatus = metav1.ConditionTrue
+ }
+ return metav1.Condition{
+ Type: GatewayProxyConditionDataPlaneAvailable,
+ Status: conditionStatus,
+ LastTransitionTime: metav1.Now(),
+ Reason: reason,
+ Message: cutils.TruncateConditionMessage(msg),
+ }
+}
+
+// recordFailedEndpointEvents fires one Warning event per failed EndpointStatus entry,
+// so each instance's own failure history (when it started, how often) is visible on
+// its own, not folded into everyone else's. The GatewayProxy is fetched fresh from the
+// API server first so the Event's involvedObject carries a real UID: a hand-built stub
+// with only Name/Namespace leaves that UID empty, and kubectl describe resolves events
+// by matching it, so an event against such a stub never shows up there.
+func (d *apisixProvider) recordFailedEndpointEvents(ctx context.Context, nnk types.NamespacedNameKind, endpoints []adctypes.EndpointStatus) {
+ if d.EventRecorder == nil {
+ return
+ }
+ hasFailure := false
+ for _, ep := range endpoints {
+ if !ep.Success {
+ hasFailure = true
+ break
+ }
+ }
+ if !hasFailure {
+ return
}
- // Mark resources that exist only in the previous failure history as successful.
- for nnk := range d.statusUpdateMap {
- if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ gatewayProxy := &apiv1alpha1.GatewayProxy{}
+ if err := d.K8sClient.Get(ctx, nnk.NamespacedName(), gatewayProxy); err != nil {
+ d.log.Error(err, "failed to get GatewayProxy to record failed endpoint events", "name", nnk.Name, "namespace", nnk.Namespace)
+ return
+ }
+
+ for _, ep := range endpoints {
+ if ep.Success {
+ continue
}
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeWarning, GatewayProxyReasonDataPlaneInstanceUnavailable,
+ fmt.Sprintf("%s: %s", ep.Server, ep.Reason))
}
- // Update the failure history with the current failure set.
- d.statusUpdateMap = statusUpdateMap
}
//nolint:gocyclo
func (d *apisixProvider) updateStatus(nnk types.NamespacedNameKind, condition metav1.Condition) {
switch nnk.Kind {
+ case types.KindGatewayProxy:
+ // Unlike the route kinds below, the condition lands on the GatewayProxy's own
+ // top-level Status.Conditions, not on a per-parent entry.
+ d.updater.Update(status.Update{
+ NamespacedName: nnk.NamespacedName(),
+ Resource: &apiv1alpha1.GatewayProxy{},
+ Mutator: status.MutatorFunc(func(obj client.Object) client.Object {
+ cp := obj.(*apiv1alpha1.GatewayProxy).DeepCopy()
+ condition.ObservedGeneration = cp.GetGeneration()
+ cp.Status.Conditions = cutils.MergeCondition(cp.Status.Conditions, condition)
Review Comment:
`MergeCondition` replaces the old condition with the freshly timestamped one even when `Status` is unchanged. If an endpoint's error message changes while availability remains `False`, this advances `lastTransitionTime` despite no True/False/Unknown transition, contradicting the `metav1.Condition` contract and the generated CRD description. Preserve the existing timestamp whenever the condition type and status match (or use `meta.SetStatusCondition`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#issuecomment-5565460007
> There is an issue that needs to be discussed:
>
> Currently, this PR's change is limited to apisix-standalone, while apisix still propagates errors to resources—whether the Admin API is unavailable or there is an error in the configuration itself. Should we distinguish between these cases?
Sync errors should be recorded on the route resources — that status is how users check whether a route has actually taken effect.
For `GatewayProxy`, it may make more sense to record only Admin API unavailability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3947161268
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
+ if nnk.Kind == types.KindGatewayProxy {
+ d.recordGatewayProxyRecoveredEvent(nnk)
+ }
}
}
// Update the failure history with the current failure set.
d.statusUpdateMap = statusUpdateMap
}
+// failureCondition and successCondition pick which condition a NamespacedNameKind
+// gets: GatewayProxyConditionDataPlaneAvailable for a GatewayProxy, the existing
+// Accepted/SyncFailed condition for everything else.
+func failureCondition(nnk types.NamespacedNameKind, msg string) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(false, GatewayProxyReasonDataPlaneInstanceUnavailable, msg)
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonSyncFailed, false, 0, msg)
+}
+
+func successCondition(nnk types.NamespacedNameKind) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(true, GatewayProxyReasonDataPlaneAvailable, "")
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonAccepted, true, 0, "")
+}
+
+func newGatewayProxyDataPlaneAvailableCondition(available bool, reason, msg string) metav1.Condition {
+ conditionStatus := metav1.ConditionFalse
+ if available {
+ conditionStatus = metav1.ConditionTrue
+ }
+ return metav1.Condition{
+ Type: GatewayProxyConditionDataPlaneAvailable,
+ Status: conditionStatus,
+ LastTransitionTime: metav1.Now(),
+ Reason: reason,
+ Message: cutils.TruncateConditionMessage(msg),
+ }
+}
+
+// recordGatewayProxyRecoveredEvent fires once a GatewayProxy leaves the failure
+// history entirely, i.e. every instance took the last sync.
+func (d *apisixProvider) recordGatewayProxyRecoveredEvent(nnk types.NamespacedNameKind) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeNormal, GatewayProxyReasonDataPlaneAvailable,
+ "every data plane instance took the last sync")
+}
+
+// recordFailedEndpointEvents fires one Warning event per failed EndpointStatus entry,
+// so each instance's own failure history (when it started, how often) is visible on
+// its own, not folded into everyone else's.
+func (d *apisixProvider) recordFailedEndpointEvents(nnk types.NamespacedNameKind, endpoints []adctypes.EndpointStatus) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
Review Comment:
This stub has no UID, so the Event's `involvedObject.uid` is empty and `kubectl describe gatewayproxy` won't show it — which is what the new troubleshooting section tells users to run: [troubleshoot.md#L77-L88](https://github.com/apache/apisix-ingress-controller/blob/6d9930834abc81efc59719d8846b5f69ece03695/docs/en/latest/reference/troubleshoot.md#L77-L88)
`client-go` adds `involvedObject.uid=<uid>` to the field selector whenever the described object has a UID, which a real GatewayProxy always does. Verified on a live cluster: same object and event, without the uid `describe` prints `Events: <none>`, with it the event shows. The e2e passes because it selects on `kind` + `name` only.
Either record against the real object from the informer cache (needs `mgr.GetClient()` threaded through from `run.go`), or change that doc block to `kubectl get events --field-selector ...`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3977206277
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
+ if nnk.Kind == types.KindGatewayProxy {
+ d.recordGatewayProxyRecoveredEvent(nnk)
+ }
}
}
// Update the failure history with the current failure set.
d.statusUpdateMap = statusUpdateMap
}
+// failureCondition and successCondition pick which condition a NamespacedNameKind
+// gets: GatewayProxyConditionDataPlaneAvailable for a GatewayProxy, the existing
+// Accepted/SyncFailed condition for everything else.
+func failureCondition(nnk types.NamespacedNameKind, msg string) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(false, GatewayProxyReasonDataPlaneInstanceUnavailable, msg)
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonSyncFailed, false, 0, msg)
+}
+
+func successCondition(nnk types.NamespacedNameKind) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(true, GatewayProxyReasonDataPlaneAvailable, "")
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonAccepted, true, 0, "")
+}
+
+func newGatewayProxyDataPlaneAvailableCondition(available bool, reason, msg string) metav1.Condition {
+ conditionStatus := metav1.ConditionFalse
+ if available {
+ conditionStatus = metav1.ConditionTrue
+ }
+ return metav1.Condition{
+ Type: GatewayProxyConditionDataPlaneAvailable,
+ Status: conditionStatus,
+ LastTransitionTime: metav1.Now(),
+ Reason: reason,
+ Message: cutils.TruncateConditionMessage(msg),
+ }
+}
+
+// recordGatewayProxyRecoveredEvent fires once a GatewayProxy leaves the failure
+// history entirely, i.e. every instance took the last sync.
+func (d *apisixProvider) recordGatewayProxyRecoveredEvent(nnk types.NamespacedNameKind) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeNormal, GatewayProxyReasonDataPlaneAvailable,
+ "every data plane instance took the last sync")
+}
+
+// recordFailedEndpointEvents fires one Warning event per failed EndpointStatus entry,
+// so each instance's own failure history (when it started, how often) is visible on
+// its own, not folded into everyone else's.
+func (d *apisixProvider) recordFailedEndpointEvents(nnk types.NamespacedNameKind, endpoints []adctypes.EndpointStatus) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ for _, ep := range endpoints {
+ if ep.Success {
+ continue
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeWarning, GatewayProxyReasonDataPlaneInstanceUnavailable,
+ fmt.Sprintf("%s: %s", ep.Server, ep.Reason))
+ }
+}
+
//nolint:gocyclo
func (d *apisixProvider) updateStatus(nnk types.NamespacedNameKind, condition metav1.Condition) {
switch nnk.Kind {
+ case types.KindGatewayProxy:
+ // Unlike the route kinds below, the condition lands on the GatewayProxy's own
+ // top-level Status.Conditions, not on a per-parent entry.
+ d.updater.Update(status.Update{
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3947160805
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
Review Comment:
Recovery is inferred from "the key left the failure set", not from "this sync succeeded".
ADC produces the former without the latter:
| ADC `/sync` response | carries `endpoint_status`? |
| --- | --- |
| 202 success | yes (apisix-standalone) |
| 400 schema/lint, 500 catch-all | **no** |
Checked at `api7/adc` main (`9b0e093`) — same in both the Node and Rust servers.
So when ADC itself fails, `unavailableEndpointsMessage` returns `""` (status.go:368), the GatewayProxy never enters `statusUpdateMap`, and this branch flips it to `True` plus a "every data plane instance took the last sync" event — while the sync is still failing. The harder ADC fails, the greener the GatewayProxy gets.
Recovery should be driven by an actually-successful sync, with "ADC Server itself failed" leaving the condition untouched.
(Possibly related to the failing e2e you linked: on the Node ADC, `findLatest()` has no per-server error handling, so one unreachable instance throws out of `dump()` before `sync()` runs — 500, no `endpoint_status`.)
##########
api/v1alpha1/gatewayproxy_types.go:
##########
@@ -178,6 +179,19 @@ type GatewayProxy struct {
// GatewayProxySpec defines configuration of gateway proxy instances,
// including networking settings, global plugins, and plugin metadata.
Spec GatewayProxySpec `json:"spec,omitempty"`
+
+ // Status defines the current state of Gateway Proxy.
+ //
+ // +kubebuilder:default={conditions: {{type: "DataPlaneAvailable", status: "Unknown", reason:"Pending", message:"Waiting for controller", lastTransitionTime: "1970-01-01T00:00:00Z"}}}
Review Comment:
This default is never overwritten on the happy path — an always-healthy GatewayProxy reports `Waiting for controller` forever. That contradicts the "good" example in the PR description, where `status: "True"` is shown as the healthy steady state.
Cause: `handleStatusUpdate` writes the success condition only for keys that were in the *previous* failure set, and nothing else ever writes `DataPlaneAvailable`. Two effects:
- healthy from the start never turns `True`
- a stale `False` survives a restart — `d.statusUpdateMap` is in-memory, so the success branch never fires after a restart
Both are apisix-standalone, so scoping the feature to standalone doesn't cover them. Writing the condition for every known config at the end of each sync — success included — fixes both.
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
+ if nnk.Kind == types.KindGatewayProxy {
+ d.recordGatewayProxyRecoveredEvent(nnk)
+ }
}
}
// Update the failure history with the current failure set.
d.statusUpdateMap = statusUpdateMap
}
+// failureCondition and successCondition pick which condition a NamespacedNameKind
+// gets: GatewayProxyConditionDataPlaneAvailable for a GatewayProxy, the existing
+// Accepted/SyncFailed condition for everything else.
+func failureCondition(nnk types.NamespacedNameKind, msg string) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(false, GatewayProxyReasonDataPlaneInstanceUnavailable, msg)
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonSyncFailed, false, 0, msg)
+}
+
+func successCondition(nnk types.NamespacedNameKind) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(true, GatewayProxyReasonDataPlaneAvailable, "")
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonAccepted, true, 0, "")
+}
+
+func newGatewayProxyDataPlaneAvailableCondition(available bool, reason, msg string) metav1.Condition {
+ conditionStatus := metav1.ConditionFalse
+ if available {
+ conditionStatus = metav1.ConditionTrue
+ }
+ return metav1.Condition{
+ Type: GatewayProxyConditionDataPlaneAvailable,
+ Status: conditionStatus,
+ LastTransitionTime: metav1.Now(),
+ Reason: reason,
+ Message: cutils.TruncateConditionMessage(msg),
+ }
+}
+
+// recordGatewayProxyRecoveredEvent fires once a GatewayProxy leaves the failure
+// history entirely, i.e. every instance took the last sync.
+func (d *apisixProvider) recordGatewayProxyRecoveredEvent(nnk types.NamespacedNameKind) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeNormal, GatewayProxyReasonDataPlaneAvailable,
+ "every data plane instance took the last sync")
+}
+
+// recordFailedEndpointEvents fires one Warning event per failed EndpointStatus entry,
+// so each instance's own failure history (when it started, how often) is visible on
+// its own, not folded into everyone else's.
+func (d *apisixProvider) recordFailedEndpointEvents(nnk types.NamespacedNameKind, endpoints []adctypes.EndpointStatus) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
Review Comment:
This stub has no UID, so the Event's `involvedObject.uid` is empty and `kubectl describe gatewayproxy` won't show it — which is what `docs/en/latest/reference/troubleshoot.md` tells users to run.
`client-go` adds `involvedObject.uid=<uid>` to the field selector whenever the described object has a UID, which a real GatewayProxy always does. Verified on a live cluster: same object and event, without the uid `describe` prints `Events: <none>`, with it the event shows. The e2e passes because it selects on `kind` + `name` only.
Either record against the real object from the informer cache (needs `mgr.GetClient()` threaded through from `run.go`), or change the doc to `kubectl get events --field-selector ...`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#issuecomment-5539737893
There is an issue that needs to be discussed:
Currently, this PR's change is limited to apisix-standalone, while apisix still propagates errors to resources—whether the Admin API is unavailable or there is an error in the configuration itself. Should we distinguish between these cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 merged PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3946920565
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
+ if nnk.Kind == types.KindGatewayProxy {
+ d.recordGatewayProxyRecoveredEvent(nnk)
+ }
}
}
// Update the failure history with the current failure set.
d.statusUpdateMap = statusUpdateMap
}
+// failureCondition and successCondition pick which condition a NamespacedNameKind
+// gets: GatewayProxyConditionDataPlaneAvailable for a GatewayProxy, the existing
+// Accepted/SyncFailed condition for everything else.
+func failureCondition(nnk types.NamespacedNameKind, msg string) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(false, GatewayProxyReasonDataPlaneInstanceUnavailable, msg)
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonSyncFailed, false, 0, msg)
+}
+
+func successCondition(nnk types.NamespacedNameKind) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(true, GatewayProxyReasonDataPlaneAvailable, "")
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonAccepted, true, 0, "")
+}
+
+func newGatewayProxyDataPlaneAvailableCondition(available bool, reason, msg string) metav1.Condition {
+ conditionStatus := metav1.ConditionFalse
+ if available {
+ conditionStatus = metav1.ConditionTrue
+ }
+ return metav1.Condition{
+ Type: GatewayProxyConditionDataPlaneAvailable,
+ Status: conditionStatus,
+ LastTransitionTime: metav1.Now(),
+ Reason: reason,
+ Message: cutils.TruncateConditionMessage(msg),
+ }
+}
+
+// recordGatewayProxyRecoveredEvent fires once a GatewayProxy leaves the failure
+// history entirely, i.e. every instance took the last sync.
+func (d *apisixProvider) recordGatewayProxyRecoveredEvent(nnk types.NamespacedNameKind) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeNormal, GatewayProxyReasonDataPlaneAvailable,
+ "every data plane instance took the last sync")
+}
+
+// recordFailedEndpointEvents fires one Warning event per failed EndpointStatus entry,
+// so each instance's own failure history (when it started, how often) is visible on
+// its own, not folded into everyone else's.
+func (d *apisixProvider) recordFailedEndpointEvents(nnk types.NamespacedNameKind, endpoints []adctypes.EndpointStatus) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ for _, ep := range endpoints {
+ if ep.Success {
+ continue
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeWarning, GatewayProxyReasonDataPlaneInstanceUnavailable,
+ fmt.Sprintf("%s: %s", ep.Server, ep.Reason))
+ }
+}
+
//nolint:gocyclo
func (d *apisixProvider) updateStatus(nnk types.NamespacedNameKind, condition metav1.Condition) {
switch nnk.Kind {
+ case types.KindGatewayProxy:
+ // Unlike the route kinds below, the condition lands on the GatewayProxy's own
+ // top-level Status.Conditions, not on a per-parent entry.
+ d.updater.Update(status.Update{
Review Comment:
This routes a new Kind through `UpdateHandler` that `statusEqual` doesn't know about. `internal/controller/status/updater.go:179` has no `*v1alpha1.GatewayProxy` case, so it falls through to `default: return false` — two identical conditions never compare equal, and the dedup shortcut at `updater.go:113` never fires.
Verified with a probe test (control included):
```
statusEqual(GatewayProxy{c}, GatewayProxy{c}) == false // no case -> default
statusEqual(Consumer{}, Consumer{}) == true // has a case
```
`newGatewayProxyDataPlaneAvailableCondition` also stamps a fresh `metav1.Now()` every cycle, so the serialized object differs too and the apiserver's no-op-write optimization doesn't save us either. Every sync issues a real `Status().Update()` and bumps `resourceVersion`.
That write then fans out. `gateway_controller.go` is the only one of the 13 GatewayProxy watchers without a `WithEventFilter`, and its `Watches(&v1alpha1.GatewayProxy{}, ...)` carries no predicate either, so a status-only update enqueues every Gateway whose `parametersRef` points here. `Reconcile` unconditionally calls `Provider.Update` (`gateway_controller.go:259`), which reaches `defer d.syncNotify()` (`provider.go:169`). `syncNotify` is an independent wake source in the sync loop's `select`, so it bypasses the retrier completely:
```
GatewayProxy status write -> Gateway reconcile -> syncNotify -> sync fails -> status write -> ...
```
While an endpoint stays down, this turns what should be a backoff (`RetryMaxDelay` 1000s, or the 1h ticker) into a continuous full resync: healthy instances get the entire config rewritten every cycle, and `failed to run http sync for server` floods the log. It stops on its own once the data plane recovers, but not before.
Adding a `case *v1alpha1.GatewayProxy` to `statusEqual` breaks the cycle at the source — `cmpIgnoreLastTT` already ignores `LastTransitionTime`, so an unchanged condition would be skipped. Giving the `gateway_controller` GatewayProxy watch a `GenerationChangedPredicate` would be a good second line of defence, though that one is pre-existing rather than introduced here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3988658560
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
+ if nnk.Kind == types.KindGatewayProxy {
+ d.recordGatewayProxyRecoveredEvent(nnk)
+ }
}
}
// Update the failure history with the current failure set.
d.statusUpdateMap = statusUpdateMap
}
+// failureCondition and successCondition pick which condition a NamespacedNameKind
+// gets: GatewayProxyConditionDataPlaneAvailable for a GatewayProxy, the existing
+// Accepted/SyncFailed condition for everything else.
+func failureCondition(nnk types.NamespacedNameKind, msg string) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(false, GatewayProxyReasonDataPlaneInstanceUnavailable, msg)
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonSyncFailed, false, 0, msg)
+}
+
+func successCondition(nnk types.NamespacedNameKind) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(true, GatewayProxyReasonDataPlaneAvailable, "")
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonAccepted, true, 0, "")
+}
+
+func newGatewayProxyDataPlaneAvailableCondition(available bool, reason, msg string) metav1.Condition {
+ conditionStatus := metav1.ConditionFalse
+ if available {
+ conditionStatus = metav1.ConditionTrue
+ }
+ return metav1.Condition{
+ Type: GatewayProxyConditionDataPlaneAvailable,
+ Status: conditionStatus,
+ LastTransitionTime: metav1.Now(),
+ Reason: reason,
+ Message: cutils.TruncateConditionMessage(msg),
+ }
+}
+
+// recordGatewayProxyRecoveredEvent fires once a GatewayProxy leaves the failure
+// history entirely, i.e. every instance took the last sync.
+func (d *apisixProvider) recordGatewayProxyRecoveredEvent(nnk types.NamespacedNameKind) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeNormal, GatewayProxyReasonDataPlaneAvailable,
+ "every data plane instance took the last sync")
+}
+
+// recordFailedEndpointEvents fires one Warning event per failed EndpointStatus entry,
+// so each instance's own failure history (when it started, how often) is visible on
+// its own, not folded into everyone else's.
+func (d *apisixProvider) recordFailedEndpointEvents(nnk types.NamespacedNameKind, endpoints []adctypes.EndpointStatus) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ for _, ep := range endpoints {
+ if ep.Success {
+ continue
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeWarning, GatewayProxyReasonDataPlaneInstanceUnavailable,
+ fmt.Sprintf("%s: %s", ep.Server, ep.Reason))
+ }
+}
+
//nolint:gocyclo
func (d *apisixProvider) updateStatus(nnk types.NamespacedNameKind, condition metav1.Condition) {
switch nnk.Kind {
+ case types.KindGatewayProxy:
+ // Unlike the route kinds below, the condition lands on the GatewayProxy's own
+ // top-level Status.Conditions, not on a per-parent entry.
+ d.updater.Update(status.Update{
Review Comment:
Confirmed `statusEqual` already special-cases GatewayProxy (see `internal/controller/status/updater.go`), so writes there dedupe the same way every other kind's do. No further change needed for this one.
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
Review Comment:
Reworked the whole status pipeline: GatewayProxy's DataPlaneAvailable is now recomputed from scratch every sync round straight from that round's actual result, never diffed against remembered history. A round with no error writes True, any error writes False — recovery is never inferred from a key merely leaving a failure set. This makes the class of bug you describe structurally impossible rather than patching the specific case. See `updateStatusFromSyncResults`/`classifySyncResult` in `internal/provider/apisix/status.go`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3988659399
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
Review Comment:
Reworked the whole status pipeline: GatewayProxy's DataPlaneAvailable is now recomputed from scratch every sync round straight from that round's actual result, never diffed against remembered history. A round with no error writes True, any error writes False — recovery is never inferred from a key merely leaving a failure set. This makes the class of bug you describe structurally impossible rather than patching the specific case. See `updateStatusFromSyncResults`/`classifySyncResult` in `internal/provider/apisix/status.go`.
Cost note, since this write path is now unconditional every round instead of only on change: the enqueue itself is cheap (a channel send), and the actual write is still deduped by the pre-existing `statusEqual` check in `UpdateHandler.updateStatus` (`internal/controller/status/updater.go`), so an unchanged GatewayProxy never triggers a real PATCH. The `client.Get` that check needs reads through the manager's shared informer cache, not the API server — GatewayProxy already has its own controller/watch (`GatewayProxyController`), so this isn't a fresh network round trip. The added cost is bounded to the (typically small) number of GatewayProxies; the per-resource path (ApisixRoute etc.) is untouched and keeps the old change-only write behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3990604036
##########
internal/provider/apisix/status.go:
##########
@@ -18,56 +18,232 @@
package apisix
import (
+ "context"
"fmt"
"strings"
+ corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
+ adctypes "github.com/apache/apisix-ingress-controller/api/adc"
+ apiv1alpha1 "github.com/apache/apisix-ingress-controller/api/v1alpha1"
apiv2 "github.com/apache/apisix-ingress-controller/api/v2"
"github.com/apache/apisix-ingress-controller/internal/controller/label"
"github.com/apache/apisix-ingress-controller/internal/controller/status"
cutils "github.com/apache/apisix-ingress-controller/internal/controller/utils"
"github.com/apache/apisix-ingress-controller/internal/types"
)
-// handleStatusUpdate updates resource conditions based on the latest sync results.
+// GatewayProxyConditionDataPlaneAvailable reports whether every APISIX instance a
+// GatewayProxy addresses took the last sync. It lands on the GatewayProxy rather than
+// a Gateway, an IngressClass, or a CRD, since GatewayProxy is what every path shares.
+const (
+ GatewayProxyConditionDataPlaneAvailable = "DataPlaneAvailable"
+
+ GatewayProxyReasonDataPlaneAvailable = "DataPlaneAvailable"
+ GatewayProxyReasonDataPlaneInstanceUnavailable = "DataPlaneInstanceUnavailable"
+)
+
+// updateStatusFromSyncResults updates every resource and GatewayProxy status this
+// round's sync results call for. results holds one entry per config sync() actually
+// reached pushConfig for this round, success (a zero-value types.ADCExecutionErrors) or
+// failure; a config sync() could not even build a SyncInput for is absent here entirely
+// and its status is left untouched this round, see sync().
//
-// It maintains a history of failed resources in d.statusUpdateMap.
+// GatewayProxy's DataPlaneAvailable condition is recomputed and written fresh every
+// round directly from this round's result, never compared against any remembered
+// history: a config with no error this round is written True, one with any error is
+// written False. That is what makes a GatewayProxy that has always been healthy actually
+// get a True the first time, and what keeps a restart from leaving a stale False stuck
+// forever: the write only ever depends on this round's actual outcome.
//
-// For resources in the current failure map (statusUpdateMap), it marks them as failed.
-// For resources that exist only in the previous failure history (i.e. not in this sync's failures),
-// it marks them as accepted (success).
-func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
- // Mark all resources in the current failure set as failed.
- for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+// Resource status can't afford the same full recompute: a config's resource set can be
+// large, and rewriting every one of them every round even when nothing changed would be
+// wasteful. So resources keep a small persisted delta in d.resourceFailures instead:
+// newly (or still) failing resources are written SyncFailed, and any resource that was
+// failing last round but isn't failing this one gets its error explicitly cleared with
+// an Accepted write.
+func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors) {
+ resourceFailures := map[types.NamespacedNameKind][]string{}
+
+ for configName, execErrs := range results {
+ var gatewayProxy types.NamespacedNameKind
+ if err := gatewayProxy.FromString(configName); err != nil {
+ d.log.Error(err, "failed to parse config name as a GatewayProxy key", "configName", configName)
+ continue
+ }
+
+ gatewayProxyMsgs, failedEndpoints := d.classifySyncResult(configName, execErrs, resourceFailures)
+ if len(gatewayProxyMsgs) > 0 {
+ d.updateStatus(gatewayProxy, failureCondition(gatewayProxy, strings.Join(gatewayProxyMsgs, "; ")))
+ d.recordFailedEndpointEvents(ctx, gatewayProxy, failedEndpoints)
+ } else {
+ d.updateStatus(gatewayProxy, successCondition(gatewayProxy))
+ }
+ }
+
+ d.applyResourceFailures(resourceFailures)
+ d.log.V(1).Info("updated status from sync results", "results", results, "resource_failures", resourceFailures)
+}
+
+// classifySyncResult splits one config's this-round result into what belongs on the
+// GatewayProxy (returned) and what belongs on specific Kubernetes resources (added into
+// resourceFailures). A FailedStatuses entry that resolves to a resource via its Event
+// goes there; everything else, no FailedStatuses at all, or a FailedStatuses entry
+// with no Event to resolve, which is what apisix-standalone's own endpoint-driven
+// rejections look like, is a GatewayProxy-level signal instead, using
+// EndpointStatuses for the message when there is one and the raw error otherwise.
+func (d *apisixProvider) classifySyncResult(
+ configName string,
+ execErrs types.ADCExecutionErrors,
+ resourceFailures map[types.NamespacedNameKind][]string,
+) (gatewayProxyMsgs []string, failedEndpoints []adctypes.EndpointStatus) {
+ unattributed := func(addrErr types.ADCExecutionServerAddrError) {
+ msg := unavailableEndpointsMessage(addrErr.EndpointStatuses)
+ if msg == "" {
+ msg = addrErr.Error()
+ }
+ gatewayProxyMsgs = append(gatewayProxyMsgs, msg)
+ failedEndpoints = append(failedEndpoints, addrErr.EndpointStatuses...)
+ }
+
+ for _, execErr := range execErrs.Errors {
+ for _, addrErr := range execErr.FailedErrors {
+ if len(addrErr.FailedStatuses) == 0 {
+ unattributed(addrErr)
+ continue
+ }
+
+ attributedAll := true
+ for _, syncStatus := range addrErr.FailedStatuses {
+ if syncStatus.Event.ResourceType == "" {
+ // standalone: this whole addrErr carries no per-resource attribution.
+ attributedAll = false
+ break
+ }
+ labels, err := d.store.GetResourceLabel(configName, syncStatus.Event.ResourceType, syncStatus.Event.ResourceID)
+ if err != nil {
+ d.log.Error(err, "failed to get resource label",
+ "configName", configName, "resourceType", syncStatus.Event.ResourceType, "id", syncStatus.Event.ResourceID)
+ continue
+ }
+ resourceKey := types.NamespacedNameKind{
+ Name: labels[label.LabelName],
+ Namespace: labels[label.LabelNamespace],
+ Kind: labels[label.LabelKind],
+ }
+ msg := fmt.Sprintf("ServerAddr: %s, Error: %s", addrErr.ServerAddr, syncStatus.Reason)
+ resourceFailures[resourceKey] = append(resourceFailures[resourceKey], msg)
+ }
+ if !attributedAll {
+ unattributed(addrErr)
+ }
+ }
+ }
+ return gatewayProxyMsgs, failedEndpoints
+}
+
+// applyResourceFailures writes this round's newly (or still) failing resources, and
+// clears the recorded error from any resource that was failing last round but isn't in
+// newFailures now. See updateStatusFromSyncResults for why resources use this delta
+// instead of GatewayProxy's full recompute.
+func (d *apisixProvider) applyResourceFailures(newFailures map[types.NamespacedNameKind][]string) {
+ for resourceKey, msgs := range newFailures {
+ d.updateStatus(resourceKey, failureCondition(resourceKey, strings.Join(msgs, "; ")))
+ }
+ for resourceKey := range d.resourceFailures {
+ if _, stillFailing := newFailures[resourceKey]; !stillFailing {
+ d.updateStatus(resourceKey, successCondition(resourceKey))
+ }
+ }
+ d.resourceFailures = newFailures
+}
+
+// failureCondition and successCondition pick which condition a NamespacedNameKind
+// gets: GatewayProxyConditionDataPlaneAvailable for a GatewayProxy, the existing
+// Accepted/SyncFailed condition for everything else.
+func failureCondition(nnk types.NamespacedNameKind, msg string) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(false, GatewayProxyReasonDataPlaneInstanceUnavailable, msg)
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonSyncFailed, false, 0, msg)
+}
+
+func successCondition(nnk types.NamespacedNameKind) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(true, GatewayProxyReasonDataPlaneAvailable, "")
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonAccepted, true, 0, "")
+}
+
+func newGatewayProxyDataPlaneAvailableCondition(available bool, reason, msg string) metav1.Condition {
+ conditionStatus := metav1.ConditionFalse
+ if available {
+ conditionStatus = metav1.ConditionTrue
+ }
+ return metav1.Condition{
+ Type: GatewayProxyConditionDataPlaneAvailable,
+ Status: conditionStatus,
+ LastTransitionTime: metav1.Now(),
+ Reason: reason,
+ Message: cutils.TruncateConditionMessage(msg),
+ }
+}
+
+// recordFailedEndpointEvents fires one Warning event per failed EndpointStatus entry,
+// so each instance's own failure history (when it started, how often) is visible on
+// its own, not folded into everyone else's. The GatewayProxy is fetched fresh from the
+// API server first so the Event's involvedObject carries a real UID: a hand-built stub
+// with only Name/Namespace leaves that UID empty, and kubectl describe resolves events
+// by matching it, so an event against such a stub never shows up there.
+func (d *apisixProvider) recordFailedEndpointEvents(ctx context.Context, nnk types.NamespacedNameKind, endpoints []adctypes.EndpointStatus) {
+ if d.EventRecorder == nil {
+ return
+ }
+ hasFailure := false
+ for _, ep := range endpoints {
+ if !ep.Success {
+ hasFailure = true
+ break
+ }
+ }
+ if !hasFailure {
+ return
}
- // Mark resources that exist only in the previous failure history as successful.
- for nnk := range d.statusUpdateMap {
- if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ gatewayProxy := &apiv1alpha1.GatewayProxy{}
+ if err := d.K8sClient.Get(ctx, nnk.NamespacedName(), gatewayProxy); err != nil {
+ d.log.Error(err, "failed to get GatewayProxy to record failed endpoint events", "name", nnk.Name, "namespace", nnk.Namespace)
+ return
+ }
+
+ for _, ep := range endpoints {
+ if ep.Success {
+ continue
}
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeWarning, GatewayProxyReasonDataPlaneInstanceUnavailable,
+ fmt.Sprintf("%s: %s", ep.Server, ep.Reason))
}
- // Update the failure history with the current failure set.
- d.statusUpdateMap = statusUpdateMap
}
//nolint:gocyclo
func (d *apisixProvider) updateStatus(nnk types.NamespacedNameKind, condition metav1.Condition) {
switch nnk.Kind {
+ case types.KindGatewayProxy:
+ // Unlike the route kinds below, the condition lands on the GatewayProxy's own
+ // top-level Status.Conditions, not on a per-parent entry.
+ d.updater.Update(status.Update{
+ NamespacedName: nnk.NamespacedName(),
+ Resource: &apiv1alpha1.GatewayProxy{},
+ Mutator: status.MutatorFunc(func(obj client.Object) client.Object {
+ cp := obj.(*apiv1alpha1.GatewayProxy).DeepCopy()
+ condition.ObservedGeneration = cp.GetGeneration()
+ cp.Status.Conditions = cutils.MergeCondition(cp.Status.Conditions, condition)
Review Comment:
This will be fixed in a future PR; it's a completely unrelated issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3988660173
##########
api/v1alpha1/gatewayproxy_types.go:
##########
@@ -178,6 +179,19 @@ type GatewayProxy struct {
// GatewayProxySpec defines configuration of gateway proxy instances,
// including networking settings, global plugins, and plugin metadata.
Spec GatewayProxySpec `json:"spec,omitempty"`
+
+ // Status defines the current state of Gateway Proxy.
+ //
+ // +kubebuilder:default={conditions: {{type: "DataPlaneAvailable", status: "Unknown", reason:"Pending", message:"Waiting for controller", lastTransitionTime: "1970-01-01T00:00:00Z"}}}
Review Comment:
Same fix as the other GatewayProxy status comment: the condition is now written unconditionally every round based on that round's actual outcome, with no dependency on any remembered history. A GatewayProxy that's always been healthy gets True the first time it syncs, and a restart can no longer leave a stale False stuck forever, since nothing carries over between rounds for GatewayProxy specifically.
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
+ if nnk.Kind == types.KindGatewayProxy {
+ d.recordGatewayProxyRecoveredEvent(nnk)
+ }
}
}
// Update the failure history with the current failure set.
d.statusUpdateMap = statusUpdateMap
}
+// failureCondition and successCondition pick which condition a NamespacedNameKind
+// gets: GatewayProxyConditionDataPlaneAvailable for a GatewayProxy, the existing
+// Accepted/SyncFailed condition for everything else.
+func failureCondition(nnk types.NamespacedNameKind, msg string) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(false, GatewayProxyReasonDataPlaneInstanceUnavailable, msg)
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonSyncFailed, false, 0, msg)
+}
+
+func successCondition(nnk types.NamespacedNameKind) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(true, GatewayProxyReasonDataPlaneAvailable, "")
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonAccepted, true, 0, "")
+}
+
+func newGatewayProxyDataPlaneAvailableCondition(available bool, reason, msg string) metav1.Condition {
+ conditionStatus := metav1.ConditionFalse
+ if available {
+ conditionStatus = metav1.ConditionTrue
+ }
+ return metav1.Condition{
+ Type: GatewayProxyConditionDataPlaneAvailable,
+ Status: conditionStatus,
+ LastTransitionTime: metav1.Now(),
+ Reason: reason,
+ Message: cutils.TruncateConditionMessage(msg),
+ }
+}
+
+// recordGatewayProxyRecoveredEvent fires once a GatewayProxy leaves the failure
+// history entirely, i.e. every instance took the last sync.
+func (d *apisixProvider) recordGatewayProxyRecoveredEvent(nnk types.NamespacedNameKind) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeNormal, GatewayProxyReasonDataPlaneAvailable,
+ "every data plane instance took the last sync")
+}
+
+// recordFailedEndpointEvents fires one Warning event per failed EndpointStatus entry,
+// so each instance's own failure history (when it started, how often) is visible on
+// its own, not folded into everyone else's.
+func (d *apisixProvider) recordFailedEndpointEvents(nnk types.NamespacedNameKind, endpoints []adctypes.EndpointStatus) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
Review Comment:
recordFailedEndpointEvents now fetches the real GatewayProxy via the client before firing the Event, instead of building a bare Name/Namespace stub, so involvedObject carries the real UID and shows up under `kubectl describe gatewayproxy`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3988647586
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
+ if nnk.Kind == types.KindGatewayProxy {
+ d.recordGatewayProxyRecoveredEvent(nnk)
+ }
}
}
// Update the failure history with the current failure set.
d.statusUpdateMap = statusUpdateMap
}
+// failureCondition and successCondition pick which condition a NamespacedNameKind
+// gets: GatewayProxyConditionDataPlaneAvailable for a GatewayProxy, the existing
+// Accepted/SyncFailed condition for everything else.
+func failureCondition(nnk types.NamespacedNameKind, msg string) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(false, GatewayProxyReasonDataPlaneInstanceUnavailable, msg)
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonSyncFailed, false, 0, msg)
+}
+
+func successCondition(nnk types.NamespacedNameKind) metav1.Condition {
+ if nnk.Kind == types.KindGatewayProxy {
+ return newGatewayProxyDataPlaneAvailableCondition(true, GatewayProxyReasonDataPlaneAvailable, "")
+ }
+ return cutils.NewConditionTypeAccepted(apiv2.ConditionReasonAccepted, true, 0, "")
+}
+
+func newGatewayProxyDataPlaneAvailableCondition(available bool, reason, msg string) metav1.Condition {
+ conditionStatus := metav1.ConditionFalse
+ if available {
+ conditionStatus = metav1.ConditionTrue
+ }
+ return metav1.Condition{
+ Type: GatewayProxyConditionDataPlaneAvailable,
+ Status: conditionStatus,
+ LastTransitionTime: metav1.Now(),
+ Reason: reason,
+ Message: cutils.TruncateConditionMessage(msg),
+ }
+}
+
+// recordGatewayProxyRecoveredEvent fires once a GatewayProxy leaves the failure
+// history entirely, i.e. every instance took the last sync.
+func (d *apisixProvider) recordGatewayProxyRecoveredEvent(nnk types.NamespacedNameKind) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
+ ObjectMeta: metav1.ObjectMeta{Name: nnk.Name, Namespace: nnk.Namespace},
+ }
+ d.EventRecorder.Event(gatewayProxy, corev1.EventTypeNormal, GatewayProxyReasonDataPlaneAvailable,
+ "every data plane instance took the last sync")
+}
+
+// recordFailedEndpointEvents fires one Warning event per failed EndpointStatus entry,
+// so each instance's own failure history (when it started, how often) is visible on
+// its own, not folded into everyone else's.
+func (d *apisixProvider) recordFailedEndpointEvents(nnk types.NamespacedNameKind, endpoints []adctypes.EndpointStatus) {
+ if d.EventRecorder == nil {
+ return
+ }
+ gatewayProxy := &apiv1alpha1.GatewayProxy{
Review Comment:
fixed
##########
api/v1alpha1/gatewayproxy_types.go:
##########
@@ -178,6 +179,19 @@ type GatewayProxy struct {
// GatewayProxySpec defines configuration of gateway proxy instances,
// including networking settings, global plugins, and plugin metadata.
Spec GatewayProxySpec `json:"spec,omitempty"`
+
+ // Status defines the current state of Gateway Proxy.
+ //
+ // +kubebuilder:default={conditions: {{type: "DataPlaneAvailable", status: "Unknown", reason:"Pending", message:"Waiting for controller", lastTransitionTime: "1970-01-01T00:00:00Z"}}}
Review Comment:
fixed
##########
internal/provider/apisix/status.go:
##########
@@ -42,32 +55,101 @@ import (
func (d *apisixProvider) handleStatusUpdate(statusUpdateMap map[types.NamespacedNameKind][]string) {
// Mark all resources in the current failure set as failed.
for nnk, msgs := range statusUpdateMap {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonSyncFailed,
- false,
- 0,
- strings.Join(msgs, "; "),
- ))
+ d.updateStatus(nnk, failureCondition(nnk, strings.Join(msgs, "; ")))
}
// Mark resources that exist only in the previous failure history as successful.
for nnk := range d.statusUpdateMap {
if _, ok := statusUpdateMap[nnk]; !ok {
- d.updateStatus(nnk, cutils.NewConditionTypeAccepted(
- apiv2.ConditionReasonAccepted,
- true,
- 0,
- "",
- ))
+ d.updateStatus(nnk, successCondition(nnk))
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: report dp instance unavailable to gateway proxy [apisix-ingress-controller]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #2856:
URL: https://github.com/apache/apisix-ingress-controller/pull/2856#discussion_r3947161008
##########
api/v1alpha1/gatewayproxy_types.go:
##########
@@ -178,6 +179,19 @@ type GatewayProxy struct {
// GatewayProxySpec defines configuration of gateway proxy instances,
// including networking settings, global plugins, and plugin metadata.
Spec GatewayProxySpec `json:"spec,omitempty"`
+
+ // Status defines the current state of Gateway Proxy.
+ //
+ // +kubebuilder:default={conditions: {{type: "DataPlaneAvailable", status: "Unknown", reason:"Pending", message:"Waiting for controller", lastTransitionTime: "1970-01-01T00:00:00Z"}}}
Review Comment:
This default is never overwritten on the happy path — an always-healthy GatewayProxy reports `Waiting for controller` forever. That contradicts the "good" example in the PR description, where `status: "True"` is shown as the healthy steady state.
Cause: `handleStatusUpdate` writes the success condition only for keys that were in the *previous* failure set, and nothing else ever writes `DataPlaneAvailable`. Two effects:
- healthy from the start never turns `True`
- a stale `False` survives a restart — `d.statusUpdateMap` is in-memory, so the success branch never fires after a restart
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org