You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "bzp2010 (via GitHub)" <gi...@apache.org> on 2026/09/21 07:47:11 UTC

[PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

bzp2010 opened a new pull request, #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890

   <!-- Please answer these questions before submitting a pull request -->
   
   ### Type of change:
   
   - [x] New feature provided
   
   ### What this PR does / why we need it:
   
   Part of #2877, the first feature slice after the store groundwork in #2880, #2883, #2884 and #2885. The rest of #2877 lands as separate PRs on top of this one, each with its own e2e.
   
   Today one resource the data plane rejects fails the whole sync for its GatewayProxy, so every other ApisixRoute, Ingress or Gateway API route under the same GatewayProxy stops updating until someone fixes it. With this change only the rejected resource is dropped.
   
   - When ADC reports a rejection on a service, or on a route, stream route or named upstream inside one, the service (one Kubernetes rule) is added to a skip table and left out of every following push. The rest of the config is pushed and the sync converges.
   - The resource that owns the service reports why: `Accepted=False` with reason `SyncFailed` for the ApisixRoute and Gateway API kinds, and a Warning event for an Ingress, which has no conditions to carry it. The GatewayProxy no longer reports the sync as failed for a rejection that names a resource.
   - An excluded service comes back when its owner is written again with different content. Rewriting identical content, which unrelated reconciles do all the time, does not retry a known-bad resource. A failure reported against content that has since been replaced is ignored, so a fix can never be excluded by a stale report. Both checks use store revisions, which only advance when what the store holds actually changes.
   - A round that excluded something new pushes again right away instead of waiting for the retry backoff, since that push carries every other resource's pending changes.
   
   What it does not do yet, all in follow-up PRs: dropping just one bad route of a service, reporting a partially served resource as `PartiallyInvalid`, and isolating consumers, credentials, certificates, global rules, GatewayProxy plugins and Gateway listener certificates. Rejections of those types are reported as before. Until the partial reporting lands, an ApisixRoute with several rules where only one is rejected is reported `Accepted=False` with the dropped rule named in the message, while its other rules are served.
   
   e2e covers a rejected route, an upstream configuration the data plane rejects, a rejected named upstream, a rejected stream route and a rejected Ingress, each applied together with a valid resource in the same first sync and then fixed to check that everything converges.
   
   ### Pre-submission checklist:
   
   - [x] Did you explain what problem does this PR solve? Or what new features have been added?
   - [x] Have you added corresponding test cases?
   - [ ] Have you modified the corresponding document?
   - [x] Is this PR backward compatible?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4080941140


##########
internal/provider/apisix/status.go:
##########
@@ -137,13 +173,50 @@ func (d *apisixProvider) classifySyncResult(
 	return gatewayProxyMsgs, failedEndpoints
 }
 
+// dropUnit maps a rejected resource to what has to be dropped for the rest to apply. A
+// service is dropped by itself. A route, stream route or named upstream is dropped along
+// with the service it lives in: the service is what one Kubernetes rule translates to,
+// and a named upstream is referenced by the service's traffic-split by id, so dropping
+// it alone would leave that reference dangling. Nested resources are found through their
+// parent, which also tells them apart when several services embed upstreams of the same
+// id.
+func (d *apisixProvider) dropUnit(configName string, ev adctypes.StatusEvent) (wireKey, exclusion, bool) {
+	serviceID := ev.ResourceID
+	switch ev.ResourceType {
+	case adctypes.TypeService:
+	case adctypes.TypeRoute, adctypes.TypeStreamRoute, adctypes.TypeUpstream:
+		serviceID = ev.ParentID
+	default:
+		return wireKey{}, exclusion{}, false
+	}
+	service, ok := d.store.Lookup(configName, adctypes.TypeService, serviceID)
+	if !ok {
+		return wireKey{}, exclusion{}, false
+	}
+	owner, name := service.Owner, service.Name
+	// A route's own owner can differ from its service's: a traffic-split service can
+	// combine rules several ApisixRoutes each contributed. Attribute to the route
+	// itself when the store can tell them apart, so fixing the actual bad route clears
+	// the exclusion instead of leaving it stuck on whichever owner the shared service
+	// happens to carry.
+	if ev.ResourceType == adctypes.TypeRoute {
+		if route, ok := d.store.Lookup(configName, adctypes.TypeRoute, ev.ResourceID); ok && route.Owner != (types.NamespacedNameKind{}) {
+			owner = route.Owner
+		}
+	}
+	return wireKey{adctypes.TypeService, service.ID}, exclusion{owner: owner, name: name}, true
+}
+
 // applyResourceFailures writes this round's newly (or still) failing resources, and
 // clears the recorded error from any resource that was failing last round but isn't in
 // newFailures now. See updateStatusFromSyncResults for why resources use this delta
 // instead of GatewayProxy's full recompute.
-func (d *apisixProvider) applyResourceFailures(newFailures map[types.NamespacedNameKind][]string) {
+func (d *apisixProvider) applyResourceFailures(ctx context.Context, newFailures map[types.NamespacedNameKind][]string) {
 	for resourceKey, msgs := range newFailures {
 		d.updateStatus(resourceKey, failureCondition(resourceKey, strings.Join(msgs, "; ")))
+		if resourceKey.Kind == types.KindIngress {
+			d.recordIngressFailureEvent(ctx, resourceKey, msgs)
+		}
 	}

Review Comment:
   This is intentional, not an oversight: client-go's EventRecorder correlates repeats of the same reason and message against the same object into one Event, PATCHing its count and lastTimestamp instead of creating a new object each time. Firing every round is what keeps a long-lived failure's lastTimestamp refreshed so it doesn't age out of `kubectl get events` once the default TTL passes; deduping on our own end would undo that. The Get is a single object read per failing Ingress per round, not a list, so its cost scales with how many Ingresses are actually failing rather than with sync frequency alone.



##########
internal/provider/apisix/status.go:
##########
@@ -153,6 +226,21 @@ func (d *apisixProvider) applyResourceFailures(newFailures map[types.NamespacedN
 	d.resourceFailures = newFailures
 }
 
+// recordIngressFailureEvent fires a Warning event for an Ingress whose status has no
+// conditions to carry a failure. It is fired every round the Ingress stays failing, since
+// events expire.
+func (d *apisixProvider) recordIngressFailureEvent(ctx context.Context, nnk types.NamespacedNameKind, msgs []string) {

Review Comment:
   Same as above, this is the same call site's correlation behavior; see the reply above.



##########
internal/provider/apisix/status.go:
##########
@@ -153,6 +226,21 @@ func (d *apisixProvider) applyResourceFailures(newFailures map[types.NamespacedN
 	d.resourceFailures = newFailures
 }
 
+// recordIngressFailureEvent fires a Warning event for an Ingress whose status has no
+// conditions to carry a failure. It is fired every round the Ingress stays failing, since
+// events expire.
+func (d *apisixProvider) recordIngressFailureEvent(ctx context.Context, nnk types.NamespacedNameKind, msgs []string) {
+	if d.EventRecorder == nil || d.K8sClient == nil {
+		return
+	}
+	ingress := &networkingv1.Ingress{}
+	if err := d.K8sClient.Get(ctx, nnk.NamespacedName(), ingress); err != nil {
+		d.log.Error(err, "failed to get Ingress to record a failure event", "name", nnk.Name, "namespace", nnk.Namespace)
+		return
+	}
+	d.EventRecorder.Event(ingress, corev1.EventTypeWarning, string(apiv2.ConditionReasonSyncFailed), strings.Join(msgs, "; "))
+}

Review Comment:
   Same as above, this is the same call site's correlation behavior; see the reply above.



##########
internal/provider/apisix/provider.go:
##########
@@ -287,6 +292,11 @@ func (d *apisixProvider) Delete(ctx context.Context, obj client.Object) error {
 // applyResourceState upserts a resource's config associations and its contribution to each
 // target config's cached resource snapshot, the AIC-side bookkeeping the adc client
 // package no longer holds itself.
+// applyResourceState upserts a resource's config associations and its contribution to each
+// target config's cached resource snapshot. Whatever the skip table excluded for a
+// resource whose content this changed gets another try; rewriting identical content, which
+// reconciles triggered by unrelated events do all the time, must not retry a known-bad
+// resource.

Review Comment:
   Fixed in 7a278941: collapsed into a single doc comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4089076345


##########
internal/provider/apisix/status.go:
##########
@@ -121,13 +152,21 @@ func (d *apisixProvider) classifySyncResult(
 
 			anyUnattributed := false
 			for _, syncStatus := range addrErr.FailedStatuses {
+				reason := fmt.Sprintf("ServerAddr: %s, Error: %s", addrErr.ServerAddr, syncStatus.Reason)
+				if key, ex, ok := d.dropUnit(configName, syncStatus.Event); ok {
+					if previous, ok := dropped[key]; ok {
+						reason = previous.reason + "; " + reason
+					}
+					ex.reason = reason
+					dropped[key] = ex
+					continue

Review Comment:
   A resource-attributed rejection still marks the GatewayProxy unavailable here. `endpointMsg` is appended before this branch, and real 422 responses carry `EndpointStatuses` with `success:false` that restate the same rejection (`internal/adc/client/executor_test.go:311-335`). Consequently `gatewayProxyMsgs` remains non-empty even when every failed status is successfully converted into an exclusion, contrary to this PR's requirement that a named resource rejection not fail the GatewayProxy. Filter endpoint failures that merely duplicate an attributed rejection while retaining independent transport/instance failures.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4068429082


##########
internal/adc/cache/store.go:
##########
@@ -86,6 +100,52 @@ func gatewayProxyOf(name string) (types.NamespacedNameKind, bool) {
 	return gatewayProxy, true
 }
 
+func (s *Store) recordChange(name string, owner types.NamespacedNameKind) {
+	s.revision++
+	if s.lastChange[name] == nil {
+		s.lastChange[name] = make(map[types.NamespacedNameKind]uint64)
+	}
+	s.lastChange[name][owner] = s.revision

Review Comment:
   These owner entries are never removed when an individual resource is deleted; only `DeleteAll` drops the entire per-config map. A long-lived GatewayProxy with resource-name churn therefore retains one key (including its strings) for every historical owner, causing unbounded controller memory growth. Please prune tombstones once no in-flight sync can refer to their revisions, or otherwise bound this history.



##########
internal/provider/apisix/status.go:
##########
@@ -137,13 +174,39 @@ func (d *apisixProvider) classifySyncResult(
 	return gatewayProxyMsgs, failedEndpoints
 }
 
+// dropUnit maps a rejected resource to what has to be dropped for the rest to apply. A
+// service is dropped by itself. A route, stream route or named upstream is dropped along
+// with the service it lives in: the service is what one Kubernetes rule translates to,
+// and a named upstream is referenced by the service's traffic-split by id, so dropping
+// it alone would leave that reference dangling. Nested resources are found through their
+// parent, which also tells them apart when several services embed upstreams of the same
+// id.
+func (d *apisixProvider) dropUnit(configName string, ev adctypes.StatusEvent) (wireKey, exclusion, bool) {
+	serviceID := ev.ResourceID
+	switch ev.ResourceType {
+	case adctypes.TypeService:
+	case adctypes.TypeRoute, adctypes.TypeStreamRoute, adctypes.TypeUpstream:
+		serviceID = ev.ParentID
+	default:
+		return wireKey{}, exclusion{}, false
+	}
+	service, ok := d.store.Lookup(configName, adctypes.TypeService, serviceID)
+	if !ok {
+		return wireKey{}, exclusion{}, false
+	}
+	return wireKey{adctypes.TypeService, service.ID}, exclusion{owner: service.Owner, name: service.Name}, true

Review Comment:
   For a rejected route, this attributes the dropped service to the service owner rather than to the rejected route's owner. The cache explicitly supports routes whose own labels differ from their containing service (`internal/adc/cache/store.go:160-176`), so in that case the wrong resource gets `SyncFailed`, and editing the actual bad route will not clear the exclusion. Keep dropping the parent service, but derive the exclusion owner from the route event when available.



##########
internal/provider/apisix/status.go:
##########
@@ -58,23 +61,29 @@ const (
 // get a True the first time, and what keeps a restart from leaving a stale False stuck
 // forever: the write only ever depends on this round's actual outcome.
 //
-// Resource status can't afford the same full recompute: a config's resource set can be
-// large, and rewriting every one of them every round even when nothing changed would be
-// wasteful. So resources keep a small persisted delta in d.resourceFailures instead:
-// newly (or still) failing resources are written SyncFailed, and any resource that was
-// failing last round but isn't failing this one gets its error explicitly cleared with
-// an Accepted write.
-func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors) {
+// Resource status follows the whole skip table instead, since an excluded resource stays
+// dropped until its owner is written again. A config's resource set can be large, so
+// resources keep a small persisted delta in d.resourceFailures rather than being rewritten
+// every round: resources that are dropped or failing are written SyncFailed, and any
+// resource that was last round but isn't now gets its error explicitly cleared with an
+// Accepted write.
+//
+// It reports whether anything was newly excluded, which is what makes the next push
+// different from the one that just failed: see sync().
+func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors, revisions map[string]uint64) bool {
 	resourceFailures := map[types.NamespacedNameKind][]string{}
+	newlyExcluded := 0
 
 	for configName, execErrs := range results {
+		dropped := map[wireKey]exclusion{}
+		gatewayProxyMsgs, failedEndpoints := d.classifySyncResult(configName, execErrs, revisions[configName], dropped, resourceFailures)
+		newlyExcluded += d.skipped.MarkFailing(configName, dropped)

Review Comment:
   `ChangedSince` and `MarkFailing` are separate critical sections. If a reconcile writes fixed content after the stale check but before this insertion, `applyResourceState` clears the old exclusions first and this code then re-adds the stale exclusion; with no later content change, the fixed resource remains excluded indefinitely. Serialize classification plus insertion with the same provider lock used by `applyResourceState`/`removeResourceState`, or make the revision check and insertion atomic.



##########
test/e2e/ingress/ingress.go:
##########
@@ -1600,4 +1600,115 @@ spec:
 		})
 
 	})
+
+	Context("Bad resource isolation", func() {
+		var gatewayProxy = `
+apiVersion: apisix.apache.org/v1alpha1
+kind: GatewayProxy
+metadata:
+  name: apisix-proxy-config
+  namespace: %s
+spec:
+  provider:
+    type: ControlPlane
+    controlPlane:
+      endpoints:
+      - %s
+      auth:
+        type: AdminKey
+        adminKey:
+          value: "%s"
+`
+		var ingressClass = `
+apiVersion: networking.k8s.io/v1
+kind: IngressClass
+metadata:
+  name: %s
+spec:
+  controller: "%s"
+  parameters:
+    apiGroup: "apisix.apache.org"
+    kind: "GatewayProxy"
+    name: "apisix-proxy-config"
+    namespace: "%s"
+    scope: "Namespace"
+`
+		// The Ingress picks up its plugins from an ApisixPluginConfig named in an
+		// annotation; limit-count refuses a count that is not greater than 0.
+		var rejectedPluginConfig = `
+apiVersion: apisix.apache.org/v2
+kind: ApisixPluginConfig
+metadata:
+  name: rejected-plugins
+spec:
+  ingressClassName: %s
+  plugins:
+  - name: limit-count
+    enable: true
+    config:
+      count: 0
+      time_window: 60
+      rejected_code: 503
+      key: remote_addr
+`
+		var ingressTemplate = `
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: rejected
+%s
+spec:
+  ingressClassName: %s
+  rules:
+  - host: rejected-ingress.example.com
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: httpbin-service-e2e-test
+            port:
+              number: 80
+`
+		const withRejectedPlugins = `  annotations:
+    k8s.apisix.apache.org/plugin-config-name: rejected-plugins`
+
+		It("a rejected Ingress is reported as an event", func() {
+			By("create GatewayProxy")
+			err := s.CreateResourceFromStringWithNamespace(fmt.Sprintf(gatewayProxy, s.Namespace(), s.Deployer.GetAdminEndpoint(), s.AdminKey()), s.Namespace())
+			Expect(err).NotTo(HaveOccurred(), "creating GatewayProxy")
+
+			By("create IngressClass")
+			err = s.CreateResourceFromStringWithNamespace(fmt.Sprintf(ingressClass, s.Namespace(), s.GetControllerName(), s.Namespace()), "")
+			Expect(err).NotTo(HaveOccurred(), "creating IngressClass")
+
+			By("create an Ingress whose plugins the data plane rejects")
+			err = s.CreateResourceFromString(fmt.Sprintf(rejectedPluginConfig, s.Namespace()))
+			Expect(err).NotTo(HaveOccurred(), "creating ApisixPluginConfig")
+			err = s.CreateResourceFromString(fmt.Sprintf(ingressTemplate, withRejectedPlugins, s.Namespace()))
+			Expect(err).NotTo(HaveOccurred(), "creating Ingress")

Review Comment:
   This case creates only the rejected Ingress. It will still pass if that rejection blocks the entire GatewayProxy, because it checks the event and only sends traffic after fixing the Ingress. Add a valid sibling Ingress in the same initial push and assert it is served before the fix to cover Ingress isolation rather than only reporting and recovery.



##########
test/e2e/crds/v2/isolation.go:
##########
@@ -0,0 +1,255 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package v2
+
+import (
+	"fmt"
+	"net/http"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	gomegatypes "github.com/onsi/gomega/types"
+
+	"github.com/apache/apisix-ingress-controller/test/e2e/scaffold"
+)
+
+// Every case applies a valid and a rejected resource together, so the first sync carries
+// both: the sync converges when the rejected one is excluded and the valid one is served.
+// Fixing the rejected one then has to bring it in too.
+//
+// What makes a resource rejected is always something the data plane checks for every
+// backend: a value outside the schema's range, or a known plugin configured in a way its
+// own check_schema refuses. An unknown plugin name is not used, since apisix-standalone
+// accepts those.
+var _ = Describe("Test bad resource isolation", Label("apisix.apache.org", "v2", "isolation"), func() {
+	s := scaffold.NewDefaultScaffold()
+
+	// rejectedPlugin is a plugin every backend loads, configured with a count the plugin
+	// itself refuses (it has to be greater than 0).
+	const rejectedPlugin = `
+    plugins:
+    - name: limit-count
+      enable: true
+      config:
+        count: 0
+        time_window: 60
+        rejected_code: 503
+        key: remote_addr
+`
+	// A consistent-hash load balancer needs a key to hash on. The data plane checks that
+	// in code rather than in its schema, so the configuration gets past ADC's schema
+	// check and is rejected only by the data plane itself.
+	const rejectedHashKey = ""
+	const acceptedHashKey = "    key: remote_addr"
+	const acceptedPlugin = `
+    plugins:
+    - name: limit-count
+      enable: true
+      config:
+        count: 100
+        time_window: 60
+        rejected_code: 503
+        key: remote_addr
+`
+	// routes is one ApisixRoute serving valid.example.com and one serving
+	// rejected.example.com, whose plugin configuration is filled in per case.
+	const routes = `
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+  name: valid
+  namespace: %s
+spec:
+  ingressClassName: %s
+  http:
+  - name: rule0
+    match:
+      hosts:
+      - valid.example.com
+      paths:
+      - /*
+    backends:
+    - serviceName: httpbin-service-e2e-test
+      servicePort: 80
+---
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+  name: rejected
+  namespace: %s
+spec:
+  ingressClassName: %s
+  http:
+  - name: rule0
+    match:
+      hosts:
+      - rejected.example.com
+      paths:
+      - /*
+    backends:
+    - serviceName: httpbin-service-e2e-test
+      servicePort: 80
+%s
+`
+
+	expectServed := func(host string) {
+		s.RequestAssert(&scaffold.RequestAssert{
+			Method: "GET",
+			Path:   "/get",
+			Host:   host,
+			Check:  scaffold.WithExpectedStatus(http.StatusOK),
+		})
+	}
+	expectStatus := func(resource, name string, matchers ...gomegatypes.GomegaMatcher) {
+		s.RetryAssertion(func() string {
+			output, _ := s.GetOutputFromString(resource, name, "-o", "yaml", "-n", s.Namespace())
+			return output
+		}).Should(And(matchers...))
+	}
+	apply := func(yaml string) {
+		Expect(s.CreateResourceFromString(yaml)).NotTo(HaveOccurred(), "applying resources")
+	}
+
+	BeforeEach(func() {
+		By("create GatewayProxy")
+		Expect(s.CreateResourceFromString(s.GetGatewayProxySpec())).NotTo(HaveOccurred(), "creating GatewayProxy")
+
+		By("create IngressClass")
+		Expect(s.CreateResourceFromStringWithNamespace(s.GetIngressClassYaml(), "")).NotTo(HaveOccurred(), "creating IngressClass")
+	})
+
+	It("isolates a rejected route", func() {
+		By("apply a valid and a rejected ApisixRoute")
+		apply(fmt.Sprintf(routes, s.Namespace(), s.Namespace(), s.Namespace(), s.Namespace(), rejectedPlugin))
+
+		By("the valid ApisixRoute is served and the rejected one reports why it is not")
+		expectServed("valid.example.com")
+		// The rejected route is the only one this ApisixRoute has, so none of it is served.
+		expectStatus("ar", "rejected",
+			ContainSubstring(`status: "False"`),
+			ContainSubstring(`reason: SyncFailed`),
+			ContainSubstring(`limit-count`),
+		)
+
+		By("fix the rejected ApisixRoute")
+		apply(fmt.Sprintf(routes, s.Namespace(), s.Namespace(), s.Namespace(), s.Namespace(), acceptedPlugin))
+
+		By("both ApisixRoutes are served")
+		expectServed("valid.example.com")
+		expectServed("rejected.example.com")
+		expectStatus("ar", "rejected", ContainSubstring(`reason: Accepted`))
+	})
+
+	It("isolates a route whose upstream configuration is rejected", func() {
+		const upstream = `
+apiVersion: apisix.apache.org/v2
+kind: ApisixUpstream
+metadata:
+  name: httpbin-service-e2e-test
+  namespace: %s
+spec:
+  ingressClassName: %s
+  loadbalancer:
+    type: chash
+    hashOn: vars
+%s
+`
+		By("apply a valid and a rejected ApisixRoute, the rejected one using a rejected upstream configuration")
+		apply(fmt.Sprintf(routes, s.Namespace(), s.Namespace(), s.Namespace(), s.Namespace(), ""))
+		apply(fmt.Sprintf(upstream, s.Namespace(), s.Namespace(), rejectedHashKey))

Review Comment:
   This setup does not actually leave a valid control route: both manifests in `routes` use `httpbin-service-e2e-test`, and an ApisixUpstream with that name is applied to every such backend. Consequently both routes receive the rejected hash configuration, and before the fix the test only checks one status without verifying that any route is served. Use a different backend for the valid route and call `expectServed` before fixing the upstream so the case fails if the whole sync is blocked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4080939837


##########
internal/adc/cache/store.go:
##########
@@ -86,6 +100,52 @@ func gatewayProxyOf(name string) (types.NamespacedNameKind, bool) {
 	return gatewayProxy, true
 }
 
+func (s *Store) recordChange(name string, owner types.NamespacedNameKind) {
+	s.revision++
+	if s.lastChange[name] == nil {
+		s.lastChange[name] = make(map[types.NamespacedNameKind]uint64)
+	}
+	s.lastChange[name][owner] = s.revision

Review Comment:
   Agreed this is a real gap: lastChange only ever grows, and DeleteAll is the only thing that prunes a cacheKey's whole map, never an individual owner. Pruning safely means knowing no in-flight sync result can still reference an owner's old revision, which touches the sync loop's own lifecycle rather than being a local change to this file, so leaving it for this PR would widen scope past what it's meant to review. Deferring this to a small follow-up PR rather than fixing it here.



##########
internal/adc/cache/store.go:
##########
@@ -118,11 +178,18 @@ func routeOwner(targetCache Cache, id string) (types.NamespacedNameKind, bool) {
 
 // setGlobalRules is Insert and SetGlobalRules' shared implementation. Callers must
 // already hold s.Lock.
-func (s *Store) setGlobalRules(targetCache Cache, owner types.NamespacedNameKind, plugins adctypes.GlobalRule) error {
+func (s *Store) setGlobalRules(name string, targetCache Cache, owner types.NamespacedNameKind, plugins adctypes.GlobalRule) error {

Review Comment:
   Fixed in 7a278941, together with the setPluginMetadata instance below: recordChange is no longer deferred, it's called explicitly right before each function's own successful return, so an error partway through (a failed Delete/InsertGlobalRule) no longer leaves a phantom revision bump behind.



##########
internal/adc/cache/store.go:
##########
@@ -118,11 +178,18 @@ func routeOwner(targetCache Cache, id string) (types.NamespacedNameKind, bool) {
 
 // setGlobalRules is Insert and SetGlobalRules' shared implementation. Callers must
 // already hold s.Lock.
-func (s *Store) setGlobalRules(targetCache Cache, owner types.NamespacedNameKind, plugins adctypes.GlobalRule) error {
+func (s *Store) setGlobalRules(name string, targetCache Cache, owner types.NamespacedNameKind, plugins adctypes.GlobalRule) error {
 	rows, err := targetCache.ListGlobalRules(&OwnerSelector{Owner: owner})
 	if err != nil {
 		return err
 	}
+	existing := make(adctypes.Plugins, len(rows))
+	for _, row := range rows {
+		existing[row.ID] = row.Config
+	}
+	if pluginsContent(existing) != pluginsContent(adctypes.Plugins(plugins)) {
+		defer s.recordChange(name, owner)
+	}

Review Comment:
   Same fix as setGlobalRules above, in 7a278941.



##########
internal/adc/cache/store.go:
##########
@@ -86,6 +100,52 @@ func gatewayProxyOf(name string) (types.NamespacedNameKind, bool) {
 	return gatewayProxy, true
 }
 
+func (s *Store) recordChange(name string, owner types.NamespacedNameKind) {
+	s.revision++
+	if s.lastChange[name] == nil {
+		s.lastChange[name] = make(map[types.NamespacedNameKind]uint64)
+	}
+	s.lastChange[name][owner] = s.revision
+}
+
+// contentOf renders items, ordered by id, the way they reach ADC, so two writes can be
+// compared by what they would push.
+func contentOf[T any](items []T, id func(T) string) string {
+	if len(items) == 0 {
+		return ""
+	}
+	sorted := slices.Clone(items)
+	slices.SortFunc(sorted, func(a, b T) int { return strings.Compare(id(a), id(b)) })
+	return canonicalJSON(sorted)
+}
+
+// canonicalJSON renders v as JSON with every object's keys sorted. A stored object's
+// plugin configs have been through a JSON round trip while a freshly translated one may
+// still hold typed structs, whose fields marshal in declaration order rather than sorted.
+func canonicalJSON(v any) string {
+	b, err := json.Marshal(v)
+	if err != nil {
+		return ""
+	}
+	var generic any
+	if err := json.Unmarshal(b, &generic); err != nil {
+		return string(b)
+	}
+	b, _ = json.Marshal(generic)
+	return string(b)
+}

Review Comment:
   canonicalJSON now logs both marshal failures instead of swallowing them (7a278941), so a real occurrence is visible. Not changing the two call sites to force changed semantics on error for now: v here is always a type this package built from already-parsed JSON, so a Marshal failure is defensive rather than expected, and forcing changed would mean plumbing an ok/error signal back through contentOf and pluginsContent into every one of their callers for a path that shouldn't be reachable in practice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4082261231


##########
internal/adc/cache/store.go:
##########
@@ -460,3 +566,29 @@ func (s *Store) OwnedEntities(name string, owner types.NamespacedNameKind) []Ent
 	}
 	return entities
 }
+
+// Revision returns the store's current revision.
+func (s *Store) Revision() uint64 {
+	s.Lock()
+	defer s.Unlock()
+	return s.revision
+}
+
+// ChangedSince reports whether owner's content in the cacheKey name, or the cacheKey as a
+// whole, changed after revision: a sync result built at revision then no longer describes
+// what the store holds for owner.
+func (s *Store) ChangedSince(name string, owner types.NamespacedNameKind, revision uint64) bool {
+	s.Lock()
+	defer s.Unlock()
+	return s.lastChange[name][owner] > revision || s.resetAt[name] > revision
+}
+
+// OwnerChangedSince reports whether owner's content changed after revision in any
+// cacheKey.
+func (s *Store) OwnerChangedSince(owner types.NamespacedNameKind, revision uint64) bool {
+	s.Lock()
+	defer s.Unlock()
+	return slices.ContainsFunc(slices.Collect(maps.Values(s.lastChange)), func(byOwner map[types.NamespacedNameKind]uint64) bool {

Review Comment:
   `maps.Values(s.lastChange)` already returns a slice, so wrapping it in `slices.Collect(...)` is invalid and likely won’t compile. Use `slices.ContainsFunc(maps.Values(s.lastChange), ...)` directly, or iterate with a simple `for _, byOwner := range maps.Values(...)` loop.



##########
internal/adc/cache/store.go:
##########
@@ -86,6 +100,61 @@ func gatewayProxyOf(name string) (types.NamespacedNameKind, bool) {
 	return gatewayProxy, true
 }
 
+func (s *Store) recordChange(name string, owner types.NamespacedNameKind) {
+	s.revision++
+	if s.lastChange[name] == nil {
+		s.lastChange[name] = make(map[types.NamespacedNameKind]uint64)
+	}
+	s.lastChange[name][owner] = s.revision
+}
+
+// contentOf renders items, ordered by id, the way they reach ADC, so two writes can be
+// compared by what they would push.
+func contentOf[T any](log logr.Logger, items []T, id func(T) string) string {
+	if len(items) == 0 {
+		return ""
+	}
+	sorted := slices.Clone(items)
+	slices.SortFunc(sorted, func(a, b T) int { return strings.Compare(id(a), id(b)) })
+	return canonicalJSON(log, sorted)
+}
+
+// canonicalJSON renders v as JSON with every object's keys sorted. A stored object's
+// plugin configs have been through a JSON round trip while a freshly translated one may
+// still hold typed structs, whose fields marshal in declaration order rather than sorted.
+//
+// A marshal error here is logged rather than propagated: v is always a type this package
+// itself built from already-parsed JSON, so this is defensive rather than expected, and
+// callers use the result only for change detection, not for anything that reaches ADC.
+func canonicalJSON(log logr.Logger, v any) string {
+	b, err := json.Marshal(v)
+	if err != nil {
+		log.Error(err, "failed to marshal value for change detection")
+		return ""
+	}
+	var generic any
+	if err := json.Unmarshal(b, &generic); err != nil {
+		return string(b)
+	}
+	if sorted, err := json.Marshal(generic); err != nil {
+		log.Error(err, "failed to re-marshal value for change detection")
+	} else {
+		b = sorted
+	}
+	return string(b)
+}

Review Comment:
   This does a marshal → unmarshal → marshal cycle for every change-detection comparison, which can get expensive for large resource sets or frequent reconciles. If this becomes hot, consider replacing it with a cheaper stable fingerprint (e.g., hashing `json.Marshal` output once per write and caching per-owner, or a structural comparison limited to fields that affect the ADC payload) to reduce allocations and CPU.



##########
internal/provider/apisix/status.go:
##########
@@ -58,23 +61,32 @@ const (
 // get a True the first time, and what keeps a restart from leaving a stale False stuck
 // forever: the write only ever depends on this round's actual outcome.
 //
-// Resource status can't afford the same full recompute: a config's resource set can be
-// large, and rewriting every one of them every round even when nothing changed would be
-// wasteful. So resources keep a small persisted delta in d.resourceFailures instead:
-// newly (or still) failing resources are written SyncFailed, and any resource that was
-// failing last round but isn't failing this one gets its error explicitly cleared with
-// an Accepted write.
-func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors) {
+// Resource status follows the whole skip table instead, since an excluded resource stays
+// dropped until its owner is written again. A config's resource set can be large, so
+// resources keep a small persisted delta in d.resourceFailures rather than being rewritten
+// every round: resources that are dropped or failing are written SyncFailed, and any
+// resource that was last round but isn't now gets its error explicitly cleared with an
+// Accepted write.
+//
+// It reports whether anything was newly excluded, which is what makes the next push
+// different from the one that just failed: see sync().
+func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors, revisions map[string]uint64) bool {
 	resourceFailures := map[types.NamespacedNameKind][]string{}
+	newlyExcluded := 0
 
 	for configName, execErrs := range results {
+		dropped := map[wireKey]exclusion{}
+		gatewayProxyMsgs, failedEndpoints := d.classifySyncResult(configName, execErrs, dropped, resourceFailures)
+		builtRevision := revisions[configName]
+		newlyExcluded += d.skipped.MarkFailing(configName, dropped, func(owner types.NamespacedNameKind) bool {
+			return d.store.ChangedSince(configName, owner, builtRevision)
+		})

Review Comment:
   `builtRevision := revisions[configName]` silently falls back to `0` if the key is missing, which would make `ChangedSince(...)` likely treat many exclusions as stale and skip recording them. Consider checking the map lookup (`rev, ok := revisions[configName]`) and logging/handling the unexpected missing-revision case explicitly so failures aren’t dropped silently if future refactors change how `results`/`revisions` are populated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4119135842


##########
internal/provider/apisix/status.go:
##########
@@ -137,13 +176,50 @@ func (d *apisixProvider) classifySyncResult(
 	return gatewayProxyMsgs, failedEndpoints
 }
 
+// dropUnit maps a rejected resource to what has to be dropped for the rest to apply. A
+// service is dropped by itself. A route, stream route or named upstream is dropped along
+// with the service it lives in: the service is what one Kubernetes rule translates to,
+// and a named upstream is referenced by the service's traffic-split by id, so dropping
+// it alone would leave that reference dangling. Nested resources are found through their
+// parent, which also tells them apart when several services embed upstreams of the same
+// id.
+func (d *apisixProvider) dropUnit(configName string, ev adctypes.StatusEvent) (wireKey, exclusion, bool) {

Review Comment:
   `dropUnit` treats every `failed[]` entry as a content rejection, but on the `apisix` and `api7ee` backends ADC also reports transient failures there. Right now a single 5xx or transport error can permanently exclude a valid service and delete it from the data plane.
   
   **Where transient failures come from (ADC master `63a0937`)**
   
   - The ADC server runs sync with `exit_on_failure: false` (Rust `adc-cli/src/server/sync.rs:95`, Node `apps/cli/src/server/sync.ts:115`). So a failed event does not abort the sync. It is returned in `failed[]` with its full `event`, and the response is still a 202.
   - **api7ee:** a failed request is not retried at all. See Rust `adc-backend-api7/src/operator.rs:137`, whose file header says "No retry wrapping here … a failed request is never retried", and Node `libs/backend-api7/src/operator.ts:79`. Any one 5xx, 502/504 from an LB, or reset connection to the control plane during a sync becomes a `failed[]` entry.
   - **apisix:** 5xx and transport errors are retried 3 times, 100ms apart (`adc-backend-core/src/retry.rs:23`). The Admin API returns `503` whenever an etcd write fails (`apisix/admin/resource.lua:253`, `:332`). An etcd leader election usually takes longer than ~300ms, so if one overlaps a sync that has already passed dump, the retries run out.
   - **apisix-standalone** is not affected: the Rust server fills `failed[]` only from `/validate`.
   - The Rust `FailedEntry` has no HTTP status (`sync.rs:139`), so AIC cannot even separate these by status code today.
   
   **Lifecycle once an entry is recorded**
   
   1. `MarkFailing` records the service. Every later push leaves it out (`provider.go:593`), including periodic `SyncPeriod` rounds, so the ADC diff DELETEs the previously serving service and its routes.
   2. Successful rounds never clear the entry (by design, see `status_test.go:545`).
   3. The entry is cleared only when:
      - the owner's translated content changes,
      - the owner is deleted or evicted,
      - the GatewayProxy config is removed, or
      - the controller restarts or leadership changes.
   
      For a transient failure the CR is already correct, so none of these is expected to happen. The route stays down while the GatewayProxy reports `DataPlaneAvailable=True`.
   
   When etcd becomes unavailable partway through a large sync, every event after that point fails. The corresponding services are all excluded, and once etcd recovers they are all deleted from the data plane with no self-healing. Before this PR, the whole sync failed and was retried with backoff until it converged.
   
   **Repro (unit level)**
   
   ```go
   func TestTransient5xxExcludesStickily(t *testing.T) {
   	d, route := newDroppingProvider(t)
   	rev := map[string]uint64{"GatewayProxy/ns/gp": d.store.Revision()}
   	results := map[string]types.ADCExecutionErrors{"GatewayProxy/ns/gp": failedEvent(
   		adctypes.StatusEvent{ResourceType: adctypes.TypeRoute, ResourceID: "r1", ParentID: "svc1", Type: "update"},
   		"API error (status 503): etcd request timed out")}
   	require.True(t, d.updateStatusFromSyncResults(context.Background(), results, rev))
   
   	for i := 0; i < 10; i++ { // data plane recovered, every round succeeds
   		d.updateStatusFromSyncResults(context.Background(), map[string]types.ADCExecutionErrors{"GatewayProxy/ns/gp": {}}, rev)
   	}
   	require.Contains(t, d.skipped.Excluded("GatewayProxy/ns/gp"), wireKey{adctypes.TypeService, "svc1"}) // passes
   	require.NotEmpty(t, d.resourceFailures[route])                                                       // passes
   }
   ```
   
   **Possible directions**
   
   - Have ADC mark each `failed[]` entry as a content rejection (non-retriable 4xx, version gate) or a retriable failure (5xx, transport), and exclude only content rejections. This needs an ADC change first.
   - Or give exclusions a bounded lifetime on the AIC side, for example re-including them with backoff, so a false positive heals without a content change or a restart.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4119135842


##########
internal/provider/apisix/status.go:
##########
@@ -137,13 +176,50 @@ func (d *apisixProvider) classifySyncResult(
 	return gatewayProxyMsgs, failedEndpoints
 }
 
+// dropUnit maps a rejected resource to what has to be dropped for the rest to apply. A
+// service is dropped by itself. A route, stream route or named upstream is dropped along
+// with the service it lives in: the service is what one Kubernetes rule translates to,
+// and a named upstream is referenced by the service's traffic-split by id, so dropping
+// it alone would leave that reference dangling. Nested resources are found through their
+// parent, which also tells them apart when several services embed upstreams of the same
+// id.
+func (d *apisixProvider) dropUnit(configName string, ev adctypes.StatusEvent) (wireKey, exclusion, bool) {

Review Comment:
   `dropUnit` treats every `failed[]` entry as a content rejection, but on the `apisix` backend ADC also reports transient failures there. Right now a single 5xx or transport error can permanently exclude a valid service and delete it from the data plane.
   
   **Where transient failures come from (ADC master `63a0937`)**
   
   - The ADC server runs sync with `exit_on_failure: false` (Rust `adc-cli/src/server/sync.rs:95`, Node `apps/cli/src/server/sync.ts:115`). So a failed event does not abort the sync. It is returned in `failed[]` with its full `event`, and the response is still a 202.
   - 5xx and transport errors are retried 3 times, 100ms apart (`adc-backend-core/src/retry.rs:23`). The Admin API returns `503` whenever an etcd write fails (`apisix/admin/resource.lua:253`, `:332`). An etcd leader election usually takes longer than ~300ms, so if one overlaps a sync that has already passed dump, the retries run out.
   - The `apisix-standalone` backend is not affected: the Rust server fills `failed[]` only from `/validate`.
   - The Rust `FailedEntry` has no HTTP status (`sync.rs:139`), so AIC cannot even separate these by status code today.
   
   **Lifecycle once an entry is recorded**
   
   1. `MarkFailing` records the service. Every later push leaves it out (`provider.go:593`), including periodic `SyncPeriod` rounds, so the ADC diff DELETEs the previously serving service and its routes.
   2. Successful rounds never clear the entry (by design, see `status_test.go:545`).
   3. The entry is cleared only when:
      - the owner's translated content changes,
      - the owner is deleted or evicted,
      - the GatewayProxy config is removed, or
      - the controller restarts or leadership changes.
   
      For a transient failure the CR is already correct, so none of these is expected to happen. The route stays down while the GatewayProxy reports `DataPlaneAvailable=True`.
   
   When etcd becomes unavailable partway through a large sync, every event after that point fails. The corresponding services are all excluded, and once etcd recovers they are all deleted from the data plane with no self-healing. Before this PR, the whole sync failed and was retried with backoff until it converged.
   
   **Repro (unit level)**
   
   ```go
   func TestTransient5xxExcludesStickily(t *testing.T) {
   	d, route := newDroppingProvider(t)
   	rev := map[string]uint64{"GatewayProxy/ns/gp": d.store.Revision()}
   	results := map[string]types.ADCExecutionErrors{"GatewayProxy/ns/gp": failedEvent(
   		adctypes.StatusEvent{ResourceType: adctypes.TypeRoute, ResourceID: "r1", ParentID: "svc1", Type: "update"},
   		"API error (status 503): etcd request timed out")}
   	require.True(t, d.updateStatusFromSyncResults(context.Background(), results, rev))
   
   	for i := 0; i < 10; i++ { // data plane recovered, every round succeeds
   		d.updateStatusFromSyncResults(context.Background(), map[string]types.ADCExecutionErrors{"GatewayProxy/ns/gp": {}}, rev)
   	}
   	require.Contains(t, d.skipped.Excluded("GatewayProxy/ns/gp"), wireKey{adctypes.TypeService, "svc1"}) // passes
   	require.NotEmpty(t, d.resourceFailures[route])                                                       // passes
   }
   ```
   
   **Possible directions**
   
   - Have ADC mark each `failed[]` entry as a content rejection (non-retriable 4xx, version gate) or a retriable failure (5xx, transport), and exclude only content rejections. This needs an ADC change first.
   - Or give exclusions a bounded lifetime on the AIC side, for example re-including them with backoff, so a false positive heals without a content change or a restart.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4080724230


##########
internal/adc/cache/store.go:
##########
@@ -118,11 +178,18 @@ func routeOwner(targetCache Cache, id string) (types.NamespacedNameKind, bool) {
 
 // setGlobalRules is Insert and SetGlobalRules' shared implementation. Callers must
 // already hold s.Lock.
-func (s *Store) setGlobalRules(targetCache Cache, owner types.NamespacedNameKind, plugins adctypes.GlobalRule) error {
+func (s *Store) setGlobalRules(name string, targetCache Cache, owner types.NamespacedNameKind, plugins adctypes.GlobalRule) error {

Review Comment:
   `recordChange` is deferred during mutations, so the store revision/lastChange can be advanced even when the operation returns an error partway through (e.g., a delete/create fails in `setGlobalRules`, or a failure occurs after `changed` is set in `Insert`). This can make `ChangedSince`/`OwnerChangedSince` report changes that never successfully happened, which can incorrectly clear/avoid exclusions and invalidate the “stale sync result” protection. Prefer recording changes only after the mutation completes successfully (e.g., compute a `shouldRecordChange` flag, perform all cache operations, and call `recordChange` only right before returning `nil`). Apply the same pattern in `Delete`, `setPluginMetadata`, and any other paths using deferred change recording.



##########
internal/adc/cache/store.go:
##########
@@ -118,11 +178,18 @@ func routeOwner(targetCache Cache, id string) (types.NamespacedNameKind, bool) {
 
 // setGlobalRules is Insert and SetGlobalRules' shared implementation. Callers must
 // already hold s.Lock.
-func (s *Store) setGlobalRules(targetCache Cache, owner types.NamespacedNameKind, plugins adctypes.GlobalRule) error {
+func (s *Store) setGlobalRules(name string, targetCache Cache, owner types.NamespacedNameKind, plugins adctypes.GlobalRule) error {
 	rows, err := targetCache.ListGlobalRules(&OwnerSelector{Owner: owner})
 	if err != nil {
 		return err
 	}
+	existing := make(adctypes.Plugins, len(rows))
+	for _, row := range rows {
+		existing[row.ID] = row.Config
+	}
+	if pluginsContent(existing) != pluginsContent(adctypes.Plugins(plugins)) {
+		defer s.recordChange(name, owner)
+	}

Review Comment:
   `recordChange` is deferred during mutations, so the store revision/lastChange can be advanced even when the operation returns an error partway through (e.g., a delete/create fails in `setGlobalRules`, or a failure occurs after `changed` is set in `Insert`). This can make `ChangedSince`/`OwnerChangedSince` report changes that never successfully happened, which can incorrectly clear/avoid exclusions and invalidate the “stale sync result” protection. Prefer recording changes only after the mutation completes successfully (e.g., compute a `shouldRecordChange` flag, perform all cache operations, and call `recordChange` only right before returning `nil`). Apply the same pattern in `Delete`, `setPluginMetadata`, and any other paths using deferred change recording.



##########
internal/provider/apisix/status.go:
##########
@@ -137,13 +173,50 @@ func (d *apisixProvider) classifySyncResult(
 	return gatewayProxyMsgs, failedEndpoints
 }
 
+// dropUnit maps a rejected resource to what has to be dropped for the rest to apply. A
+// service is dropped by itself. A route, stream route or named upstream is dropped along
+// with the service it lives in: the service is what one Kubernetes rule translates to,
+// and a named upstream is referenced by the service's traffic-split by id, so dropping
+// it alone would leave that reference dangling. Nested resources are found through their
+// parent, which also tells them apart when several services embed upstreams of the same
+// id.
+func (d *apisixProvider) dropUnit(configName string, ev adctypes.StatusEvent) (wireKey, exclusion, bool) {
+	serviceID := ev.ResourceID
+	switch ev.ResourceType {
+	case adctypes.TypeService:
+	case adctypes.TypeRoute, adctypes.TypeStreamRoute, adctypes.TypeUpstream:
+		serviceID = ev.ParentID
+	default:
+		return wireKey{}, exclusion{}, false
+	}
+	service, ok := d.store.Lookup(configName, adctypes.TypeService, serviceID)
+	if !ok {
+		return wireKey{}, exclusion{}, false
+	}
+	owner, name := service.Owner, service.Name
+	// A route's own owner can differ from its service's: a traffic-split service can
+	// combine rules several ApisixRoutes each contributed. Attribute to the route
+	// itself when the store can tell them apart, so fixing the actual bad route clears
+	// the exclusion instead of leaving it stuck on whichever owner the shared service
+	// happens to carry.
+	if ev.ResourceType == adctypes.TypeRoute {
+		if route, ok := d.store.Lookup(configName, adctypes.TypeRoute, ev.ResourceID); ok && route.Owner != (types.NamespacedNameKind{}) {
+			owner = route.Owner
+		}
+	}
+	return wireKey{adctypes.TypeService, service.ID}, exclusion{owner: owner, name: name}, true
+}
+
 // applyResourceFailures writes this round's newly (or still) failing resources, and
 // clears the recorded error from any resource that was failing last round but isn't in
 // newFailures now. See updateStatusFromSyncResults for why resources use this delta
 // instead of GatewayProxy's full recompute.
-func (d *apisixProvider) applyResourceFailures(newFailures map[types.NamespacedNameKind][]string) {
+func (d *apisixProvider) applyResourceFailures(ctx context.Context, newFailures map[types.NamespacedNameKind][]string) {
 	for resourceKey, msgs := range newFailures {
 		d.updateStatus(resourceKey, failureCondition(resourceKey, strings.Join(msgs, "; ")))
+		if resourceKey.Kind == types.KindIngress {
+			d.recordIngressFailureEvent(ctx, resourceKey, msgs)
+		}
 	}

Review Comment:
   This emits a Warning event for every failing Ingress on every sync round, and performs a live GET each time. In steady-state failures (or frequent syncs) this can generate high write volume to the Events API and unnecessary apiserver reads. Consider emitting the event only when the failure message changes or when the Ingress transitions into failing (compare against `d.resourceFailures`), and/or add a simple rate limit (e.g., per Ingress key + time window) to reduce event spam while still providing visibility.



##########
internal/provider/apisix/provider.go:
##########
@@ -287,6 +292,11 @@ func (d *apisixProvider) Delete(ctx context.Context, obj client.Object) error {
 // applyResourceState upserts a resource's config associations and its contribution to each
 // target config's cached resource snapshot, the AIC-side bookkeeping the adc client
 // package no longer holds itself.
+// applyResourceState upserts a resource's config associations and its contribution to each
+// target config's cached resource snapshot. Whatever the skip table excluded for a
+// resource whose content this changed gets another try; rewriting identical content, which
+// reconciles triggered by unrelated events do all the time, must not retry a known-bad
+// resource.

Review Comment:
   The `applyResourceState` doc comment is duplicated/repeated (two consecutive “applyResourceState upserts…” paragraphs). Please collapse into a single coherent comment to avoid confusion and keep documentation consistent.



##########
internal/provider/apisix/status.go:
##########
@@ -153,6 +226,21 @@ func (d *apisixProvider) applyResourceFailures(newFailures map[types.NamespacedN
 	d.resourceFailures = newFailures
 }
 
+// recordIngressFailureEvent fires a Warning event for an Ingress whose status has no
+// conditions to carry a failure. It is fired every round the Ingress stays failing, since
+// events expire.
+func (d *apisixProvider) recordIngressFailureEvent(ctx context.Context, nnk types.NamespacedNameKind, msgs []string) {
+	if d.EventRecorder == nil || d.K8sClient == nil {
+		return
+	}
+	ingress := &networkingv1.Ingress{}
+	if err := d.K8sClient.Get(ctx, nnk.NamespacedName(), ingress); err != nil {
+		d.log.Error(err, "failed to get Ingress to record a failure event", "name", nnk.Name, "namespace", nnk.Namespace)
+		return
+	}
+	d.EventRecorder.Event(ingress, corev1.EventTypeWarning, string(apiv2.ConditionReasonSyncFailed), strings.Join(msgs, "; "))
+}

Review Comment:
   This emits a Warning event for every failing Ingress on every sync round, and performs a live GET each time. In steady-state failures (or frequent syncs) this can generate high write volume to the Events API and unnecessary apiserver reads. Consider emitting the event only when the failure message changes or when the Ingress transitions into failing (compare against `d.resourceFailures`), and/or add a simple rate limit (e.g., per Ingress key + time window) to reduce event spam while still providing visibility.



##########
internal/adc/cache/store.go:
##########
@@ -86,6 +100,52 @@ func gatewayProxyOf(name string) (types.NamespacedNameKind, bool) {
 	return gatewayProxy, true
 }
 
+func (s *Store) recordChange(name string, owner types.NamespacedNameKind) {
+	s.revision++
+	if s.lastChange[name] == nil {
+		s.lastChange[name] = make(map[types.NamespacedNameKind]uint64)
+	}
+	s.lastChange[name][owner] = s.revision
+}
+
+// contentOf renders items, ordered by id, the way they reach ADC, so two writes can be
+// compared by what they would push.
+func contentOf[T any](items []T, id func(T) string) string {
+	if len(items) == 0 {
+		return ""
+	}
+	sorted := slices.Clone(items)
+	slices.SortFunc(sorted, func(a, b T) int { return strings.Compare(id(a), id(b)) })
+	return canonicalJSON(sorted)
+}
+
+// canonicalJSON renders v as JSON with every object's keys sorted. A stored object's
+// plugin configs have been through a JSON round trip while a freshly translated one may
+// still hold typed structs, whose fields marshal in declaration order rather than sorted.
+func canonicalJSON(v any) string {
+	b, err := json.Marshal(v)
+	if err != nil {
+		return ""
+	}
+	var generic any
+	if err := json.Unmarshal(b, &generic); err != nil {
+		return string(b)
+	}
+	b, _ = json.Marshal(generic)
+	return string(b)
+}

Review Comment:
   `canonicalJSON` silently returns `""` on `json.Marshal` error and ignores the error on the second `json.Marshal`. For change detection, this can mask non-JSON-marshalable plugin configs and produce false “no change” results (or inconsistent behavior), which directly affects retry/exclusion logic. Consider surfacing errors to the caller (best), or at least logging and returning a deterministic fallback that forces “changed” semantics when marshaling fails; also handle/propagate the second marshal error rather than discarding it.



##########
internal/provider/apisix/status.go:
##########
@@ -153,6 +226,21 @@ func (d *apisixProvider) applyResourceFailures(newFailures map[types.NamespacedN
 	d.resourceFailures = newFailures
 }
 
+// recordIngressFailureEvent fires a Warning event for an Ingress whose status has no
+// conditions to carry a failure. It is fired every round the Ingress stays failing, since
+// events expire.
+func (d *apisixProvider) recordIngressFailureEvent(ctx context.Context, nnk types.NamespacedNameKind, msgs []string) {

Review Comment:
   This emits a Warning event for every failing Ingress on every sync round, and performs a live GET each time. In steady-state failures (or frequent syncs) this can generate high write volume to the Events API and unnecessary apiserver reads. Consider emitting the event only when the failure message changes or when the Ingress transitions into failing (compare against `d.resourceFailures`), and/or add a simple rate limit (e.g., per Ingress key + time window) to reduce event spam while still providing visibility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4084063472


##########
internal/adc/cache/store.go:
##########
@@ -460,3 +566,29 @@ func (s *Store) OwnedEntities(name string, owner types.NamespacedNameKind) []Ent
 	}
 	return entities
 }
+
+// Revision returns the store's current revision.
+func (s *Store) Revision() uint64 {
+	s.Lock()
+	defer s.Unlock()
+	return s.revision
+}
+
+// ChangedSince reports whether owner's content in the cacheKey name, or the cacheKey as a
+// whole, changed after revision: a sync result built at revision then no longer describes
+// what the store holds for owner.
+func (s *Store) ChangedSince(name string, owner types.NamespacedNameKind, revision uint64) bool {
+	s.Lock()
+	defer s.Unlock()
+	return s.lastChange[name][owner] > revision || s.resetAt[name] > revision
+}
+
+// OwnerChangedSince reports whether owner's content changed after revision in any
+// cacheKey.
+func (s *Store) OwnerChangedSince(owner types.NamespacedNameKind, revision uint64) bool {
+	s.Lock()
+	defer s.Unlock()
+	return slices.ContainsFunc(slices.Collect(maps.Values(s.lastChange)), func(byOwner map[types.NamespacedNameKind]uint64) bool {

Review Comment:
   This isn't actually a bug: we're using the standard library's `maps` package (Go 1.26, go.mod), where `maps.Values` returns `iter.Seq[V]`, an iterator, not a slice. `slices.Collect` is required to turn that into a []V before slices.ContainsFunc can use it. This builds and is covered by tests; no change needed.



##########
internal/adc/cache/store.go:
##########
@@ -86,6 +100,61 @@ func gatewayProxyOf(name string) (types.NamespacedNameKind, bool) {
 	return gatewayProxy, true
 }
 
+func (s *Store) recordChange(name string, owner types.NamespacedNameKind) {
+	s.revision++
+	if s.lastChange[name] == nil {
+		s.lastChange[name] = make(map[types.NamespacedNameKind]uint64)
+	}
+	s.lastChange[name][owner] = s.revision
+}
+
+// contentOf renders items, ordered by id, the way they reach ADC, so two writes can be
+// compared by what they would push.
+func contentOf[T any](log logr.Logger, items []T, id func(T) string) string {
+	if len(items) == 0 {
+		return ""
+	}
+	sorted := slices.Clone(items)
+	slices.SortFunc(sorted, func(a, b T) int { return strings.Compare(id(a), id(b)) })
+	return canonicalJSON(log, sorted)
+}
+
+// canonicalJSON renders v as JSON with every object's keys sorted. A stored object's
+// plugin configs have been through a JSON round trip while a freshly translated one may
+// still hold typed structs, whose fields marshal in declaration order rather than sorted.
+//
+// A marshal error here is logged rather than propagated: v is always a type this package
+// itself built from already-parsed JSON, so this is defensive rather than expected, and
+// callers use the result only for change detection, not for anything that reaches ADC.
+func canonicalJSON(log logr.Logger, v any) string {
+	b, err := json.Marshal(v)
+	if err != nil {
+		log.Error(err, "failed to marshal value for change detection")
+		return ""
+	}
+	var generic any
+	if err := json.Unmarshal(b, &generic); err != nil {
+		return string(b)
+	}
+	if sorted, err := json.Marshal(generic); err != nil {
+		log.Error(err, "failed to re-marshal value for change detection")
+	} else {
+		b = sorted
+	}
+	return string(b)
+}

Review Comment:
   Agreed this has real cost for large resource sets, but fixing it means changing what change detection compares against (a cached per-owner fingerprint computed once at write time, rather than a fresh marshal/unmarshal/marshal on every comparison), which is a bigger design change than fits this PR. Tracking this alongside the lastChange growth follow-up rather than fixing it here.



##########
internal/provider/apisix/status.go:
##########
@@ -58,23 +61,32 @@ const (
 // get a True the first time, and what keeps a restart from leaving a stale False stuck
 // forever: the write only ever depends on this round's actual outcome.
 //
-// Resource status can't afford the same full recompute: a config's resource set can be
-// large, and rewriting every one of them every round even when nothing changed would be
-// wasteful. So resources keep a small persisted delta in d.resourceFailures instead:
-// newly (or still) failing resources are written SyncFailed, and any resource that was
-// failing last round but isn't failing this one gets its error explicitly cleared with
-// an Accepted write.
-func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors) {
+// Resource status follows the whole skip table instead, since an excluded resource stays
+// dropped until its owner is written again. A config's resource set can be large, so
+// resources keep a small persisted delta in d.resourceFailures rather than being rewritten
+// every round: resources that are dropped or failing are written SyncFailed, and any
+// resource that was last round but isn't now gets its error explicitly cleared with an
+// Accepted write.
+//
+// It reports whether anything was newly excluded, which is what makes the next push
+// different from the one that just failed: see sync().
+func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors, revisions map[string]uint64) bool {
 	resourceFailures := map[types.NamespacedNameKind][]string{}
+	newlyExcluded := 0
 
 	for configName, execErrs := range results {
+		dropped := map[wireKey]exclusion{}
+		gatewayProxyMsgs, failedEndpoints := d.classifySyncResult(configName, execErrs, dropped, resourceFailures)
+		builtRevision := revisions[configName]
+		newlyExcluded += d.skipped.MarkFailing(configName, dropped, func(owner types.NamespacedNameKind) bool {
+			return d.store.ChangedSince(configName, owner, builtRevision)
+		})

Review Comment:
   Good catch, fixed in 7d7f424d: the lookup is now checked with the ok form, and skips recording that config's newly rejected resources with a logged error instead of silently treating a missing revision as 0. As you noted this can't happen with sync() as written today (revisions and results are always populated together in the same build closure), but the guard is cheap and protects against a future refactor breaking that invariant.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4090309309


##########
internal/provider/apisix/status.go:
##########
@@ -121,13 +152,21 @@ func (d *apisixProvider) classifySyncResult(
 
 			anyUnattributed := false
 			for _, syncStatus := range addrErr.FailedStatuses {
+				reason := fmt.Sprintf("ServerAddr: %s, Error: %s", addrErr.ServerAddr, syncStatus.Reason)
+				if key, ex, ok := d.dropUnit(configName, syncStatus.Event); ok {
+					if previous, ok := dropped[key]; ok {
+						reason = previous.reason + "; " + reason
+					}
+					ex.reason = reason
+					dropped[key] = ex
+					continue

Review Comment:
   Not fixing this: GatewayProxy's condition is recomputed fresh every round directly from that round's own result, not carried forward, so this only shows up for the one round before the rejected resource is excluded. Excluding it triggers an immediate retry (syncNotify), and the next round no longer has this addrErr's EndpointStatuses at all since the resource is now excluded from the push, so the condition clears itself right away. At worst this leaves one stale event behind; it doesn't leave the GatewayProxy stuck reporting unavailable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4080935628


##########
internal/provider/apisix/status.go:
##########
@@ -58,23 +61,29 @@ const (
 // get a True the first time, and what keeps a restart from leaving a stale False stuck
 // forever: the write only ever depends on this round's actual outcome.
 //
-// Resource status can't afford the same full recompute: a config's resource set can be
-// large, and rewriting every one of them every round even when nothing changed would be
-// wasteful. So resources keep a small persisted delta in d.resourceFailures instead:
-// newly (or still) failing resources are written SyncFailed, and any resource that was
-// failing last round but isn't failing this one gets its error explicitly cleared with
-// an Accepted write.
-func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors) {
+// Resource status follows the whole skip table instead, since an excluded resource stays
+// dropped until its owner is written again. A config's resource set can be large, so
+// resources keep a small persisted delta in d.resourceFailures rather than being rewritten
+// every round: resources that are dropped or failing are written SyncFailed, and any
+// resource that was last round but isn't now gets its error explicitly cleared with an
+// Accepted write.
+//
+// It reports whether anything was newly excluded, which is what makes the next push
+// different from the one that just failed: see sync().
+func (d *apisixProvider) updateStatusFromSyncResults(ctx context.Context, results map[string]types.ADCExecutionErrors, revisions map[string]uint64) bool {
 	resourceFailures := map[types.NamespacedNameKind][]string{}
+	newlyExcluded := 0
 
 	for configName, execErrs := range results {
+		dropped := map[wireKey]exclusion{}
+		gatewayProxyMsgs, failedEndpoints := d.classifySyncResult(configName, execErrs, revisions[configName], dropped, resourceFailures)
+		newlyExcluded += d.skipped.MarkFailing(configName, dropped)

Review Comment:
   Fixed in 63068bfa: MarkFailing now takes a stale callback and checks it inside the same lock that performs the write, instead of checking ChangedSince beforehand in a separate critical section.



##########
internal/provider/apisix/status.go:
##########
@@ -137,13 +174,39 @@ func (d *apisixProvider) classifySyncResult(
 	return gatewayProxyMsgs, failedEndpoints
 }
 
+// dropUnit maps a rejected resource to what has to be dropped for the rest to apply. A
+// service is dropped by itself. A route, stream route or named upstream is dropped along
+// with the service it lives in: the service is what one Kubernetes rule translates to,
+// and a named upstream is referenced by the service's traffic-split by id, so dropping
+// it alone would leave that reference dangling. Nested resources are found through their
+// parent, which also tells them apart when several services embed upstreams of the same
+// id.
+func (d *apisixProvider) dropUnit(configName string, ev adctypes.StatusEvent) (wireKey, exclusion, bool) {
+	serviceID := ev.ResourceID
+	switch ev.ResourceType {
+	case adctypes.TypeService:
+	case adctypes.TypeRoute, adctypes.TypeStreamRoute, adctypes.TypeUpstream:
+		serviceID = ev.ParentID
+	default:
+		return wireKey{}, exclusion{}, false
+	}
+	service, ok := d.store.Lookup(configName, adctypes.TypeService, serviceID)
+	if !ok {
+		return wireKey{}, exclusion{}, false
+	}
+	return wireKey{adctypes.TypeService, service.ID}, exclusion{owner: service.Owner, name: service.Name}, true

Review Comment:
   Fixed in 63068bfa: dropUnit now looks up the route's own owner via store.Lookup(configName, TypeRoute, ev.ResourceID) and uses it when the route has labels of its own, falling back to the service's owner otherwise. TypeStreamRoute/TypeUpstream still fall back to the service's owner since there is no per-nested-owner lookup for those yet; tracked separately.



##########
test/e2e/crds/v2/isolation.go:
##########
@@ -0,0 +1,255 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package v2
+
+import (
+	"fmt"
+	"net/http"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	gomegatypes "github.com/onsi/gomega/types"
+
+	"github.com/apache/apisix-ingress-controller/test/e2e/scaffold"
+)
+
+// Every case applies a valid and a rejected resource together, so the first sync carries
+// both: the sync converges when the rejected one is excluded and the valid one is served.
+// Fixing the rejected one then has to bring it in too.
+//
+// What makes a resource rejected is always something the data plane checks for every
+// backend: a value outside the schema's range, or a known plugin configured in a way its
+// own check_schema refuses. An unknown plugin name is not used, since apisix-standalone
+// accepts those.
+var _ = Describe("Test bad resource isolation", Label("apisix.apache.org", "v2", "isolation"), func() {
+	s := scaffold.NewDefaultScaffold()
+
+	// rejectedPlugin is a plugin every backend loads, configured with a count the plugin
+	// itself refuses (it has to be greater than 0).
+	const rejectedPlugin = `
+    plugins:
+    - name: limit-count
+      enable: true
+      config:
+        count: 0
+        time_window: 60
+        rejected_code: 503
+        key: remote_addr
+`
+	// A consistent-hash load balancer needs a key to hash on. The data plane checks that
+	// in code rather than in its schema, so the configuration gets past ADC's schema
+	// check and is rejected only by the data plane itself.
+	const rejectedHashKey = ""
+	const acceptedHashKey = "    key: remote_addr"
+	const acceptedPlugin = `
+    plugins:
+    - name: limit-count
+      enable: true
+      config:
+        count: 100
+        time_window: 60
+        rejected_code: 503
+        key: remote_addr
+`
+	// routes is one ApisixRoute serving valid.example.com and one serving
+	// rejected.example.com, whose plugin configuration is filled in per case.
+	const routes = `
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+  name: valid
+  namespace: %s
+spec:
+  ingressClassName: %s
+  http:
+  - name: rule0
+    match:
+      hosts:
+      - valid.example.com
+      paths:
+      - /*
+    backends:
+    - serviceName: httpbin-service-e2e-test
+      servicePort: 80
+---
+apiVersion: apisix.apache.org/v2
+kind: ApisixRoute
+metadata:
+  name: rejected
+  namespace: %s
+spec:
+  ingressClassName: %s
+  http:
+  - name: rule0
+    match:
+      hosts:
+      - rejected.example.com
+      paths:
+      - /*
+    backends:
+    - serviceName: httpbin-service-e2e-test
+      servicePort: 80
+%s
+`
+
+	expectServed := func(host string) {
+		s.RequestAssert(&scaffold.RequestAssert{
+			Method: "GET",
+			Path:   "/get",
+			Host:   host,
+			Check:  scaffold.WithExpectedStatus(http.StatusOK),
+		})
+	}
+	expectStatus := func(resource, name string, matchers ...gomegatypes.GomegaMatcher) {
+		s.RetryAssertion(func() string {
+			output, _ := s.GetOutputFromString(resource, name, "-o", "yaml", "-n", s.Namespace())
+			return output
+		}).Should(And(matchers...))
+	}
+	apply := func(yaml string) {
+		Expect(s.CreateResourceFromString(yaml)).NotTo(HaveOccurred(), "applying resources")
+	}
+
+	BeforeEach(func() {
+		By("create GatewayProxy")
+		Expect(s.CreateResourceFromString(s.GetGatewayProxySpec())).NotTo(HaveOccurred(), "creating GatewayProxy")
+
+		By("create IngressClass")
+		Expect(s.CreateResourceFromStringWithNamespace(s.GetIngressClassYaml(), "")).NotTo(HaveOccurred(), "creating IngressClass")
+	})
+
+	It("isolates a rejected route", func() {
+		By("apply a valid and a rejected ApisixRoute")
+		apply(fmt.Sprintf(routes, s.Namespace(), s.Namespace(), s.Namespace(), s.Namespace(), rejectedPlugin))
+
+		By("the valid ApisixRoute is served and the rejected one reports why it is not")
+		expectServed("valid.example.com")
+		// The rejected route is the only one this ApisixRoute has, so none of it is served.
+		expectStatus("ar", "rejected",
+			ContainSubstring(`status: "False"`),
+			ContainSubstring(`reason: SyncFailed`),
+			ContainSubstring(`limit-count`),
+		)
+
+		By("fix the rejected ApisixRoute")
+		apply(fmt.Sprintf(routes, s.Namespace(), s.Namespace(), s.Namespace(), s.Namespace(), acceptedPlugin))
+
+		By("both ApisixRoutes are served")
+		expectServed("valid.example.com")
+		expectServed("rejected.example.com")
+		expectStatus("ar", "rejected", ContainSubstring(`reason: Accepted`))
+	})
+
+	It("isolates a route whose upstream configuration is rejected", func() {
+		const upstream = `
+apiVersion: apisix.apache.org/v2
+kind: ApisixUpstream
+metadata:
+  name: httpbin-service-e2e-test
+  namespace: %s
+spec:
+  ingressClassName: %s
+  loadbalancer:
+    type: chash
+    hashOn: vars
+%s
+`
+		By("apply a valid and a rejected ApisixRoute, the rejected one using a rejected upstream configuration")
+		apply(fmt.Sprintf(routes, s.Namespace(), s.Namespace(), s.Namespace(), s.Namespace(), ""))
+		apply(fmt.Sprintf(upstream, s.Namespace(), s.Namespace(), rejectedHashKey))

Review Comment:
   Fixed in 63068bfa: the rejected route now gets its own backend (an alias Service, same pattern as streamroute.go's Bad resource isolation test) instead of sharing httpbin-service-e2e-test with the valid route, and the test asserts the valid route stays served right after applying the bad upstream config, before the fix.



##########
test/e2e/ingress/ingress.go:
##########
@@ -1600,4 +1600,115 @@ spec:
 		})
 
 	})
+
+	Context("Bad resource isolation", func() {
+		var gatewayProxy = `
+apiVersion: apisix.apache.org/v1alpha1
+kind: GatewayProxy
+metadata:
+  name: apisix-proxy-config
+  namespace: %s
+spec:
+  provider:
+    type: ControlPlane
+    controlPlane:
+      endpoints:
+      - %s
+      auth:
+        type: AdminKey
+        adminKey:
+          value: "%s"
+`
+		var ingressClass = `
+apiVersion: networking.k8s.io/v1
+kind: IngressClass
+metadata:
+  name: %s
+spec:
+  controller: "%s"
+  parameters:
+    apiGroup: "apisix.apache.org"
+    kind: "GatewayProxy"
+    name: "apisix-proxy-config"
+    namespace: "%s"
+    scope: "Namespace"
+`
+		// The Ingress picks up its plugins from an ApisixPluginConfig named in an
+		// annotation; limit-count refuses a count that is not greater than 0.
+		var rejectedPluginConfig = `
+apiVersion: apisix.apache.org/v2
+kind: ApisixPluginConfig
+metadata:
+  name: rejected-plugins
+spec:
+  ingressClassName: %s
+  plugins:
+  - name: limit-count
+    enable: true
+    config:
+      count: 0
+      time_window: 60
+      rejected_code: 503
+      key: remote_addr
+`
+		var ingressTemplate = `
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: rejected
+%s
+spec:
+  ingressClassName: %s
+  rules:
+  - host: rejected-ingress.example.com
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: httpbin-service-e2e-test
+            port:
+              number: 80
+`
+		const withRejectedPlugins = `  annotations:
+    k8s.apisix.apache.org/plugin-config-name: rejected-plugins`
+
+		It("a rejected Ingress is reported as an event", func() {
+			By("create GatewayProxy")
+			err := s.CreateResourceFromStringWithNamespace(fmt.Sprintf(gatewayProxy, s.Namespace(), s.Deployer.GetAdminEndpoint(), s.AdminKey()), s.Namespace())
+			Expect(err).NotTo(HaveOccurred(), "creating GatewayProxy")
+
+			By("create IngressClass")
+			err = s.CreateResourceFromStringWithNamespace(fmt.Sprintf(ingressClass, s.Namespace(), s.GetControllerName(), s.Namespace()), "")
+			Expect(err).NotTo(HaveOccurred(), "creating IngressClass")
+
+			By("create an Ingress whose plugins the data plane rejects")
+			err = s.CreateResourceFromString(fmt.Sprintf(rejectedPluginConfig, s.Namespace()))
+			Expect(err).NotTo(HaveOccurred(), "creating ApisixPluginConfig")
+			err = s.CreateResourceFromString(fmt.Sprintf(ingressTemplate, withRejectedPlugins, s.Namespace()))
+			Expect(err).NotTo(HaveOccurred(), "creating Ingress")

Review Comment:
   Fixed in 63068bfa: added a sibling valid Ingress created alongside the rejected one, and the test asserts it stays served both while the rejected Ingress is failing and after it is fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4119135842


##########
internal/provider/apisix/status.go:
##########
@@ -137,13 +176,50 @@ func (d *apisixProvider) classifySyncResult(
 	return gatewayProxyMsgs, failedEndpoints
 }
 
+// dropUnit maps a rejected resource to what has to be dropped for the rest to apply. A
+// service is dropped by itself. A route, stream route or named upstream is dropped along
+// with the service it lives in: the service is what one Kubernetes rule translates to,
+// and a named upstream is referenced by the service's traffic-split by id, so dropping
+// it alone would leave that reference dangling. Nested resources are found through their
+// parent, which also tells them apart when several services embed upstreams of the same
+// id.
+func (d *apisixProvider) dropUnit(configName string, ev adctypes.StatusEvent) (wireKey, exclusion, bool) {

Review Comment:
   `dropUnit` treats every `failed[]` entry as a content rejection, but on the `apisix` backend ADC also reports transient failures there. Right now a single 5xx or transport error can permanently exclude a valid service and delete it from the data plane.
   
   **Where transient failures come from (ADC master `63a0937`)**
   
   The ADC server runs sync with `exit_on_failure: false`. A failed event does not abort the sync. It is returned in `failed[]` with its full `event`, and the response is still a 202.
   
   ```rust
   // adc: rust/crates/adc-cli/src/server/sync.rs
   let sync_opts = BackendSyncOptions {
       concurrent: Some(opts.request_concurrent),
       exit_on_failure: Some(false),
   };
   ```
   
   ```ts
   // adc: apps/cli/src/server/sync.ts (the Node server, same behavior)
   backend.sync(diff, { exitOnFailure: false, concurrent: 4 })
   ```
   
   The apisix backend only retries 5xx and transport errors briefly, then gives up and reports the event as failed:
   
   ```rust
   // adc: rust/crates/adc-backend-core/src/retry.rs
   fn default() -> Self {
       Self { retries: 3, delay: Duration::from_millis(100) }
   }
   ```
   
   The Admin API returns `503` for any etcd write error:
   
   ```lua
   -- apisix: apisix/admin/resource.lua
   local res, err = core.etcd.set(key, conf, ttl)
   if not res then
       core.log.error("failed to put ", self.kind, "[", key, "] to etcd: ", err)
       return 503, {error_msg = err}
   end
   ```
   
   An etcd leader election usually lasts longer than ~300ms. If one overlaps a sync that has already passed dump, the retries run out and the event ends up in `failed[]`.
   
   The failed entry carries no HTTP status, so AIC cannot even separate these by status code today:
   
   ```rust
   // adc: rust/crates/adc-cli/src/server/sync.rs
   struct FailedEntry {
       server: Option<String>,
       event: Option<Value>,
       failed_at: String,
       reason: String,
   }
   ```
   
   The `apisix-standalone` backend is not affected: the Rust server fills its `failed[]` only from `/validate`.
   
   **Lifecycle once an entry is recorded**
   
   1. Every later push leaves the service out, including periodic `SyncPeriod` rounds. The ADC diff then DELETEs the previously serving service and its routes:
   
      ```go
      // internal/provider/apisix/provider.go, sync()
      revisions[config.Name] = revision
      resources = exclude(resources, d.skipped.Excluded(config.Name))
      ```
   
   2. Successful rounds never clear the entry. This PR's own test asserts it:
   
      ```go
      // internal/provider/apisix/status_test.go
      ok := map[string]types.ADCExecutionErrors{"GatewayProxy/ns/gp": {}}
      require.False(t, d.updateStatusFromSyncResults(context.Background(), ok, revisions), "nothing new to exclude")
      require.Len(t, d.resourceFailures[route], 1, "still dropped: ADC no longer mentioning it says nothing about a fix")
      ```
   
   3. The entry is cleared only when:
      - the owner's translated content actually changes,
      - the owner is deleted or evicted,
      - the GatewayProxy config is removed, or
      - the controller restarts or leadership changes.
   
      The content check is the only path a healthy resource could take:
   
      ```go
      // internal/provider/apisix/provider.go, applyResourceState()
      before := d.store.Revision()
      defer func() {
          if d.store.OwnerChangedSince(rk, before) {
              d.skipped.ClearOwner(rk)
          }
      }()
      ```
   
      For a transient failure the CR is already correct, so none of these is expected to happen. The route stays down while the GatewayProxy reports `DataPlaneAvailable=True`.
   
   When etcd becomes unavailable partway through a large sync, every event after that point fails. The corresponding services are all excluded, and once etcd recovers they are all deleted from the data plane with no self-healing. Before this PR, the whole sync failed and was retried with backoff until it converged.
   
   **Repro (unit level)**
   
   ```go
   func TestTransient5xxExcludesStickily(t *testing.T) {
   	d, route := newDroppingProvider(t)
   	rev := map[string]uint64{"GatewayProxy/ns/gp": d.store.Revision()}
   	results := map[string]types.ADCExecutionErrors{"GatewayProxy/ns/gp": failedEvent(
   		adctypes.StatusEvent{ResourceType: adctypes.TypeRoute, ResourceID: "r1", ParentID: "svc1", Type: "update"},
   		"API error (status 503): etcd request timed out")}
   	require.True(t, d.updateStatusFromSyncResults(context.Background(), results, rev))
   
   	for i := 0; i < 10; i++ { // data plane recovered, every round succeeds
   		d.updateStatusFromSyncResults(context.Background(), map[string]types.ADCExecutionErrors{"GatewayProxy/ns/gp": {}}, rev)
   	}
   	require.Contains(t, d.skipped.Excluded("GatewayProxy/ns/gp"), wireKey{adctypes.TypeService, "svc1"}) // passes
   	require.NotEmpty(t, d.resourceFailures[route])                                                       // passes
   }
   ```
   
   **Possible directions**
   
   - Have ADC mark each `failed[]` entry as a content rejection (non-retriable 4xx, version gate) or a retriable failure (5xx, transport), and exclude only content rejections. This needs an ADC change first.
   - Or give exclusions a bounded lifetime on the AIC side, for example re-including them with backoff, so a false positive heals without a content change or a restart.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(apisix): isolate rejected routes instead of failing the whole sync [apisix-ingress-controller]

Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on code in PR #2890:
URL: https://github.com/apache/apisix-ingress-controller/pull/2890#discussion_r4119862127


##########
internal/provider/apisix/status.go:
##########
@@ -137,13 +176,50 @@ func (d *apisixProvider) classifySyncResult(
 	return gatewayProxyMsgs, failedEndpoints
 }
 
+// dropUnit maps a rejected resource to what has to be dropped for the rest to apply. A
+// service is dropped by itself. A route, stream route or named upstream is dropped along
+// with the service it lives in: the service is what one Kubernetes rule translates to,
+// and a named upstream is referenced by the service's traffic-split by id, so dropping
+// it alone would leave that reference dangling. Nested resources are found through their
+// parent, which also tells them apart when several services embed upstreams of the same
+// id.
+func (d *apisixProvider) dropUnit(configName string, ev adctypes.StatusEvent) (wireKey, exclusion, bool) {

Review Comment:
   It seems like we can just disable bad resource isolation directly in APISIX; it's meaningless for this scenario, and failures are tolerable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org