Re: [I] bug: ADC execution error [apisix-ingress-controller]

["arjunbhut (via GitHub)" <gi...@apache.org>]

arjunbhut commented on issue #2749:
URL: https://github.com/apache/apisix-ingress-controller/issues/2749#issuecomment-4421267853

   Hi @Baoyuantop — I'd like to add a second-trigger to this issue. We're hitting the same `*_conf_version must be greater than or equal to (X)` rejections, but our setup never touches the Admin API, never mixes management, and the issue still recurs every sync cycle. I believe there's a structural multi-replica drift in api-driven standalone that the current diagnosis doesn't cover.
   
   ## Setup
   
   | Component | Version |
   |---|---|
   | Helm chart | `apisix/apisix:2.14.0` |
   | APISIX | `apache/apisix:3.16.0-ubuntu` |
   | Ingress controller | `apache/apisix-ingress-controller:2.0.1` |
   | ADC sidecar | `ghcr.io/api7/adc:0.25.0` (overridden from chart default `0.23.1`) |
   | Replicas | **10** |
   | Provider | `apisix-standalone` (per official install guide) |
   
   Deployment values follow the docs verbatim:
   
   ```yaml
   apisix.deployment.role: traditional
   apisix.deployment.role_traditional.config_provider: yaml
   etcd.enabled: false
   ingress-controller.config.provider.type: apisix-standalone
   ```
   
   ## What we *don't* do
   
   - No `curl` against the Admin API (port 9180).
   - No `kubectl exec` to write config files.
   - No APISIX Dashboard.
   - Config managed exclusively via `Gateway`, `HTTPRoute`, and `GatewayProxy` CRDs (Gateway API + `apisix.apache.org/v1alpha1`).
   - Fresh cluster: `helm uninstall` + reinstall, fresh CRD apply, fresh secrets — same drift recurs within minutes.
   
   ## Symptom
   
   ADC sync cycle every ~minute. Each cycle, **a different subset of pods rejects with 400**, citing some `*_conf_version must be >= (epoch_ms)`. Sample over a 5-minute window (output from ADC 0.25.0's per-endpoint status — much appreciated, by the way):
   
   ```
   cycle 1: success=9, failed=1   pod 10.53.4.224  ssls_conf_version must be >= 1778497564298
   cycle 2: success=8, failed=2   pod 10.53.4.224  ssls_conf_version
                                   pod 10.53.4.221  upstreams_conf_version
   cycle 3: success=9, failed=1   pod 10.53.3.55   services_conf_version
   cycle 4: success=10            (one good sync)
   cycle 5: success=8, failed=2   pod 10.53.4.222, 10.53.4.220   ssls_conf_version
   ```
   
   Note that **the stuck pod rotates**. It's not stuck-forever-on-one-pod (which would match the user-triggered single-spike scenario in this issue). External effect: TLS handshake succeeds for traffic landing on the "good" pods, fails (`tlsv1 alert internal_error` because no SSL loaded) on the "stuck" pods. Hit rate fluctuates between ~30% and ~85% depending on which pods are currently behind.
   
   ## Why "don't mix management" doesn't apply here
   
   The diagnosis above suggests the user pushed a high version via Admin API, so ADC's lower number gets rejected. That's a single-trigger scenario, recoverable by restart-both-sides. We've:
   
   1. Confirmed no Admin API access from our side (verified via apisix pod access logs — only `axios/1.13.2` UA, the controller's ADC client).
   2. Done the recommended `delete apisix pod + restart ingress-controller` recovery. Works for a sync cycle or two, then drift returns.
   3. Fully torn down: `helm uninstall apisix`, delete all CRs, fresh `helm install`. Drift still recurs within minutes.
   
   What we're seeing is **organic, structural drift** between N independent in-memory version counters and ADC's local CacheKey, with no user-induced trigger.
   
   ## Hypothesis
   
   `*_conf_version` is per-pod in-memory state. ADC sends one canonical version to N pods in parallel. Any time *one* pod's accept lands a moment later, or a retry pushes a slightly higher version to a single pod, that pod is now \"ahead\" of ADC's local CacheKey *and* ahead of the other N-1 pods. On the next periodic sync, ADC's version is below the ahead-pod's required minimum → 400. ADC may not be updating its CacheKey from the maximum observed across pods, only from its local notion.
   
   In api-driven standalone with multiple replicas, this divergence appears unbounded: each cycle can produce a new \"ahead\" pod (because every cycle has its own micro-race), so the set of stuck pods rotates over time.
   
   ## Reproducible?
   
   Yes, in any cluster with `replicaCount: 10` + `apisix-standalone` provider + only-CRD management. Happy to provide:
   
   - Helm values used.
   - ADC controller logs across many cycles showing the rotation.
   - Full per-pod 204/400 distribution.
   
   ## Questions
   
   1. Is there a documented multi-replica-safe configuration for `apisix-standalone`, or is the deployment mode currently single-replica by design intent?
   2. Would it be feasible for ADC to update its local CacheKey from `max(local, last_observed_per_pod)` rather than only from its own writes? That would eliminate the drift after one full cycle.
   3. Is there a \`disable_conf_version_check\` or similar option for environments where the controller is the only writer?
   
   If the answer to (1) is \"single replica only,\" that should probably be more prominent in the [Install with Helm](https://apisix.apache.org/docs/ingress-controller/install/) guide — the canonical example does not call this out, and 10 replicas seemed like a reasonable default for an ingress.
   
   Happy to file as a separate \`bug:\` if the maintainers prefer keeping this issue scoped to the original (Admin API mixed-management) trigger.
   
   ---
   
   *Comment drafted with Claude Code assistance, reviewed and posted by @arjunbhut.*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org