You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "kworkbee (via GitHub)" <gi...@apache.org> on 2024/03/06 08:35:28 UTC

[I] bug: Upstream HealthCheck Issue - Unhealthy Upstream won't be excluded temporarily [apisix-ingress-controller]

kworkbee opened a new issue, #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176

   ### Current Behavior
   
   There is a problem with the unhealthy external service being delivered as it is without being excluded from routing targets.
   
   ### Expected Behavior
   
   Two external services (ALB configured in front of each) are configured as upstream nodes and should be temporarily excluded from routing if a 5XX error occurs through health check configuration.
   
   ### Error Logs
   
   _No response_
   
   ### Steps to Reproduce
   
   ```
   apiVersion: apisix.apache.org/v2
   kind: ApisixRoute
   metadata:
     name: route
   spec:
     http:
       - match:
           hosts:
             - service.corp.com
           methods:
             - POST
           paths:
             - /svc/*
         name: route
         plugins:
           - enable: true
             name: traffic-split
           - config:
               regex_uri:
                 - ^\/svc\/(.+)$
                 - /$1
             enable: true
             name: proxy-rewrite
         upstreams:
           - name: upstream
             weight: 100
   ```
   ```
   apiVersion: apisix.apache.org/v2
   kind: ApisixUpstream
   metadata:
     name: nodes-upstream
     namespace: apisix
   spec:
     externalNodes:
       - name: svc01.corp.com
         port: 443
         type: Domain
         weight: 50
       - name: svc02.corp.com
         port: 443
         type: Domain
         weight: 50
     healthCheck:
       active:
         healthy:
           httpCodes:
             - 404
           interval: 3s
           successes: 1
         httpPath: /
         type: https
         unhealthy:
           httpCodes:
             - 503
             - 504
           httpFailures: 1
           interval: 3s
           timeouts: 3
       passive:
         healthy:
           httpCodes:
             - 404
           successes: 1
         type: https
         unhealthy:
           httpCodes:
             - 503
             - 504
           httpFailures: 1
           timeouts: 3
     loadbalancer:
       type: roundrobin
     passHost: node
     scheme: https
   ```
   
   ### Environment
   
   - APISIX Ingress controller version (run `apisix-ingress-controller version --long`)
   - Kubernetes cluster version (run `kubectl version`)
   - OS version if running APISIX Ingress controller in a bare-metal environment (run `uname -a`)
   
   Runs on an AWS EKS Cluster (Kubernetes v1.25). Uses APISIX Helm Chart (1.11.0, App 3.8.0).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1982342081

   @Revolyssup @shreemaan-abhishek 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993673066

   > @Revolyssup ![image](https://private-user-images.githubusercontent.com/18034626/312336717-d8388e78-8328-4c4e-a365-1c247baa8ddc.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTAzMTIyNDksIm5iZiI6MTcxMDMxMTk0OSwicGF0aCI6Ii8xODAzNDYyNi8zMTIzMzY3MTctZDgzODhlNzgtODMyOC00YzRlLWEzNjUtMWMyNDdiYWE4ZGRjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzEzVDA2MzkwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTgxYmVlN2Q0ZWIyYjE0NzlmNzNmMjU3Y2Q5MzQ4NDRiMjhjMjUzNDliYThlNTg3OWEwODA3NTRhMGY2YWI5MmEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ZA2tBjOZwzRSWBQTSkYJBbrf-7GDxkUMJc8ieqZbh-Q)
   
   Okay so there is no sync issue. This upstream configuration matches exactly what you provided via ApisixUpstream. So there is no issue at ingress controller level. Are you still running into the issue where unhealthy upstream is not excluded? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1996891518

   > The following specific logs are visible in the APISIX Log. Did SSL Handshake Failure Affect?
   > 
   > ```
   > 2024/03/13 15:42:17 [error] 50#50: *33439 [lua] healthcheck.lua:1383: log(): [healthcheck] (upstream#/apisix/upstreams/32eb11c7) failed SSL handshake with 'X.X.X.X (X.X.X.X:443)', using server name (sni) 'svc01.corp.com': 19: self-signed certificate in certificate chain, context: ngx.timer, client: X.X.X.X, server: 0.0.0.0:9080
   > ```
   
   he active and passive health checks on upstreams work on HTTP codes but in case the HTTP connection is never established with the upstream, like in the case of SSL handshake failure then the upstream will not be excluded from the list of available upstreams. As the healthchecks wont work, the client will get 503.
   This is a tcp connection failure, you can use something like unhealthy.tcp_failures


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993643334

   @Revolyssup It is distributed in the form of Helm Chart using ArgoCD, a CD tool, so I think you should see that it is distributed in the same namespace.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993665202

   @kworkbee What you're seeing for `healthcheck.active.healthy.httpCodes` is the default value which is passed when you don't explicitly pass the value. It looks like at some point in time ApisixUpstream resource was deployed without passing the `healthcheck.active.healthy.httpCodes`. The upstream you showed from admin api might have been from that time. I can confirm that httpCodes passed in Upstream is succesfully translated to httpStatuses.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993669083

   @Revolyssup
   ![image](https://github.com/apache/apisix-ingress-controller/assets/18034626/d8388e78-8328-4c4e-a365-1c247baa8ddc)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993659262

   > 200,
   >             201,
   >             202,
   >             203,
   >             204,
   >             205,
   >             206,
   >             207,
   >             208,
   >             226,
   >             300,
   >             301,
   >             302,
   >             303,
   >             304,
   >             305,
   >             306,
   >             307,
   >             308
   
   Is this the only upstream? Can you send the output of 
   ```bash
    curl http://127.0.0.1:9180/apisix/admin/upstreams -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' | jq 
   ```
   After port forwarding to apisix pod. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream won't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1980565808

   In Admin API, a state different from the `.spec.healthCheck.passive.unhealth.httpCodes` defined in `ApisixUpstream` is being returned.
   
   ```
   "active": {
           "type": "http",
           "concurrency": 10,
           "http_path": "/",
           "timeout": 1,
           "healthy": {
             "http_statuses": [
               404
             ],
             "successes": 1,
             "interval": 3
           },
           "unhealthy": {
             "timeouts": 3,
             "interval": 3,
             "http_statuses": [
               503,
               504
             ],
             "http_failures": 1,
             "tcp_failures": 2
           },
           "https_verify_certificate": true
         },
         "passive": {
           "healthy": {
             "http_statuses": [
               200,
               201,
               202,
               203,
               204,
               205,
               206,
               207,
               208,
               226,
               300,
               301,
               302,
               303,
               304,
               305,
               306,
               307,
               308
             ],
             "successes": 5
           },
           "type": "http",
           "unhealthy": {
             "http_statuses": [
               503,
               504
             ],
             "timeouts": 7,
             "http_failures": 1,
             "tcp_failures": 2
           }
         }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1982341702

   It seems that health checks are made as they are without being rewritten to each host defined in `externalNodes` on the log.
   
   ```
   2024/03/07 13:27:02 [warn] 51#51: *1196791 [lua] balancer.lua:82: fetch_health_nodes(): failed to get health check target status, addr: 123.456.789.876:443, host: nil, err: target not found, client: 123.789.345.678, server: _, request: "POST /svc HTTP/1.1", host: "svc01.corp.com"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993647507

   > mespace: apisix
   
   Also then provide the original route configuration


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993598535

   @kworkbee Can you show the spec of `apisix_feature-flags-nodes-upstream`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993602454

   @Revolyssup The domains are modified and provided for security reasons.
   
   ```yaml
   apiVersion: apisix.apache.org/v2
   kind: ApisixUpstream
   metadata:
     name: feature-flags-nodes-upstream
   spec:
     loadbalancer:
       type: roundrobin
     passHost: node
     scheme: https
     externalNodes:
     - type: Domain
       name: svc01.corp.com
       port: 443
       weight: 50
     - type: Domain
       name: svc02.corp.com
       port: 443
       weight: 50
     healthCheck:
       active:
         type: https
         port: 443
         strictTLS: true
         httpPath: /
         healthy:
           successes: 1
           interval: 3s
           httpCodes:
           - 404
         unhealthy:
           httpCodes:
           - 503
           - 504
           httpFailures: 1
           interval: 3s
           timeouts: 3
       passive:
         healthy:
           successes: 1
           httpCodes:
           - 404
         unhealthy:
           httpCodes:
           - 503
           - 504
           httpFailures: 1
           timeouts: 3
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993647765

   > @Revolyssup The domains are modified and provided for security reasons.
   > 
   > ```yaml
   > apiVersion: apisix.apache.org/v2
   > kind: ApisixUpstream
   > metadata:
   >   name: feature-flags-nodes-upstream
   > spec:
   >   loadbalancer:
   >     type: roundrobin
   >   passHost: node
   >   scheme: https
   >   externalNodes:
   >   - type: Domain
   >     name: svc01.corp.com
   >     port: 443
   >     weight: 50
   >   - type: Domain
   >     name: svc02.corp.com
   >     port: 443
   >     weight: 50
   >   healthCheck:
   >     active:
   >       type: https
   >       port: 443
   >       strictTLS: true
   >       httpPath: /
   >       healthy:
   >         successes: 1
   >         interval: 3s
   >         httpCodes:
   >         - 404
   >       unhealthy:
   >         httpCodes:
   >         - 503
   >         - 504
   >         httpFailures: 1
   >         interval: 3s
   >         timeouts: 3
   >     passive:
   >       healthy:
   >         successes: 1
   >         httpCodes:
   >         - 404
   >       unhealthy:
   >         httpCodes:
   >         - 503
   >         - 504
   >         httpFailures: 1
   >         timeouts: 3
   > ```
   
   Also then provide the original route configuration


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1999129672

   @Revolyssup It will take some time to check. I will mention it again as soon as I check. Thank you for your quick response.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-2003023271

   > > @Revolyssup TCP Failures option also doesn't work. There is no special log in the ingress controller.
   > 
   > What configuration did you use to add tcp healthcheck?
   
   I set both `healthcheck.active.unhealthy.tcpFailures` and `healthcheck.passive.unhealthy.tcpFailures` to 1.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-2005609814

   > @kworkbee Thats weird as you can see here that this error log corresponds to reporting a tcp healthcheck failure https://github.com/api7/lua-resty-healthcheck/blob/09a672e0784c75231ceca51e0a4cb39313c020f5/lib/resty/healthcheck.lua#L1078. Can you send the configuration directly from admin API showing the exact configuration?
   
   ```yaml
   {
     "value": {
       "name": "apisix_feature-flags-nodes-upstream",
       "labels": {
         "k8slens-edit-resource-version": "v2",
         "meta_weight": "100",
         "managed-by": "apisix-ingress-controller",
         "argocd.argoproj.io/instance": "dev-tool-apisix"
       },
       "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
       "type": "roundrobin",
       "id": "32eb11c7",
       "pass_host": "node",
       "nodes": [
         {
           "host": "svc01.corp.com",
           "weight": 50,
           "port": 443,
           "priority": 0
         },
         {
           "host": "svc02.corp.com",
           "weight": 50,
           "port": 443,
           "priority": 0
         }
       ],
       "update_time": 1710793944,
       "checks": {
         "active": {
           "http_path": "/",
           "https_verify_certificate": true,
           "port": 443,
           "type": "https",
           "timeout": 1,
           "healthy": {
             "http_statuses": [
               404
             ],
             "successes": 1,
             "interval": 3
           },
           "unhealthy": {
             "http_statuses": [
               503,
               504
             ],
             "http_failures": 1,
             "tcp_failures": 1,
             "timeouts": 3,
             "interval": 3
           },
           "concurrency": 10
         },
         "passive": {
           "healthy": {
             "http_statuses": [
               404
             ],
             "successes": 1
           },
           "unhealthy": {
             "http_statuses": [
               503,
               504
             ],
             "timeouts": 7,
             "tcp_failures": 1,
             "http_failures": 1
           },
           "type": "http"
         }
       },
       "create_time": 1710309045,
       "hash_on": "vars",
       "scheme": "https"
     },
     "key": "/apisix/upstreams/32eb11c7",
     "createdIndex": 151,
     "modifiedIndex": 538
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1990852495

   @kworkbee Thanks for reporting. I'll take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1999102190

   @kworkbee Did you try the above solution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993637784

   @kworkbee I just noticed that you have deployed apisixroute and apisixupstream in different namespaces. This is why while deploying apisixroute, upstream can't be found. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993655347

   The intended structure is as follows:
   ![image](https://github.com/apache/apisix-ingress-controller/assets/18034626/daa93daa-44ca-4c20-a6e8-5eccdcf5c933)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993673420

   @Revolyssup Unlike a few days ago, Status Code is reflected well. But the phenomenon still seems to continue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-2002769426

   @Revolyssup TCP Failures option also doesn't work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-2209150291

   @Revolyssup The main reason seems to be that ALB does not have a fixed IP. I plan to solve it by putting NLB at the front end to make them look at the fixed IP, and it seems that it is not an issue of APISIX itself.
   
   Closing this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1990909221

   ```yaml
   2024-03-12T14:58:40+08:00	info	ingress/ingress.go:121	init apisix ingress controller
   2024-03-12T14:58:40+08:00	info	ingress/ingress.go:123	version:
   Version: 1.8.0
   Git SHA: no-git-module
   Go Version: go1.20.13
   Building OS/Arch: linux/amd64
   Running OS/Arch: linux/amd64
   2024-03-12T14:58:40+08:00	info	ingress/ingress.go:133	use configuration
   {
     "cert_file": "/etc/webhook/certs/cert.pem",
     "key_file": "/etc/webhook/certs/key.pem",
     "log_level": "info",
     "log_output": "stderr",
     "log_rotate_output_path": "",
     "log_rotation_max_size": 100,
     "log_rotation_max_age": 0,
     "log_rotation_max_backups": 0,
     "http_listen": ":8080",
     "https_listen": ":8443",
     "ingress_publish_service": "",
     "ingress_status_address": [],
     "enable_profiling": true,
     "kubernetes": {
       "kubeconfig": "",
       "resync_interval": "6h0m0s",
       "namespace_selector": [],
       "election_id": "ingress-apisix-leader",
       "ingress_class": "apisix",
       "ingress_version": "networking/v1",
       "watch_endpoint_slices": false,
       "api_version": "apisix.apache.org/v2",
       "enable_gateway_api": false,
       "disable_status_updates": false,
       "enable_admission": false
     },
     "apisix": {
       "admin_api_version": "v3",
       "default_cluster_name": "default",
       "default_cluster_base_url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin",
       "default_cluster_admin_key": "******"
     },
     "apisix_resource_sync_interval": "1h0m0s",
     "apisix_resource_sync_comparison": true,
     "plugin_metadata_cm": "",
     "etcdserver": {
       "enabled": false,
       "prefix": "/apisix",
       "listen_address": ":12379",
       "ssl_key_encrypt_salt": "edd1c9f0985e76a2"
     }
   }
   2024-03-12T14:58:40+08:00	info	ingress/ingress.go:145	start ingress controller
   2024-03-12T14:58:40+08:00	info	providers/controller.go:172	start leader election
   I0312 14:58:40.791429       1 leaderelection.go:250] attempting to acquire leader lease apisix/ingress-apisix-leader...
   2024-03-12T14:58:40+08:00	info	providers/controller.go:157	start api server
   2024-03-12T14:58:40+08:00	info	providers/controller.go:143	LeaderElection	{"message": "dev-tool-apisix-ingress-controller-56bc5c7ff-5q6wp became leader", "event_type": "Normal"}
   I0312 14:58:40.814029       1 leaderelection.go:260] successfully acquired lease apisix/ingress-apisix-leader
   2024-03-12T14:58:40+08:00	warn	providers/controller.go:222	found a new leader dev-tool-apisix-ingress-controller-56bc5c7ff-5q6wp
   2024-03-12T14:58:40+08:00	info	providers/controller.go:214	controller now is running as leader	{"namespace": "apisix", "pod": "dev-tool-apisix-ingress-controller-56bc5c7ff-5q6wp"}
   2024-03-12T14:58:40+08:00	info	providers/controller.go:389	controller tries to leading ...	{"namespace": "apisix", "pod": "dev-tool-apisix-ingress-controller-56bc5c7ff-5q6wp"}
   2024-03-12T14:58:41+08:00	info	apisix/stream_route.go:38	resource stream_routes is disabled
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:423	waiting cluster default to ready, it may takes a while
   2024-03-12T14:58:41+08:00	info	apisix/cluster.go:248	syncing cache	{"cluster": "default"}
   2024-03-12T14:58:41+08:00	info	apisix/cluster.go:463	syncing schema	{"cluster": "default"}
   2024-03-12T14:58:41+08:00	info	apisix/cluster.go:252	cache synced	{"cost_time": "23.597419ms", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:434	cluster default now is ready, cost time 23.795508ms
   2024-03-12T14:58:41+08:00	info	providers/controller.go:433	creating controller
   2024-03-12T14:58:41+08:00	info	providers/controller.go:508	init namespaces
   2024-03-12T14:58:41+08:00	info	providers/controller.go:515	wait for resource sync
   2024-03-12T14:58:41+08:00	info	providers/controller.go:523	init providers
   2024-03-12T14:58:41+08:00	info	providers/controller.go:535	try to run providers
   2024-03-12T14:58:41+08:00	info	configmap/configmap.go:87	configmap controller started
   2024-03-12T14:58:41+08:00	info	apisix/apisix_global_rule.go:65	ApisixGlobalRule controller started
   2024-03-12T14:58:41+08:00	info	ingress/ingress.go:85	ingress controller started
   2024-03-12T14:58:41+08:00	info	k8s/secret.go:79	secret controller started
   2024-03-12T14:58:41+08:00	info	endpoint/endpoint.go:70	endpoints controller started
   2024-03-12T14:58:41+08:00	info	apisix/apisix_upstream.go:91	ApisixUpstream controller started
   2024-03-12T14:58:41+08:00	info	apisix/apisix_route.go:104	ApisixRoute controller started
   2024-03-12T14:58:41+08:00	info	apisix/apisix_tls.go:78	ApisixTls controller started
   2024-03-12T14:58:41+08:00	info	apisix/apisix_cluster_config.go:65	ApisixClusterConfig controller started
   2024-03-12T14:58:41+08:00	info	apisix/apisix_consumer.go:69	ApisixConsumer controller started
   2024-03-12T14:58:41+08:00	info	apisix/apisix_plugin_config.go:70	ApisixPluginConfig controller started
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "32eb11c7", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/32eb11c7", "cluster": "default"}
   2024-03-12T14:58:41+08:00	error	apisix/apisix_upstream.go:376	failed to get upstream apisix_feature-flags-nodes-upstream: not found
   2024-03-12T14:58:41+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"False","observedGeneration":1,"lastTransitionTime":"2024-03-12T06:58:41Z","reason":"ResourceSyncAborted","message":"not found"}]}}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9756d1ef", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9756d1ef", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "5d575ecc", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/5d575ecc", "cluster": "default"}
   W0312 14:58:41.373281       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
   2024-03-12T14:58:41+08:00	warn	apisix/apisix_upstream.go:489	sync ApisixUpstream failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-nodes-upstream","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "3f53b397", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/3f53b397", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c11717fc", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c11717fc", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "bbc21df0", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/bbc21df0", "cluster": "default"}
   W0312 14:58:41.418441       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "4586b99b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/4586b99b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "971e4f30", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/971e4f30", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c661d56", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c661d56", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f7be7887", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f7be7887", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "738f1a00", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/738f1a00", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "6bbfb9e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/6bbfb9e", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2b57804a", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2b57804a", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "307b185f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/307b185f", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "477c28c9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/477c28c9", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "b552709e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/b552709e", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "7a4e2355", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/7a4e2355", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "715168ec", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/715168ec", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "eb4e7b5d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/eb4e7b5d", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1595d1c9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1595d1c9", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "4dfc631e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/4dfc631e", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "606b3be0", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/606b3be0", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "dcf65d48", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/dcf65d48", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "abf16dde", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/abf16dde", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "269a612a", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/269a612a", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "55bfce9b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/55bfce9b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "aaa91dc", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/aaa91dc", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "7dada14a", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/7dada14a", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "3669021e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/3669021e", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1d272733", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1d272733", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2ed9e1f1", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2ed9e1f1", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "90640ae5", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/90640ae5", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a51b5d6", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a51b5d6", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c18026ba", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c18026ba", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "80649f40", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/80649f40", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2cac3481", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2cac3481", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "5c1c1952", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/5c1c1952", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "345a1e06", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/345a1e06", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "811258cf", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/811258cf", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1dd1830e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1dd1830e", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c662ff0", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c662ff0", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a9aefac7", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a9aefac7", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "3a3b07f7", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/3a3b07f7", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2ba1a8b9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2ba1a8b9", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "b232187b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/b232187b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "30555848", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/30555848", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "e3ca777b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/e3ca777b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "105d31bd", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/105d31bd", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f3118c88", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f3118c88", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f3118c88", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f3118c88", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "23ef4d0", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/23ef4d0", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "4a344633", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/4a344633", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "40be5e95", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/40be5e95", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "cb84cd70", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/cb84cd70", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "b85d412e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/b85d412e", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1580b33d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1580b33d", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "28399d36", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/28399d36", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9ee809e8", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9ee809e8", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "ee82fd67", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/ee82fd67", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "21194849", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/21194849", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f28d69a9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f28d69a9", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "4815345b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/4815345b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "7a7eff70", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/7a7eff70", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "b4b958f6", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/b4b958f6", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "77fa0f3b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/77fa0f3b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "8f79a5c3", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/8f79a5c3", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "fe03eb03", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/fe03eb03", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "ded031c", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/ded031c", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "3686d301", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/3686d301", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "34aa912b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/34aa912b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2194b02b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2194b02b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a2a099bb", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a2a099bb", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a428cf6f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a428cf6f", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "6bff0afa", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/6bff0afa", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f661101d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f661101d", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "39b6d588", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/39b6d588", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "d63de15d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/d63de15d", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "ac24d8ca", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/ac24d8ca", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "63f31d5f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/63f31d5f", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "6e45694f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/6e45694f", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "a192acda", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/a192acda", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "6fbc2a7f", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/6fbc2a7f", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "bce313ae", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/bce313ae", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "7334d63b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/7334d63b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "426a4118", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/426a4118", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "8dbd848d", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/8dbd848d", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9bccea51", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9bccea51", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "541b2fc4", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/541b2fc4", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "cd127e7e", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/cd127e7e", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c85d3ec9", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c85d3ec9", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "78afb5c", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/78afb5c", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9e83aae6", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9e83aae6", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c8628e4c", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c8628e4c", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "631510da", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/631510da", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c2b246fe", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c2b246fe", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "5283d6b", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/5283d6b", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "771ab980", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/771ab980", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "2cd3d003", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/2cd3d003", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "264dfd09", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/264dfd09", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "c661d56", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/c661d56", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "f7be7887", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/f7be7887", "cluster": "default"}
   2024-03-12T14:58:41+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "738f1a00", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/738f1a00", "cluster": "default"}
   2024-03-12T14:58:42+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":"2024-03-12T06:58:42Z","reason":"ResourcesSynced","message":"Sync Successfully"}]}}
   W0312 14:58:42.417094       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993615797

   @Revolyssup Got these two lines below:
   ```
   2024-03-13T13:46:13+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":"2024-03-13T05:46:13Z","reason":"ResourcesSynced","message":"Sync Successfully"}]}}
   W0313 13:46:13.985770       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993615070

   > @kworkbee What logs do you get immediately after `kubectl apply -f <the above provided upstream>` ?
   
   Can you delete both ApisixRoute and ApisixUpstream, reapply and show logs?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993611408

   @kworkbee What logs do you get immediately after `kubectl apply -f <the above provided upstream>` ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-2003014980

   In APISIX Log `19: self-signed certificate in certificate chain`  I suspect this point.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-2009285981

   > In APISIX Log `19: self-signed certificate in certificate chain` I suspect this point.
   
   Related to this error above, any prerequisites needed to resolve?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993646392

   @kworkbee In the original issue description, the route and upstream you have provided are not in same namespace. So you mean to say in your case, both are in default namespace?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993673633

   > > @Revolyssup ![image](https://private-user-images.githubusercontent.com/18034626/312336717-d8388e78-8328-4c4e-a365-1c247baa8ddc.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTAzMTIyNDksIm5iZiI6MTcxMDMxMTk0OSwicGF0aCI6Ii8xODAzNDYyNi8zMTIzMzY3MTctZDgzODhlNzgtODMyOC00YzRlLWEzNjUtMWMyNDdiYWE4ZGRjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzEzVDA2MzkwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTgxYmVlN2Q0ZWIyYjE0NzlmNzNmMjU3Y2Q5MzQ4NDRiMjhjMjUzNDliYThlNTg3OWEwODA3NTRhMGY2YWI5MmEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ZA2tBjOZwzRSWBQTSkYJBbrf-7GDxkUMJc8ieqZbh-Q)
   > 
   > Okay so there is no sync issue. This upstream configuration matches exactly what you provided via ApisixUpstream. So there is no issue at ingress controller level. Are you still running into the issue where unhealthy upstream is not excluded? Can you try again?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993665881

   @Revolyssup Yes, only one upstream is currently deployed.
   ![image](https://github.com/apache/apisix-ingress-controller/assets/18034626/4d23254f-dc26-4035-ba48-c25c6a00e8e1)
   ![image](https://github.com/apache/apisix-ingress-controller/assets/18034626/f9c2c19f-b9f8-4dad-83f7-a1a20c8d4a90)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee closed issue #2176: bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily
URL: https://github.com/apache/apisix-ingress-controller/issues/2176


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993653448

   > In the process of copying YAML, the information `.metadata.namespace` was incorrectly written, but in reality, both the `ApisixRoute` object and the `ApisixUpstream` object were deployed in the same `apisix` namespace.
   
   Got it. Checking.....


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993649923

   In the process of copying YAML, the information `.metadata.namespace` was incorrectly written, but in reality, both the `ApisixRoute` object and the `ApisixUpstream` object were distributed in the same `apisix` namespace.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1990906123

   @kworkbee Meanwhile can you paste any relevant logs from ingress controller?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993667656

   > @Revolyssup Yes, only one upstream is currently �deployed. ![image](https://private-user-images.githubusercontent.com/18034626/312335683-4d23254f-dc26-4035-ba48-c25c6a00e8e1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTAzMTIwODIsIm5iZiI6MTcxMDMxMTc4MiwicGF0aCI6Ii8xODAzNDYyNi8zMTIzMzU2ODMtNGQyMzI1NGYtZGMyNi00MDM1LWJhNDgtYzI1YzZhMDBlOGUxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzEzVDA2MzYyMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTUxODhkN2FmNzg3NDk2ZDRjYzM2ZTZlYjcwNWY2NjNjMDJmZDc3MTY1YzM2OTk4YWQwMzQ5MjZkZGM2OTJmYjkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.yiuf60kGGGE5mRZ5z10eHtN6tZMLc3QyuR-_p0KJVxw) ![image](https://private-user-images.githubusercontent.com/18034626/312335938-f9c2c19f-b9f8-4dad-83f7-a
 1a20c8d4a90.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTAzMTIwODIsIm5iZiI6MTcxMDMxMTc4MiwicGF0aCI6Ii8xODAzNDYyNi8zMTIzMzU5MzgtZjljMmMxOWYtYjlmOC00ZGFkLTgzZjctYTFhMjBjOGQ0YTkwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzEzVDA2MzYyMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNhMzYxZmFkM2NmMTY4OWNhYjU0ZDdhYzFiYmIyODQwNWZmNjA2OWI0Y2I1NzdmZDU5N2ViNWMxZjM5MGU5ODMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.g4irhGKf93STjGfLZkq5C_Xns-8XGX6Q6iboRcxp2ho)
   
   please click on the passive and active, I wanted to see those values. Please show whole json


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993674755

   The following specific logs are visible in the APISIX Log.
   ```
   2024/03/13 15:42:17 [error] 50#50: *33439 [lua] healthcheck.lua:1383: log(): [healthcheck] (upstream#/apisix/upstreams/32eb11c7) failed SSL handshake with 'X.X.X.X (X.X.X.X:443)', using server name (sni) 'svc01.corp.com': 19: self-signed certificate in certificate chain, context: ngx.timer, client: X.X.X.X, server: 0.0.0.0:9080
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-1993620196

   @Revolyssup As you just mentioned, this is the result of erasing and re-applying both `ApisixRoute` and `ApisixUpstream`.
   ```
   2024-03-13T13:49:24+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
   2024-03-13T13:49:29+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
   2024-03-13T13:49:34+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
   2024-03-13T13:49:34+08:00	warn	providers/controller.go:598	failed to check health for default cluster: timed out waiting for the condition, give up leader
   2024-03-13T13:49:34+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
   2024-03-13T13:49:39+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
   2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
   2024-03-13T13:49:44+08:00	warn	providers/controller.go:598	failed to check health for default cluster: timed out waiting for the condition, give up leader
   2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:599	failed to check health for cluster default: dial tcp 172.20.44.139:9180: connect: connection refused, will retry
   2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "971e4f30", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/971e4f30", "cluster": "default"}
   2024-03-13T13:49:44+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "9756d1ef", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/9756d1ef", "cluster": "default"}
   2024-03-13T13:50:42+08:00	error	translation/apisix_route.go:238	failed to translate ApisixUpstream at Upstream[0]	{"error": "apisixupstream.apisix.apache.org \"feature-flags-nodes-upstream\" not found", "apisix_upstream": "apisix/feature-flags-nodes-upstream"}
   2024-03-13T13:50:42+08:00	error	apisix/route.go:138	failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [32eb11c7], response code: 404"}
   2024-03-13T13:50:42+08:00	error	apisix/apisix_route.go:380	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
   2024-03-13T13:50:42+08:00	warn	apisix/apisix_route.go:482	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
   W0313 13:50:42.477556       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
   2024-03-13T13:50:43+08:00	error	translation/apisix_route.go:238	failed to translate ApisixUpstream at Upstream[0]	{"error": "apisixupstream.apisix.apache.org \"feature-flags-nodes-upstream\" not found", "apisix_upstream": "apisix/feature-flags-nodes-upstream"}
   2024-03-13T13:50:43+08:00	error	apisix/route.go:138	failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [32eb11c7], response code: 404"}
   2024-03-13T13:50:43+08:00	error	apisix/apisix_route.go:380	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
   2024-03-13T13:50:43+08:00	warn	apisix/apisix_route.go:482	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
   2024-03-13T13:50:44+08:00	error	translation/apisix_route.go:238	failed to translate ApisixUpstream at Upstream[0]	{"error": "apisixupstream.apisix.apache.org \"feature-flags-nodes-upstream\" not found", "apisix_upstream": "apisix/feature-flags-nodes-upstream"}
   2024-03-13T13:50:44+08:00	error	apisix/route.go:138	failed to create route: unexpected status code 400; error message: {"error_msg":"failed to fetch upstream info by upstream id [32eb11c7], response code: 404"}
   2024-03-13T13:50:44+08:00	error	apisix/apisix_route.go:380	failed to sync ApisixRoute to apisix	{"error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
   2024-03-13T13:50:44+08:00	warn	apisix/apisix_route.go:482	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "1 error occurred:\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [32eb11c7], response code: 404\"}\n\n\n"}
   2024-03-13T13:50:45+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "32eb11c7", "url": "http://dev-tool-apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/32eb11c7", "cluster": "default"}
   2024-03-13T13:50:45+08:00	error	apisix/apisix_upstream.go:376	failed to get upstream apisix_feature-flags-nodes-upstream: not found
   2024-03-13T13:50:45+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"False","observedGeneration":1,"lastTransitionTime":"2024-03-13T05:50:45Z","reason":"ResourceSyncAborted","message":"not found"}]}}
   W0313 13:50:45.220898       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
   2024-03-13T13:50:45+08:00	warn	apisix/apisix_upstream.go:489	sync ApisixUpstream failed, will retry	{"object": {"Type":1,"Object":{"Key":"apisix/feature-flags-nodes-upstream","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "not found"}
   W0313 13:50:45.273798       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"
   2024-03-13T13:50:46+08:00	error	apisix/apisix_upstream.go:845	update status	{"status": {"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":"2024-03-13T05:50:46Z","reason":"ResourcesSynced","message":"Sync Successfully"}]}}
   W0313 13:50:46.274203       1 warnings.go:70] unknown field "status.conditions[0].lastTransitionTime"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-2004051354

   @kworkbee Thats weird as you can see here that this error log corresponds to reporting a tcp healthcheck failure https://github.com/api7/lua-resty-healthcheck/blob/09a672e0784c75231ceca51e0a4cb39313c020f5/lib/resty/healthcheck.lua#L1078. 
   Can you send the configuration directly from admin API showing what value this tcp_failures is set? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-2003019939

   > @Revolyssup TCP Failures option also doesn't work. There is no special log in the ingress controller.
   
   What configuration did you use to add tcp healthcheck? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: Upstream HealthCheck Issue - Unhealthy Upstream doesn't be excluded temporarily [apisix-ingress-controller]

Posted by "kworkbee (via GitHub)" <gi...@apache.org>.
kworkbee commented on issue #2176:
URL: https://github.com/apache/apisix-ingress-controller/issues/2176#issuecomment-2024447208

   The certificate problem was solved. It was a problem that occurred when terminated traffic went to Secure ALB. However, the `ApisixUpstream` health check still does not work.
   
   @Revolyssup @shreemaan-abhishek 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org