You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "unsw-luopan (via GitHub)" <gi...@apache.org> on 2026/01/09 07:43:44 UTC
[I] bug: ai-proxy-multi health check not work [apisix]
unsw-luopan opened a new issue, #12879:
URL: https://github.com/apache/apisix/issues/12879
### Current Behavior
My config about ai-proxy-multi health check:
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" \
-d '{
"id": "ai-proxy-multi-route",
"uri": "/llm/qwen-test",
"methods": [
"POST"
],
"plugins": {
"ai-proxy-multi": {
"instances": [
{
"name": "qwen32b-test",
"provider": "openai-compatible",
"weight": 8,
"auth": {
"header": {
"Authorization": "Bearer aaa"
}
},
"options": {
"model": "qwen3-32b-int4-local"
},
"override": {
"endpoint": "http://10.150.9.179:9998/v1/chat/completions"
},
"checks": {
"active": {
"type": "http",
"host": "10.150.9.179",
"port": 9998,
"https_verify_certificate": false,
"http_path": "/v1/models/qwen3-32b-int4-local",
"healthy": {
"interval": 1,
"successes": 1
},
"unhealthy": {
"interval": 1,
"http_failures": 1
}
}
}
},
{
"name": "qwen8b-test",
"provider": "openai-compatible",
"weight": 2,
"auth": {
"header": {
"Authorization": "Bearer aaa"
}
},
"options": {
"model": "qwen3-8b"
},
"override": {
"endpoint": "http://172.16.3.30:9997/v1/chat/completions"
},
"checks": {
"active": {
"type": "http",
"host": "172.16.3.30",
"port": 9997,
"https_verify_certificate": false,
"http_path": "/v1/models/qwen3-8b",
"healthy": {
"interval": 1,
"successes": 1
},
"unhealthy": {
"interval": 1,
"http_failures": 1
}
}
}
}
]
}
}
}'
After create the route, I can see health check logs in my llm log, so this route create successfully. Then I kill one of my model, and create request for /llm/qwen-test, which is the load balancer in my config. However, apisix still route my request to my dead llm.
By the way, my health check endpoint return 400 instead of 500 if model is dead. I checked the document about health check topic in apisix, which use 500 code as defult. But in ai-proxy-multi health check, there is no way to change the unhealthy http code. Then I wraped a service about health check, and return code 500 if llm is dead, but apisix still route my request to dead llm.
### Expected Behavior
Apisix does not route request to dead llm.
### Error Logs
No error log in apisix.
### Steps to Reproduce
apisix: 3.14.1-debian
### Environment
- APISIX version (run `apisix version`):
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: ai-proxy-multi health check not work [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12879:
URL: https://github.com/apache/apisix/issues/12879#issuecomment-3727614880
The unhealthy status code can be changed through configuration `instances[0].checks.active.unhealthy.http_statuses`. You can refer to https://docs.api7.ai/hub/ai-proxy-multi/configuration
It appears that the documentation here does not describe all the configuration items https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/#attributes. cc @kayx23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: ai-proxy-multi health check not work [apisix]
Posted by "unsw-luopan (via GitHub)" <gi...@apache.org>.
unsw-luopan commented on issue #12879:
URL: https://github.com/apache/apisix/issues/12879#issuecomment-3727755106
Hi @Baoyuantop , thanks for your reply. I add "http_statuses" in my ai-proxy-multi health check config, but apisix still route my request to dead llm service.
My config:
{
"id": "ai-proxy-multi-route",
"uri": "/llm/qwen-test",
"methods": [
"POST"
],
"plugins": {
"ai-proxy-multi": {
"instances": [
{
"name": "qwen32b-test",
"provider": "openai-compatible",
"weight": 8,
"auth": {
"header": {
"Authorization": "Bearer aaa"
}
},
"options": {
"model": "qwen3-32b-int4-local"
},
"override": {
"endpoint": "http://10.150.9.179:9998/v1/chat/completions"
},
"checks": {
"active": {
"type": "[http](https://www.json.cn/http)",
"host": "10.150.9.179",
"port": 9998,
"https_verify_certificate": false,
"http_path": "/v1/models/qwen3-32b-int4-local",
"healthy": {
"interval": 1,
"successes": 1,
"http_statuses": [
200
]
},
"unhealthy": {
"interval": 1,
"http_failures": 1,
"http_statuses": [
400
]
}
}
}
},
{
"name": "qwen8b-test",
"provider": "openai-compatible",
"weight": 2,
"auth": {
"header": {
"Authorization": "Bearer aaa"
}
},
"options": {
"model": "qwen3-8b"
},
"override": {
"endpoint": "http://172.16.3.30:9997/v1/chat/completions"
},
"checks": {
"active": {
"type": "[http](https://www.json.cn/http)",
"host": "172.16.3.30",
"port": 9997,
"https_verify_certificate": false,
"http_path": "/v1/models/qwen3-8b",
"healthy": {
"interval": 1,
"successes": 1,
"http_statuses": [
200
]
},
"unhealthy": {
"interval": 1,
"http_failures": 1,
"http_statuses": [
400
]
}
}
}
}
]
}
}
}
Reproduce: create ai-proxy-multi route to two llms, and test this lb route. Then kill one llm service, and test this route. Apisix
health checking not works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: ai-proxy-multi health check not work [apisix]
Posted by "nic-6443 (via GitHub)" <gi...@apache.org>.
nic-6443 commented on issue #12879:
URL: https://github.com/apache/apisix/issues/12879#issuecomment-4998361476
Went through this while triaging open issues against master. The crash half of this was fixed in 3.17.0 (#13322/#13441). The remaining cause of what you observed — the balancer picker created before the health checker exists gets cached under a key that never changes once the checker shows up, so a picker without health filtering keeps being reused after node changes — is fixed on master by #13505 (with a regression test covering exactly this stale-picker case).
It will ship with the next release, expected later this month. Closing; please reopen if you can still reproduce there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: ai-proxy-multi health check not work [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12879:
URL: https://github.com/apache/apisix/issues/12879#issuecomment-3770874564
Could you explain this in more detail? I don't understand what you mean. Do you mean the code files in the Docker container are inconsistent with those in the GitHub repository?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: ai-proxy-multi health check not work [apisix]
Posted by "nic-6443 (via GitHub)" <gi...@apache.org>.
nic-6443 closed issue #12879: bug: ai-proxy-multi health check not work
URL: https://github.com/apache/apisix/issues/12879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: ai-proxy-multi health check not work [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on issue #12879:
URL: https://github.com/apache/apisix/issues/12879#issuecomment-3822532982
> It appears that the documentation here does not describe all the configuration items https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/#attributes. cc [@kayx23](https://github.com/kayx23)
@Baoyuantop - will get @Yilialinn 's help on this one.
> in docker image apache/apisix:3.14.1-debian, there is no health-checking function in apisix/plugins/ai-proxy-multi.lua
@unsw-luopan - double check your apisix version?
<img width="1434" height="673" alt="Image" src="https://github.com/user-attachments/assets/53c12226-86c4-47ab-9bcf-8eb1f00bc508" />
<img width="1432" height="560" alt="Image" src="https://github.com/user-attachments/assets/590477b7-2197-46fa-8243-f0401047a5f6" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: ai-proxy-multi health check not work [apisix]
Posted by "unsw-luopan (via GitHub)" <gi...@apache.org>.
unsw-luopan commented on issue #12879:
URL: https://github.com/apache/apisix/issues/12879#issuecomment-3728043391
Maybe I find the problem. I use docker to deploy apisix, but in docker image apache/apisix:3.14.1-debian, there is no health-checking function in apisix/plugins/ai-proxy-multi.lua. However, in apisix offcial github, there is a function named "fetch_health_instances". @Baoyuantop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org