You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "Revolyssup (via GitHub)" <gi...@apache.org> on 2025/09/01 06:28:59 UTC
[PR] feat: add fallback mechanism for specific error codes in ai-proxy-multi [apisix]
Revolyssup opened a new pull request, #12571:
URL: https://github.com/apache/apisix/pull/12571
### Description
<!-- Please include a summary of the change and which issue is fixed. -->
<!-- Please also include relevant motivation and context. -->
#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
-->
Fixes #
### Checklist
- [ ] I have explained the need for this PR and the problem it solves
- [ ] I have explained the changes or the new features added to this PR
- [ ] I have added tests corresponding to this change
- [ ] I have updated the documentation to reflect this change
- [ ] I have verified that this change is backward compatible (If not, please discuss on the [APISIX mailing list](https://github.com/apache/apisix/tree/master#community) first)
<!--
Note
1. Mark the PR as draft until it's ready to be reviewed.
2. Always add/update tests for any changes unless you have a good reason.
3. Always update the documentation to reflect the changes made in the PR.
4. Make a new commit to resolve conversations instead of `push -f`.
5. To resolve merge conflicts, merge master instead of rebasing.
6. Use "request review" to notify the reviewer after making changes.
7. Only a reviewer can mark a conversation as resolved.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add fallback mechanism for specific error codes in ai-proxy-multi [apisix]
Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup merged PR #12571:
URL: https://github.com/apache/apisix/pull/12571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add fallback mechanism for specific error codes in ai-proxy-multi [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #12571:
URL: https://github.com/apache/apisix/pull/12571#discussion_r2315224892
##########
docs/en/latest/plugins/ai-proxy-multi.md:
##########
@@ -51,7 +51,7 @@ In addition, the Plugin also supports logging LLM request information in the acc
| Name | Type | Required | Default | Valid Values | Description |
|------------------------------------|----------------|----------|-----------------------------------|--------------|-------------|
-| fallback_strategy | string | False | instance_health_and_rate_limiting | instance_health_and_rate_limiting | Fallback strategy. When set, the Plugin will check whether the specified instance’s token has been exhausted when a request is forwarded. If so, forward the request to the next instance regardless of the instance priority. When not set, the Plugin will not forward the request to low priority instances when token of the high priority instance is exhausted. |
+| fallback_strategy | string or array | False | | "instance_health_and_rate_limiting" "http_429", "http_5xx" or ["rate_limiting", "http_429", "http_5xx"] | Fallback strategy. When set, the Plugin will check whether the specified instance’s token has been exhausted when a request is forwarded. If so, forward the request to the next instance regardless of the instance priority. When not set, the Plugin will not forward the request to low priority instances when token of the high priority instance is exhausted. |
Review Comment:
The valid value format is slightly confusing, since this doc uses `[ ]` to list all the configurable values for strings as well. How about:
```suggestion
| fallback_strategy | string or array | False | | string: "instance_health_and_rate_limiting", "http_429", "http_5xx"<br>array: ["rate_limiting", "http_429", "http_5xx"] | Fallback strategy. When set, the Plugin will check whether the specified instance’s token has been exhausted when a request is forwarded. If so, forward the request to the next instance regardless of the instance priority. When not set, the Plugin will not forward the request to low priority instances when token of the high priority instance is exhausted. |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add fallback mechanism for specific error codes in ai-proxy-multi [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #12571:
URL: https://github.com/apache/apisix/pull/12571#discussion_r2315403911
##########
docs/en/latest/plugins/ai-rate-limiting.md:
##########
@@ -413,9 +413,9 @@ X-AI-RateLimit-Reset-deepseek-instance: 0
### Configure Instance Priority and Rate Limiting
-The following example demonstrates how you can configure two models with different priorities and apply rate limiting on the instance with a higher priority. In the case where `fallback_strategy` is set to `instance_health_and_rate_limiting`, the Plugin should continue to forward requests to the low priority instance once the high priority instance's rate limiting quota is fully consumed.
+The following example demonstrates how you can configure two models with different priorities and apply rate limiting on the instance with a higher priority. In the case where `fallback_strategy` is set to `["rate_limiting"]`, the Plugin should continue to forward requests to the low priority instance once the high priority instance's rate limiting quota is fully consumed.
Review Comment:
Sounds good. Just a reminder that there's more than one example to update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add fallback mechanism for specific error codes in ai-proxy-multi [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #12571:
URL: https://github.com/apache/apisix/pull/12571#discussion_r2315386840
##########
docs/en/latest/plugins/ai-proxy-multi.md:
##########
@@ -51,7 +51,7 @@ In addition, the Plugin also supports logging LLM request information in the acc
| Name | Type | Required | Default | Valid Values | Description |
|------------------------------------|----------------|----------|-----------------------------------|--------------|-------------|
-| fallback_strategy | string | False | instance_health_and_rate_limiting | instance_health_and_rate_limiting | Fallback strategy. When set, the Plugin will check whether the specified instance’s token has been exhausted when a request is forwarded. If so, forward the request to the next instance regardless of the instance priority. When not set, the Plugin will not forward the request to low priority instances when token of the high priority instance is exhausted. |
+| fallback_strategy | string or array | False | | "instance_health_and_rate_limiting" "http_429", "http_5xx" or ["rate_limiting", "http_429", "http_5xx"] | Fallback strategy. When set, the Plugin will check whether the specified instance’s token has been exhausted when a request is forwarded. If so, forward the request to the next instance regardless of the instance priority. When not set, the Plugin will not forward the request to low priority instances when token of the high priority instance is exhausted. |
Review Comment:
```suggestion
| fallback_strategy | string or array | False | | string: "instance_health_and_rate_limiting", "http_429", "http_5xx"<br>array: ["rate_limiting", "http_429", "http_5xx"] | Fallback strategy. The option `instance_health_and_rate_limiting` is kept for backward compatibility and is functionally the same as `rate_limiting`. With `rate_limiting` or `instance_health_and_rate_limiting`, when the current instance’s quota is exhausted, the request is forwarded to the next instance regardless of priority. With `http_429`, if an instance returns status code 429, the request is retried with other instances. With `http_5xx`, if an instance returns a 5xx status code, the request is retried with other instances. If all instances fail, the plugin returns the last error response code. When not set, the Plugin will not forward the request to low priority instances when token of the high priority instance is exhausted. |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add fallback mechanism for specific error codes in ai-proxy-multi [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #12571:
URL: https://github.com/apache/apisix/pull/12571#discussion_r2315382117
##########
docs/en/latest/plugins/ai-proxy-multi.md:
##########
@@ -51,7 +51,7 @@ In addition, the Plugin also supports logging LLM request information in the acc
| Name | Type | Required | Default | Valid Values | Description |
|------------------------------------|----------------|----------|-----------------------------------|--------------|-------------|
-| fallback_strategy | string | False | instance_health_and_rate_limiting | instance_health_and_rate_limiting | Fallback strategy. When set, the Plugin will check whether the specified instance’s token has been exhausted when a request is forwarded. If so, forward the request to the next instance regardless of the instance priority. When not set, the Plugin will not forward the request to low priority instances when token of the high priority instance is exhausted. |
+| fallback_strategy | string or array | False | | "instance_health_and_rate_limiting" "http_429", "http_5xx" or ["rate_limiting", "http_429", "http_5xx"] | Fallback strategy. When set, the Plugin will check whether the specified instance’s token has been exhausted when a request is forwarded. If so, forward the request to the next instance regardless of the instance priority. When not set, the Plugin will not forward the request to low priority instances when token of the high priority instance is exhausted. |
Review Comment:
Maybe mention the option `instance_health_and_rate_limiting` is kept for backward compatibility and functionally, it is the same as `rate_limiting` as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add fallback mechanism for specific error codes in ai-proxy-multi [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #12571:
URL: https://github.com/apache/apisix/pull/12571#discussion_r2315203614
##########
docs/en/latest/plugins/ai-rate-limiting.md:
##########
@@ -413,9 +413,9 @@ X-AI-RateLimit-Reset-deepseek-instance: 0
### Configure Instance Priority and Rate Limiting
-The following example demonstrates how you can configure two models with different priorities and apply rate limiting on the instance with a higher priority. In the case where `fallback_strategy` is set to `instance_health_and_rate_limiting`, the Plugin should continue to forward requests to the low priority instance once the high priority instance's rate limiting quota is fully consumed.
+The following example demonstrates how you can configure two models with different priorities and apply rate limiting on the instance with a higher priority. In the case where `fallback_strategy` is set to `["rate_limiting"]`, the Plugin should continue to forward requests to the low priority instance once the high priority instance's rate limiting quota is fully consumed.
Review Comment:
1. Why is the example changed to use the array format? The string is also supported.
2. The same example is also in the `ai-proxy-multi` doc. If it needs to be changed, both should have the same change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add fallback mechanism for specific error codes in ai-proxy-multi [apisix]
Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on code in PR #12571:
URL: https://github.com/apache/apisix/pull/12571#discussion_r2315207668
##########
docs/en/latest/plugins/ai-rate-limiting.md:
##########
@@ -413,9 +413,9 @@ X-AI-RateLimit-Reset-deepseek-instance: 0
### Configure Instance Priority and Rate Limiting
-The following example demonstrates how you can configure two models with different priorities and apply rate limiting on the instance with a higher priority. In the case where `fallback_strategy` is set to `instance_health_and_rate_limiting`, the Plugin should continue to forward requests to the low priority instance once the high priority instance's rate limiting quota is fully consumed.
+The following example demonstrates how you can configure two models with different priorities and apply rate limiting on the instance with a higher priority. In the case where `fallback_strategy` is set to `["rate_limiting"]`, the Plugin should continue to forward requests to the low priority instance once the high priority instance's rate limiting quota is fully consumed.
Review Comment:
Yes string is also supported but using array is recommended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add fallback mechanism for specific error codes in ai-proxy-multi [apisix]
Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on code in PR #12571:
URL: https://github.com/apache/apisix/pull/12571#discussion_r2315208469
##########
docs/en/latest/plugins/ai-rate-limiting.md:
##########
@@ -413,9 +413,9 @@ X-AI-RateLimit-Reset-deepseek-instance: 0
### Configure Instance Priority and Rate Limiting
-The following example demonstrates how you can configure two models with different priorities and apply rate limiting on the instance with a higher priority. In the case where `fallback_strategy` is set to `instance_health_and_rate_limiting`, the Plugin should continue to forward requests to the low priority instance once the high priority instance's rate limiting quota is fully consumed.
+The following example demonstrates how you can configure two models with different priorities and apply rate limiting on the instance with a higher priority. In the case where `fallback_strategy` is set to `["rate_limiting"]`, the Plugin should continue to forward requests to the low priority instance once the high priority instance's rate limiting quota is fully consumed.
Review Comment:
okay i will change the other doc as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org