You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "leooamaral (via GitHub)" <gi...@apache.org> on 2025/07/29 18:29:42 UTC
[I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
leooamaral opened a new issue, #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498
### Description
As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use Gateway API to optimize self-hosting Generative Models on Kubernetes.
This issue is related to the [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension) project and aims to support its integration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "leooamaral (via GitHub)" <gi...@apache.org>.
leooamaral commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-3206876562
Hello @Baoyuantop,
Yes, of course. Is there somewhere I can follow the discussion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-3208687413
Hi @leooamaral, thanks for your proposal, we look forward to your contribution to this feature. Can you propose a further technical implementation plan? I will ask other maintainers to review your technical implementation plan. Once it is confirmed, you can start contributing code. If you have any questions, please feel free to ask.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-3248031059
I've reviewed this extension project, and it might be feasible for APISIX. However, I must point out that we face significant technical hurdles here.
A particularly critical issue is that this extension requires the gateway data plane to implement the EPP client protocol. This protocol is based on ext-proc, an internal protocol incubated by Envoy that relies on gRPC bidirectional streams instead of standard HTTP requests/responses or WebSockets.
Implementing a gRPC client on an OpenResty-based (essentially nginx) tech stack presents structural complexity. While nginx can proxy gRPC, it lacks a mechanism to autonomously generate gRPC requests internally. A request cannot originate and process itself; it must be initiated by an external client outside nginx.
Numerous OpenResty-based API gateways exist, e.g., APISIX, Kong, and others. None offer native mechanisms for efficiently implementing gRPC clients (though some may use WASM workarounds, which are far from optimal).
How should I summarize this reality? Discrimination brought by Envoy? A significant portion of the ecosystem's established technologies—far more historic and reputable than Envoy—are simply ignored. There's an arrogant assumption that all implementations must conform to this proprietary technology, forced into a “de facto standard” (which it most certainly is not), rather than any neutral, implementation-agnostic approach. This is truly regrettable.
Perhaps we will have to implement an external component to address this, i.e., bridge ext-proc to standard HTTP.
Perhaps you could offer alternative perspectives to inspire a different approach for integrating APISIX with this new ecosystem. I remain open to this and look forward to hearing from you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-3177440884
Hi @leooamaral, any updates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-3994665693
This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "leooamaral (via GitHub)" <gi...@apache.org>.
leooamaral commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-3189996295
Hello @Baoyuantop,
Sorry for the delay.
While checking the `apisix-ingress-controller` code, I noticed it doesn't currently use the `sigs.k8s.io/gateway-api-inference-extension` package.
**The Gateway API Inference Extension** is a project sponsored by [SIG Network](https://github.com/kubernetes/community/blob/master/sig-network/README.md#gateway-api-inference-extension) that extends Gateway API to support advanced routing for LLM traffic. It introduces:
- **Two new CRDs:** `InferenceModel` and `InferencePool`,
- **A load balancing algorithm** optimized for inference workloads,
- **Controllers** to support advanced routing of LLM traffic.
But to accomplish this the project should support:
- [ext-proc](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter),
- [Gateway API](https://github.com/kubernetes-sigs/gateway-api).
The Inference Extension leverages metrics from underlying LLM to make smarter routing and load balancing decisions.
It would be valuable for `apisix-ingress-controller` to integrate with this extension to better support AI/LLM workloads.
Would it be possible for adding this support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-3166949644
Hi @leooamaral, can you describe your proposal in more detail? The new version of apisix ingress controller has complete support for the Gateway API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-3604647164
This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-4181312560
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2498: feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes
URL: https://github.com/apache/apisix-ingress-controller/issues/2498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat: As a Kubernetes user, I want to leverage Gateway API Inference Extension, so that I can use the Gateway API to optimize self-hosting Generative Models on Kubernetes [apisix-ingress-controller]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #2498:
URL: https://github.com/apache/apisix-ingress-controller/issues/2498#issuecomment-3195478047
Hi @leooamaral, thank you for your report, we will discuss this proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org