[I] bug: ai-proxy crashes worker with SIGILL (signal 4) on outbound HTTPS request to LLM endpoint [apisix]

["namannagar89 (via GitHub)" <gi...@apache.org>]

namannagar89 opened a new issue, #13736:
URL: https://github.com/apache/apisix/issues/13736

   ### Current Behavior
   
   When `ai-proxy` makes the outbound **HTTPS** request to a live LLM endpoint, the nginx **worker process crashes with `SIGILL` (signal 4, "illegal instruction")**. The client receives an empty reply, and the gateway log shows:
   
   ```
   [alert] 1#1: worker process NN exited on signal 4
   ```
   
   One crash per request. The worker is respawned by the master, so there's no permanent outage, but every ai-proxy request that reaches the TLS handshake to the upstream LLM kills its worker.
   
   The crash is **not** provider-specific and **not** in the Vertex JWT-signing/token path (we initially suspected that and disproved it — see below). It reproduces whenever ai-proxy opens a real HTTPS/TLS connection to the LLM endpoint.
   
   ### Expected Behavior
   
   ai-proxy proxies the request to the LLM over HTTPS and returns the response (or a normal error), without crashing the worker.
   
   ### What we observed / how we isolated it
   
   | Test | Result |
   |------|--------|
   | `provider: vertex-ai` → Vertex (HTTPS) | worker `exited on signal 4`, empty reply |
   | `provider: openai`, `override.endpoint: http://127.0.0.1:1/...` (unreachable, **no TLS**) | **no crash**, clean HTTP 500 |
   | `provider: openai` → `https://api.openai.com/...` (real **HTTPS**, dummy key) | worker `exited on signal 4`, empty reply (reproduced twice) |
   | Same request after moving the gateway to a different machine family (Intel E2 → AMD T2D) | still crashes |
   
   So the trigger is the **outbound HTTPS/TLS call to a live endpoint** (reproduced with both `openai` and `vertex-ai`); there's no crash when the endpoint is unreachable (no TLS handshake). It reproduces on both Intel and AMD CPUs.
   
   Note: the gateway's normal reverse-proxy TLS to other HTTPS upstreams works fine on the same nodes; only ai-proxy's outbound HTTPS path crashes.
   
   ### Steps to Reproduce
   
   1. Run APISIX `3.17.0` (image `apache/apisix:3.17.0-ubuntu`).
   2. Create a route using `ai-proxy` with any provider whose endpoint is real HTTPS, e.g.:
   
   ```jsonc
   // PUT /apisix/admin/routes/1
   {
     "uri": "/ai",
     "plugins": {
       "ai-proxy": {
         "provider": "openai",
         "auth": { "header": { "Authorization": "Bearer sk-dummy" } },
         "options": { "model": "gpt-4" }
       }
     },
     "upstream": { "type": "roundrobin", "nodes": { "127.0.0.1:1": 1 } }
   }
   ```
   
   3. Send a request:
   
   ```bash
   curl -sv http://<gateway>/ai -H 'Content-Type: application/json' \
     -d '{"messages":[{"role":"user","content":"hi"}]}'
   # => curl: (52) Empty reply from server
   ```
   
   4. Gateway log shows `[alert] ... worker process NN exited on signal 4`.
   
   (A dummy key is fine — the crash happens during/around the TLS connection, before a valid API response is needed.)
   
   ### Environment
   
   - APISIX version: **3.17.0** (`apache/apisix:3.17.0-ubuntu`)
   - OS/Platform: GKE (Ubuntu-based image); reproduced on `e2-standard-4` (Intel Xeon, no AVX-512) and `t2d-standard-8` (AMD EPYC Milan, no AVX-512)
   - ai-proxy plugins load correctly (note: on clusters where the `prometheus` plugin is disabled, ai-proxy fails to load because it requires the `prometheus-cache` shared dict — a separate issue; we declared that dict so ai-proxy loads, then hit this crash).
   
   ### What we could not determine
   
   We do not yet have a coredump/backtrace, so the exact faulting library/instruction is unknown. `SIGILL` (illegal instruction) on a stock amd64 build, in the outbound TLS/crypto path, on CPUs lacking AVX-512, suggests a native code path emitting an instruction the CPU doesn't support — but that's unconfirmed. Happy to gather a backtrace with guidance (e.g. recommended way to enable coredumps for the worker in the official image).
   
   ### Question
   
   Has anyone else hit a `SIGILL` from ai-proxy's outbound HTTPS path, particularly on non-AVX-512 CPUs? Any guidance on capturing a usable backtrace from the official image would help pinpoint the fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] bug: ai-proxy crashes worker with SIGILL (signal 4) on outbound HTTPS request to LLM endpoint [apisix]

["nic-6443 (via GitHub)" <gi...@apache.org>]

nic-6443 closed issue #13736: bug: ai-proxy crashes worker with SIGILL (signal 4) on outbound HTTPS request to LLM endpoint
URL: https://github.com/apache/apisix/issues/13736


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] bug: ai-proxy crashes worker with SIGILL (signal 4) on outbound HTTPS request to LLM endpoint [apisix]

["namannagar89 (via GitHub)" <gi...@apache.org>]

namannagar89 commented on issue #13736:
URL: https://github.com/apache/apisix/issues/13736#issuecomment-5066839634

   ### Correction / refinement: not HTTPS-specific — reproduces over plain HTTP too
   
   Follow-up testing shows the crash is **not** tied to TLS/HTTPS or to the Vertex token path. It reproduces whenever ai-proxy **successfully connects to a reachable upstream and processes the request/response**, regardless of scheme:
   
   | Upstream | Result |
   |----------|--------|
   | Unreachable endpoint (`http://127.0.0.1:1`, connection refused) | no crash (graceful HTTP 500) |
   | Reachable **HTTP** endpoint (in-cluster echo server returning OpenAI-style JSON) | **worker `exited on signal 4`** (reproduced 2/2) |
   | Reachable **HTTPS** endpoint (`api.openai.com`, Vertex) | **worker `exited on signal 4`** |
   
   So the trigger is a **successful upstream connection + ai-proxy's request/response handling**, not the TLS handshake and not Vertex JWT signing.
   
   ### Simplified, provider-agnostic reproduction (no real LLM or TLS needed)
   
   1. Run any plain-HTTP server that returns an OpenAI-style JSON body, e.g.:
      ```
      http-echo -listen=:5678 -text='{"id":"x","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}],"usage":{"total_tokens":2}}'
      ```
   2. Route it through ai-proxy over HTTP:
      ```jsonc
      {"uri":"/ai","plugins":{"ai-proxy":{"provider":"openai","auth":{"header":{"Authorization":"Bearer x"}},
        "options":{"model":"gpt-4"},"override":{"endpoint":"http://<echo-host>:5678/v1/chat/completions"}}},
        "upstream":{"type":"roundrobin","nodes":{"127.0.0.1:1":1}}}
      ```
   3. `POST /ai` with `{"messages":[{"role":"user","content":"hi"}]}` → nginx worker `exited on signal 4`, client gets empty reply.
   
   This should make it reproducible without any external provider/credentials. Title/description effectively broaden from "outbound HTTPS request" to "any successful outbound request the plugin then processes."


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org