[I] Fail fast on an unmapped port, and report the mapping back (tooling-llmao)

["andrewmusselman (via GitHub)" <gi...@apache.org>]

andrewmusselman opened a new issue, #26:
URL: https://github.com/apache/tooling-llmao/issues/26

   A vLLM server on an unmapped Vast port starts cleanly, passes its local health
   check, and is unreachable from outside. Nothing in the logs says so — the
   failure only appears when something tries to route to it.
    
   ### Background
    
   Vast maps container ports to public ones **at instance creation**. A port not
   declared on the template cannot be added afterwards; the instance has to be
   recreated. The external port is assigned from the instance's range and is not
   the same number as the internal one:
    
   ```
   $ env | grep VAST_TCP_PORT
   VAST_TCP_PORT_10100=12711
   $ echo "$PUBLIC_IPADDR"
   80.188.223.202
   ```
    
   So the server is reachable at `80.188.223.202:12711`, and `10100` appears
   nowhere outside the box.
    
   ### Three changes
    
   **1. Refuse to launch on an unmapped port.**
    
   `install_set.py` should read `VAST_TCP_PORT_<port>` before writing the
   Supervisor unit. If it is absent, fail with a message naming the port rather
   than starting a server nobody can reach.
    
   This is the highest-value part. The current failure is silent and looks like
   success.
    
   **2. Force `--host 0.0.0.0` in the generated unit.**
    
   The Vast template's own vLLM launches with `--host 127.0.0.1`:
    
   ```
   vllm serve Qwen/Qwen3.5-9B ... --host 127.0.0.1 --port 18000
   ```
    
   Bound to loopback, it is unreachable from outside regardless of how the port is
   mapped — and it looks identical to a mapping problem from the outside. Worth
   setting explicitly rather than inheriting whatever the template does.
    
   **3. Report the public address back to llmao.**
    
   The box knows `PUBLIC_IPADDR` and its `VAST_TCP_PORT_*` map at exactly the
   moment it fetches `GET /vllm/config`. Having `install_set.py` POST them back
   means llmao never has to be told:
    
   ```json
   POST /vllm/observed
   { "public_ip": "80.188.223.202",
     "port_map": { "10100": 12711 },
     "name": "gemma4-26b" }
   ```
    
   llmao can then build `api_base` for the route itself. This is an extension of
   the observed-state reporting already planned, not a new mechanism.
    
   ### Also worth doing
    
   **Declare ports on the template, not per instance.** Every box then gets them
   without anyone remembering, and a missing port becomes a template fix rather
   than a rebuild.
    
   ### Note for anything pinning the address
    
   The external port differs per instance, so a CDN backend or hiera value pinned
   to `IP:port` needs updating on every rebuild. Worth asking whether the CDN can
   point at a hostname we control instead, so a rebuild becomes a DNS update
   rather than a ticket.
    
   ### Reproducing
    
   ```bash
   # on the box -- is vLLM listening?
   ss -tlnp | grep 10100
   curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:10100/v1/models
    
   # from outside -- is the mapping routing?
   curl -sS -m 10 -o /dev/null -w '%{http_code}\n' http://$PUBLIC_IPADDR:<external>/v1/models
   ```
    
   Local works and external times out → the port was not declared at creation.
   Both fail → vLLM is not up, or it is bound to loopback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org