[jira] [Resolved] (HDDS-16360) Add a health check endpoint to S3 Gateway for load balancer probes

["Wei-Chiu Chuang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wei-Chiu Chuang resolved HDDS-16360.
------------------------------------
    Fix Version/s: 2.3.0
       Resolution: Fixed

> Add a health check endpoint to S3 Gateway for load balancer probes
> ------------------------------------------------------------------
>
>                 Key: HDDS-16360
>                 URL: https://issues.apache.org/jira/browse/HDDS-16360
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: s3gateway
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.3.0
>
>
> h3. Problem
> S3 Gateway is horizontally scaled and fronted by a load balancer (the distribution ships {{s3-haproxy.cfg}} for exactly this). Today S3G exposes no purpose-built health endpoint, so operators fall back to probing an incidental URL:
>  * The S3 listener on {{9878}} — but {{GET /}} is {{ListBuckets}} and any other path is interpreted as a bucket/object, so there's no path that reliably means "am I healthy" without colliding with a bucket name, and in secure mode it requires AWS SigV4.
>  * An admin servlet on {{19878}} ({{{}/conf{}}}, {{{}/jmx{}}}, {{{}/prom{}}}) — but these report only that the Jetty process is up; a 200 there does *not* confirm the gateway can actually serve S3 (e.g. its {{{}OzoneClient{}}}/OM connection may be broken). The bundled {{s3-haproxy.cfg}} in fact does no HTTP health check at all — it balances raw TCP.
> The result: a load balancer can keep routing traffic to an S3G instance that is listening but cannot reach OM, or that is still starting up.
>  
> h3. Goal
> Provide a stable, documented, unauthenticated HTTP endpoint that a load balancer can poll to decide whether an S3G instance should receive traffic, reflecting both process liveness and the gateway's ability to serve S3 requests.
> h3. Proposed approach
> Add a lightweight health servlet on {{S3GatewayWebAdminServer}} (the plain Jetty admin server, port {{{}19878{}}}) at a fixed path (e.g. {{{}/health{}}}), returning {{200}} when healthy and a non-2xx (e.g. {{{}503{}}}) when not, with a tiny plaintext/JSON body for humans.
> Serving it on the admin server rather than the S3 listener avoids the S3 path-router bucket-name collision entirely and keeps it off the SigV4-authenticated path. Load balancers can still gate the S3 traffic port on this endpoint: HAProxy in {{mode tcp}} performs Layer 7 HTTP health checks independently of the traffic mode and can target a different port than the one it balances ({{{}check port 19878{}}}), so it balances raw S3 TCP on {{9878}} while HTTP-probing {{/health}} on {{{}19878{}}}.
> The endpoint reports *Readiness* — the gateway can serve S3: its cached {{OzoneClient}} is initialized and a cheap OM reachability check succeeds. This is what the LB actually needs to gate traffic on.
> The endpoint must be reachable by an LB without Kerberos/SPNEGO. Requires explicitly excluding the health path from the S3G admin-server auth filters.
> New {{@Config}} keys (e.g. enable/disable,  probe timeout), defaulting to enabled.
> h3. Scope
>  * New health servlet + registration on {{{}S3GatewayWebAdminServer{}}}.
>  * Auth-filter exclusion for the health path in secure mode.
>  * {{@Config}} keys with getters (self-documenting; no {{ozone-default.xml}} edits).
>  * Update {{s3-haproxy.cfg}} (and {{{}common/s3-haproxy.cfg{}}}) to add an {{option httpchk}} against {{/health}} on the admin port, as a worked example.
> h3. Out of scope
>  * Per-bucket / OM-side health, cluster-wide readiness aggregation.
>  * A health path on the S3 listener ({{{}9878{}}}) itself — unnecessary given HAProxy can check the admin port.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org