[jira] [Created] (LIVY-1062) Livy Java HTTP client does not follow redirects for POST/DELETE in HA mode

["Nilesh Rathi (Jira)" <ji...@apache.org>]

Nilesh Rathi created LIVY-1062:
----------------------------------

             Summary: Livy Java HTTP client does not follow redirects for POST/DELETE in HA mode
                 Key: LIVY-1062
                 URL: https://issues.apache.org/jira/browse/LIVY-1062
             Project: Livy
          Issue Type: Bug
            Reporter: Nilesh Rathi


When Apache Livy runs in High Availability (HA) mode, passive (standby) Livy server instances redirect incoming client requests to the active leader using HTTP {{307 Temporary Redirect}} responses with a {{Location}} header pointing to the active server.

The Livy Java HTTP client ({{LivyConnection}} in {{client-http}}) does not handle these redirects for {{POST}} and {{DELETE}} requests. Apache HttpClient does not automatically follow redirects for non-idempotent methods, and {{LivyConnection.sendRequest()}} treats any non-200 response as a fatal error. As a result, applications using the Livy Job API (Java/Scala programmatic client) fail when their initial connection lands on a standby node.
h3. Problem

In HA deployments, clients often connect to a load-balanced or round-robin Livy endpoint. When a {{POST}} (e.g., session creation, job submission) or {{DELETE}} (e.g., session/job cancellation) request reaches a passive Livy instance, the server responds with HTTP 307 and a {{Location}} header redirecting to the active leader. {{LivyConnection}} does not follow this redirect and instead throws an {{IOException}}.
h3. Root Cause

In {{client-http/src/main/java/org/apache/livy/client/http/LivyConnection.java}}, the {{sendRequest()}} method executes the HTTP request and only accepts {{HTTP 200 (SC_OK)}} as success. Any other status code (including {{307 Temporary Redirect}}) causes an {{IOException}} to be thrown:

{code:java}
int status = (res.getStatusLine().getStatusCode() / 100) * 100;
if (status == HttpStatus.SC_OK) {
  // success
} else {
  throw new IOException(String.format("%s: %s", res.getStatusLine().getReasonPhrase(), error));
}
{code}

Apache HttpClient follows redirects automatically for {{GET}} requests but *not* for {{POST}} or {{DELETE}}. Since the Livy Job API relies heavily on {{POST}} (submit jobs, create sessions) and {{DELETE}} (cancel jobs, stop sessions), HA redirect handling is broken for the primary programmatic client use cases.
h3. Steps to Reproduce
 # Deploy Livy in HA mode with multiple Livy server instances (active/passive or multi-active with leader redirect).
 # Configure a client application using the Livy Java HTTP client ({{LivyClientBuilder}} / Job API).
 # Point the client at a Livy endpoint where the initial request may land on a passive/standby instance (e.g., via load balancer or by connecting directly to a known passive node).
 # Submit a job or create a session via {{POST}}, or cancel via {{DELETE}}.
 # Observe the client-side failure.
h3. Expected Behavior

{{LivyConnection}} should transparently follow HTTP {{307}} (and {{302}}) redirects for all supported HTTP methods ({{GET}}, {{POST}}, {{DELETE}}), re-issuing the original request (including request body and authentication headers) to the URL specified in the {{Location}} header. The client should return the final successful response from the active Livy server.
h3. Actual Behavior

The client throws an exception on receiving a redirect response:

{code:java}
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Temporary Redirect: <redirect body>
{code}
h3. Impact

* Java/Scala applications using the Livy Job API cannot operate reliably in HA environments.
* Any workload that submits Spark jobs programmatically via {{LivyClient}} (e.g., custom ETL pipelines, integration tests, embedded Spark job runners) fails when the initial TCP connection reaches a non-leader Livy instance.
* {{GET}} requests may appear to work in some cases because HttpClient follows GET redirects by default, masking the problem for read-only operations while write operations consistently fail.
h3. Proposed Fix

Update {{LivyConnection.sendRequest()}} to detect HTTP {{307}} / {{302}} redirect responses, extract the {{Location}} header, and re-issue the original request to the redirect target. The fix should:

* Preserve the original HTTP method, request body, and headers (including {{X-Requested-By}} and authentication credentials).
* Support SPNEGO/Kerberos re-authentication on the redirected request.
* Limit redirect loops (e.g., max redirect count) to prevent infinite loops.
* Add unit tests covering redirect handling for {{POST}} and {{DELETE}}.
h3. Related Work

* [LIVY-718|https://issues.apache.org/jira/browse/LIVY-718] — Support multi-active high availability in Livy
* [LIVY-744|https://issues.apache.org/jira/browse/LIVY-744] — Livy Job API support multi-active HA
* Livy HA design uses HTTP 307 redirects for request routing between cluster nodes (see [LIVY-698|https://issues.apache.org/jira/browse/LIVY-698]).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)