[jira] [Created] (CXF-9250) HttpClientHTTPConduit: failed exchange without request body is re-sent endlessly from the exceptionally callback (e.g. remote WSDL fetch)

["Sebastian Schubert (Jira)" <ji...@apache.org>]

Sebastian Schubert created CXF-9250:
---------------------------------------

             Summary: HttpClientHTTPConduit: failed exchange without request body is re-sent endlessly from the exceptionally callback (e.g. remote WSDL fetch)
                 Key: CXF-9250
                 URL: https://issues.apache.org/jira/browse/CXF-9250
             Project: CXF
          Issue Type: Bug
          Components: Transports
    Affects Versions: 4.1.8, 4.2.3, 4.2.2, 3.6.12, 4.1.7, 4.1.6, 3.6.11, 4.2.1, 4.0.11, 4.1.5, 4.0.10, 3.6.9, 4.1.4, 3.6.8, 4.0.9, 4.1.3, 3.6.7, 4.0.8, 4.1.2, 4.2.0, 4.2.4
            Reporter: Sebastian Schubert


When an exchange sent through {{HttpClientHTTPConduit}} fails (TLS handshake failure, connection reset, connection refused, ...) and the request had no body (e.g. a GET), the conduit sends the same request again from the {{future.exceptionally(...)}} callback. That request fails too, its callback sends it again, and so on. The loop only ends when an exchange eventually succeeds. 


The most common trigger is the remote WSDL download done by {{TransportURIResolver}} when a JAX-WS client is created with an http(s) WSDL URL ({{{}Service.create(url, qname){}}} or a generated {{Service}} subclass). The caller gets a normal {{{}WebServiceException{}}}, but a background resend loop keeps running for every failed attempt.
h3. Impact (observed in production)

A JAX-WS client whose remote WSDL host started failing TLS handshakes (server certificate chain not in the client truststore) produced ~150,000-200,000 requests/min to that host from a handful of application calls. Side effects:
 * ~250 {{ForkJoinPool.commonPool}} workers blocked in {{HttpClientHTTPConduit$HttpClientWrappedOutputStream.getResponse}} -> {{{}CompletableFuture.get{}}}, called from {{{}lambda$setProtocolHeaders$...{}}}; {{compensatedBlock}} spawning replacement workers
 * high CPU from repeated PKIX path building
 * no log output: {{TransportURIResolver}} logs the failure at FINEST, and the future returned by {{exceptionally}} is never observed
 * application-level retry/circuit-breaker logic does not see the resends, because they happen after the original call has already failed Traffic stopped immediately once the certificate was added to the truststore, i.e. once the next resend succeeded.

h3.  
h3. Root cause

{{HttpClientHTTPConduit.HttpClientWrappedOutputStream.setProtocolHeaders()}} builds the request, calls {{sendAsync}} and registers:
{code:java}
  future = cl.sendAsync(request, handler);
  future.exceptionally(ex -> {
      if (pout != null) {
          synchronized (pout) {
              pout.notifyAll();
          }
      }
      try {
          close();
      } catch (IOException e) {
          ex.addSuppressed(e);
      }
      return null;
  });
  {code}
{{close()}} is the full {{{}HTTPConduit.WrappedOutputStream.close(){}}}:
{code:java}
  boolean exceptionSet = outMessage.getContent(Exception.class) != null;
  if (!written && !exceptionSet) {
      handleHeadersTrustCaching();   // -> setProtocolHeaders() -> sendAsync(...) again
  }
  ...
  handleResponse();
  {code}
For a request without body, {{written}} is never set (it is only set by {{{}AbstractWrappedOutputStream.write{}}}), and on the synchronous path the out message never gets an {{Exception}} content. So every invocation of {{close()}} from the callback runs {{handleHeadersTrustCaching()}} -> {{{}setProtocolHeaders(){}}}, which sends a new request and registers a new callback. The callback thread then blocks in {{handleResponse()}} -> {{getResponse()}} on the new future. Each failed attempt therefore
starts a self-sustaining chain with roughly one blocked thread per chain.
With {{{}URLConnectionHTTPConduit{}}}, {{setProtocolHeaders()}} only sets headers on the connection object, so running it twice has no network effect. In {{HttpClientHTTPConduit}} the same template hook performs the send, which makes the stream's {{close()}} non-idempotent.
The {{close()}} call in the callback was added in 3.6.7 / 4.0.8 / 4.1.2 (not present in 3.6.6 / 4.0.7 / 4.1.1), likely as part of CXF-9115. {{HttpClientWrappedOutputStream.exception}} is read in {{setupWrappedStream()}} and {{handleNoOutput()}} but never assigned.
Requests with a body (e.g. SOAP POST) are mostly unaffected because {{written}} becomes true after the first write. However, {{written = true}} is only set after {{onFirstWrite()}} returns, while the callback is already registered inside {{onFirstWrite()}} -> {{{}setProtocolHeaders(){}}}. So an exchange that fails inside that window can be re-sent as well.
h3. Reproducer

A server that accepts and immediately closes every connection, and one JAX-WS {{Service.create}} with a WSDL URL pointing to it:
{code:java}
  import java.net.*;
  import java.util.concurrent.atomic.AtomicInteger;
  import javax.xml.namespace.QName;
  import jakarta.xml.ws.Service;   // javax.xml.ws.Service on 3.6.x
  public class ResendLoopRepro {
      public static void main(String[] args) throws Exception {
          AtomicInteger requests = new AtomicInteger();
          ServerSocket server = new ServerSocket(0, 200, InetAddress.getLoopbackAddress());
          Thread acceptor = new Thread(() -> {
              while (true) {
                  try (Socket s = server.accept()) {
                      requests.incrementAndGet();          // close without responding
                  } catch (Exception e) {
                      return;
                  }
              }
          });
          acceptor.setDaemon(true);
          acceptor.start();
          URL wsdl = new URL("http://localhost:" + server.getLocalPort() + "/service.wsdl");
          try {
              Service.create(wsdl, new QName("urn:test", "TestService"));
          } catch (Exception expected) {
              System.out.println("Service.create failed as expected; requests so far: " + requests.get());
          }
          Thread.sleep(2000);
          System.out.println("Requests 2s later: " + requests.get());   // keeps growing
          System.exit(0);
      }
  }
  {code}
Observed with one {{Service}} creation:
||CXF / Java||requests when the call failed||requests after 2 s||
|3.6.11 / Java 17|102|460|
|4.1.7 / Java 21|143|442|

Expected: 1 request (or a small bounded number); nothing after the call has failed.
A TLS server with an untrusted certificate shows the same behaviour with {{{}SSLHandshakeException: PKIX path building failed{}}}.
h3. Workarounds
 * {{{}-Dorg.apache.cxf.transport.http.forceURLConnection=true{}}}, or the contextual property {{force.urlconnection.http.conduit=true}} (note: a client/request-context property does not reach the WSDL fetch in {{{}TransportURIResolver{}}}, which uses a bare message)
 * load the WSDL from the classpath instead of http(s)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)