[jira] [Created] (YUNIKORN-3470) Flaky webservice shutdown caused by incomplete HTTP response body cleanup

["Yen-Hua Chen (Jira)" <ji...@apache.org>]

Yen-Hua Chen created YUNIKORN-3470:
--------------------------------------

             Summary: Flaky webservice shutdown caused by incomplete HTTP response body cleanup
                 Key: YUNIKORN-3470
                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3470
             Project: Apache YuniKorn
          Issue Type: Bug
          Components: test - unit
         Environment: Linux arm64
Go 1.27.1
Race detector enabled
Tested on commit a38869d52bd089404e05b8e733e16b84de36035e
            Reporter: Yen-Hua Chen
            Assignee: Yen-Hua Chen


{{Test_RedirectDebugHandler}} intermittently fails during webservice shutdown under repeated race-enabled test execution.

h3. Reproduction

Tested against:

{code}
a38869d52bd089404e05b8e733e16b84de36035e
{code}

Command:

{code}
go test -race ./pkg/webservice \
  -run '^Test_RedirectDebugHandler$' \
  -count=100 -v
{code}

Observed 5 failures in 100 iterations.

Representative failure:

{code}
=== RUN   Test_RedirectDebugHandler/statedump
=== RUN   Test_RedirectDebugHandler/stacks
=== NAME  Test_RedirectDebugHandler
    webservice_test.go:61: failed to stop webapp
--- FAIL: Test_RedirectDebugHandler (5.02s)
    --- PASS: Test_RedirectDebugHandler/statedump (0.00s)
    --- PASS: Test_RedirectDebugHandler/stacks (0.01s)
{code}

A longer stress run reproduced 37 failures in 500 iterations.

h3. Root cause

The successful HTTP responses in {{Test_RedirectDebugHandler}} are closed without first consuming their finite response bodies to EOF.

Under timing-sensitive execution, response draining can overlap the next request. This can leave an additional accepted TCP connection in {{StateNew}}, which prevents {{http.Server.Shutdown()}} from reaching quiescence before YuniKorn's 5-second shutdown deadline.

A similar response-lifecycle issue exists in two successful GET requests in {{Test_RouterHandling}}.

The surviving connection can also remain associated with the shared {{http.DefaultTransport}}, allowing connection state from the retired webservice instance to affect the following test.

h3. Proposed fix

Fully consume the finite HTTP response bodies before closing them.

The change is test-only and does not modify production webservice shutdown behavior.

Validation of the proposed fix:

{code}
Test_RedirectDebugHandler -race -count=500:
  before: 37 failures / 500
  after:   0 failures / 500

Test_RedirectDebugHandler + Test_RouterHandling -race -count=100:
  PASS

pkg/webservice -race -count=20:
  PASS

make test:
  PASS
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org