You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Charles E. Rolke (Jira)" <ji...@apache.org> on 2021/02/18 15:00:00 UTC

[jira] [Created] (DISPATCH-1968) Crash after running series of 1Mb iperf3 against TCP adaptor

Charles E. Rolke created DISPATCH-1968:
------------------------------------------

             Summary: Crash after running series of 1Mb iperf3 against TCP adaptor
                 Key: DISPATCH-1968
                 URL: https://issues.apache.org/jira/browse/DISPATCH-1968
             Project: Qpid Dispatch
          Issue Type: Bug
          Components: Protocol Adaptors
    Affects Versions: 1.15.0
         Environment: Fedora 32 bare metal 64-bit.

Dispatch at 1.15 release

Proton git branch master @ 5e7d7af8f
            Reporter: Charles E. Rolke


h2. Setup

Running with a minimal TCP adaptor listener / connector on a single router. See attached INTA.conf. These processes run on a single laptop.

Start a iperf3 server on default port 5201: 
    iperf3 -s

Run iperf3 client in a loop to port 5202 served by the TCP adaptor.
    iperf3 -c hostname -p 5202 -n 1000000
h2. Issues

After a few loops the router crashes with malloc having a corrupted doubly linked list.

Sometimes the test client hangs for a few seconds until the iperf server times out.

Qdstat shows many resource leaks of  qd_buffer_t and stream data objects.
h2. Observations
h3. Tracing a single iperf3 session

A wireshark trace of a single iperf3 session shows the client opening two connections to the router and the router opening two connections to the server. This is expected.

As the test runs there is a certain amount of chat between the client and server that works as expected. These messages are test setup and are not part of the iperf mission payload data.

Then the payload data starts. After the server has accepted 8kbytes of iperf payload (in 16 512-byte network packets!!!) the server closes the connection to the TCP connector with a FIN. A few microseconds later the TCP connector sends another 512-byte packet to which the the iperf server responds with a RST.

Shortly thereafter the connections close with a bunch of TCP FIN packets.

The router did not crash.
h3. Running with asan and valgrind memcheck

Running with either of these tools was inconslusive and did not reveal any stray memory writes or double frees that could corrupt the malloc heap.
h2. Next steps

Having the network peer of the TCP connector close the connection mid-stream is a pattern that is not tested in the self tests. A test to generate this pattern is in progress.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org