You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/02/18 17:07:00 UTC

[jira] [Commented] (DISPATCH-1968) Crash after running series of 1Mb iperf3 against TCP adaptor

    [ https://issues.apache.org/jira/browse/DISPATCH-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286596#comment-17286596 ] 

ASF GitHub Bot commented on DISPATCH-1968:
------------------------------------------

ChugR opened a new pull request #1045:
URL: https://github.com/apache/qpid-dispatch/pull/1045


   This patch is for testing DISPATCH-1968 only.
   
   In this patch the tcp echo server used in system_tests_tcp_adaptor is modified to close the connection to the tcp connector as soon as any data is received.
   
   The tcp tests normally close the tcpListener's connection from the echo client and the closure propagates through dispatch so that the connection through the tcpConnector is initiated from the dispatch side. Normal self tests never close the tcpConnector's connection from the network side.
   
   This self test patch provides an easy test bed for investigating what goes wrong when a server disconnects.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Crash after running series of 1Mb iperf3 against TCP adaptor
> ------------------------------------------------------------
>
>                 Key: DISPATCH-1968
>                 URL: https://issues.apache.org/jira/browse/DISPATCH-1968
>             Project: Qpid Dispatch
>          Issue Type: Bug
>          Components: Protocol Adaptors
>    Affects Versions: 1.15.0
>         Environment: Fedora 32 bare metal 64-bit.
> Dispatch at 1.15 release
> Proton git branch master @ 5e7d7af8f
>            Reporter: Charles E. Rolke
>            Priority: Major
>         Attachments: INTA.conf
>
>
> h2. Setup
> Running with a minimal TCP adaptor listener / connector on a single router. See attached INTA.conf. These processes run on a single laptop.
> Start a iperf3 server on default port 5201: 
>     iperf3 -s
> Run iperf3 client in a loop to port 5202 served by the TCP adaptor.
>     iperf3 -c hostname -p 5202 -n 1000000
> h2. Issues
> After a few loops the router crashes with malloc having a corrupted doubly linked list.
> Sometimes the test client hangs for a few seconds until the iperf server times out.
> Qdstat shows many resource leaks of  qd_buffer_t and stream data objects.
> h2. Observations
> h3. Tracing a single iperf3 session
> A wireshark trace of a single iperf3 session shows the client opening two connections to the router and the router opening two connections to the server. This is expected.
> As the test runs there is a certain amount of chat between the client and server that works as expected. These messages are test setup and are not part of the iperf mission payload data.
> Then the payload data starts. After the server has accepted 8kbytes of iperf payload (in 16 512-byte network packets!!!) the server closes the connection to the TCP connector with a FIN. A few microseconds later the TCP connector sends another 512-byte packet to which the the iperf server responds with a RST.
> Shortly thereafter the connections close with a bunch of TCP FIN packets.
> The router did not crash.
> h3. Running with asan and valgrind memcheck
> Running with either of these tools was inconslusive and did not reveal any stray memory writes or double frees that could corrupt the malloc heap.
> h2. Next steps
> Having the network peer of the TCP connector close the connection mid-stream is a pattern that is not tested in the self tests. A test to generate this pattern is in progress.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org