[jira] Resolved: (SYNAPSE-321) HTTP-NIO transport can permanently lock-up with larger messages and moderate concurrency

["Asankha C. Perera (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYNAPSE-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Asankha C. Perera resolved SYNAPSE-321.
---------------------------------------

       Resolution: Fixed
    Fix Version/s: 1.2
         Assignee: Asankha C. Perera

Replaced the dependency on OS/Virtual Pipes altogether, and implemented the suggestion by Oleg to use shared buffers. The solution also uses throttling, to prevent the requirement to expand buffers and JProfiler shows constant memory usage and zero garbage now.

> HTTP-NIO transport can permanently lock-up with larger messages and moderate concurrency
> ----------------------------------------------------------------------------------------
>
>                 Key: SYNAPSE-321
>                 URL: https://issues.apache.org/jira/browse/SYNAPSE-321
>             Project: Synapse
>          Issue Type: Bug
>          Components: Transports
>    Affects Versions: 1.1.1
>         Environment: Windows (XP, Vista) and Linux (Red-Hat Enterprise 5 and Ubuntu 7.10, 8.04) platforms,  performance tuned and not.
>            Reporter: Jake Lambert
>            Assignee: Asankha C. Perera
>            Priority: Critical
>             Fix For: 1.2
>
>         Attachments: myService_500K_Request.xml, sample-MyService.aar
>
>
> I can consistently reproduce an HTTP-NIO lockup by sending larger messages (> 300KB - both with and without attachments) *and* >50 threads of concurrency using SoapUI as a client. I'm directing the messages through a simple forwarding proxy service. This happens for me on both Windows and Linux (Red-Hat and Ubuntu) platforms, both tuned and not.
> After a lot of investigation, here's what I've found:
> - each of the transport receiver I/O dispatcher threads can block writing to the request pipe sink in ServerHandler's inputReady() method when there are no ServerWorker processing threads left
> - the block occurs because the pipe's sink can only buffer a limited number of bytes until a ServerWorker thread is actively reading from the pipe's source
> - a blocked I/O dispatcher thread stops all incoming reads from the client and writes back to the client for its associated connections
> - as more requests come in with no free ServerWorker threads, more of the incoming I/O dispatcher threads are blocked until they are *all* permanently blocked
> - the ServerWorker threads are all blocked either waiting for a free ClientWorker thread or blocked waiting for more input from the client (because the incoming mediation can be complete before a request has been fully read from the incoming socket - this is where I think the larger messages come into play)
> - the ClientWorker threads are all busy waiting to send to their responses back to the client (as previously mentioned, the socket writes back to the client have been for all I/O dispatcher threads)
> - there's no way out of the situation, so Synapse is effectively disabled. As you can see, increasing the number of I/O dispatchers and worker threads can only delay and not fix the problem.
> Since ClientHandler's inputReady() can also block in this way, it probably should be fixed also. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org