[jira] Commented: (SYNAPSE-321) HTTP-NIO transport can permanently lock-up with larger messages and moderate concurrency

["Asankha C. Perera (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SYNAPSE-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598347#action_12598347 ] 

Asankha C. Perera commented on SYNAPSE-321:
-------------------------------------------

What is your backend service implementation? and what happens when you directly invoke it using SOAP UI with 50 threads? Can you share this information and any other artifacts (including a sample request used) to reproduce this scenario

> HTTP-NIO transport can permanently lock-up with larger messages and moderate concurrency
> ----------------------------------------------------------------------------------------
>
>                 Key: SYNAPSE-321
>                 URL: https://issues.apache.org/jira/browse/SYNAPSE-321
>             Project: Synapse
>          Issue Type: Bug
>          Components: Transports
>    Affects Versions: 1.1.1
>         Environment: Windows (XP, Vista) and Linux (Red-Hat Enterprise 5 and Ubuntu 7.10, 8.04) platforms,  performance tuned and not.
>            Reporter: Jake Lambert
>            Priority: Critical
>
> I can consistently reproduce an HTTP-NIO lockup by sending larger messages (> 300KB - both with and without attachments) *and* >50 threads of concurrency using SoapUI as a client. I'm directing the messages through a simple forwarding proxy service. This happens for me on both Windows and Linux (Red-Hat and Ubuntu) platforms, both tuned and not.
> After a lot of investigation, here's what I've found:
> - each of the transport receiver I/O dispatcher threads can block writing to the request pipe sink in ServerHandler's inputReady() method when there are no ServerWorker processing threads left
> - the block occurs because the pipe's sink can only buffer a limited number of bytes until a ServerWorker thread is actively reading from the pipe's source
> - a blocked I/O dispatcher thread stops all incoming reads from the client and writes back to the client for its associated connections
> - as more requests come in with no free ServerWorker threads, more of the incoming I/O dispatcher threads are blocked until they are *all* permanently blocked
> - the ServerWorker threads are all blocked either waiting for a free ClientWorker thread or blocked waiting for more input from the client (because the incoming mediation can be complete before a request has been fully read from the incoming socket - this is where I think the larger messages come into play)
> - the ClientWorker threads are all busy waiting to send to their responses back to the client (as previously mentioned, the socket writes back to the client have been for all I/O dispatcher threads)
> - there's no way out of the situation, so Synapse is effectively disabled. As you can see, increasing the number of I/O dispatchers and worker threads can only delay and not fix the problem.
> Since ClientHandler's inputReady() can also block in this way, it probably should be fixed also. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org