You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@synapse.apache.org by ant elder <an...@gmail.com> on 2007/03/23 09:51:01 UTC
Fwd: [jira] Resolved: (HTTPCORE-60) Transport appears to be hanging because an unchecked exception caused the I/O dispatch thread to terminate
I've tried on several JDKs now and _always_ get similar intermittent I/O
related errors. I can use JMeter directly against Axis2-1.1.1 without any
problems at all, so this does look like some issue with the NIO transport.
Be really good to hear from other Windows users to see if this is just my
specific environment or a more general problem problem.
To recreate:
1) build Synapse server sample by running 'ant' in the
samples\axis2Server\src\SimpleStockQuoteService directory
2) start the sample service by running samples\axis2Server\axis2server.bat
3) get the Synapse config (either 8 or 501) from
http://people.apache.org/~antelder/temp/, put in repository\conf\sample and
start syanps: bin\synapse.bat -sample=8
4) get the JMeter config test1.jmx from
http://people.apache.org/~antelder/temp/, start Jmeter and File -> Open and
point to the test1.jmx file
5) JMeter Run -> Start and after not to long IO errors should appear in the
Syanpse console
...ant
---------- Forwarded message ----------
From: Asankha C. Perera <as...@wso2.com>
Date: Mar 22, 2007 4:58 PM
Subject: Re: [jira] Resolved: (HTTPCORE-60) Transport appears to be hanging
because an unchecked exception caused the I/O dispatch thread to terminate
To: HttpComponents Project <ht...@jakarta.apache.org>
Oleg/Ant
I am guessing this is something to do with Windows or the JDK you use.. But
I am unable to test this week, so will try to my best to try this sometime
next week. As I said, on Linux I have run the system through thousands of
messages and multiple threads concurrently and have fixed all the issues I
came across.
So Oleg, I do not see this as a blocker for the HttpCore release - but I
will use your latest snapshots in Synapse to check on this in future if it
occurs again
thanks
asankha
Oleg Kalnichevski (JIRA) wrote:
[ https://issues.apache.org/jira/browse/HTTPCORE-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Kalnichevski resolved HTTPCORE-60.
---------------------------------------
Resolution: Fixed
Anthony
It turned out ClosedChannelException is a checked I/O exception so it
cannot kill the I/O dispatch thread. So, apparently I was wrong in my
initial assertion about the cause of the Synapse I/O transport lockup.
I tweaked HttpCore code a little and changed the IOSessionImpl to
catch all ChannelClosedException-s thrown by the underlying byte
channel just in case.
Please review the changes and let me know if it is okay to proceed
with the release
Oleg
Transport appears to be hanging because an unchecked exception
caused the I/O dispatch thread to terminate
----------------------------------------------------------------------------------------------------------
Key: HTTPCORE-60
URL: https://issues.apache.org/jira/browse/HTTPCORE-60
Project: HttpComponents Core
Issue Type: Bug
Affects Versions: 4.0-alpha4
Reporter: ant elder
Assigned To: Oleg Kalnichevski
Fix For: 4.0-alpha4
See discussion on synapse-dev mailing list:
http://www.nabble.com/Intermittent-IO-Errors-using-Synapse-tf3439957.html
The transport appears to be hanging because an unchecked exception
caused the I/O dispatch thread to terminate. I believe there are several
different types of problems (at least two) that we are seeing here.
[I/O reactor worker thread 5] ERROR ServerHandler - I/O Error : null
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:112)
at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:139)
--------------------------------------------------------------------- To
unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.org For
additional commands, e-mail: httpcomponents-dev-help@jakarta.apache.org
HttpCore stability and performance was Re: Fwd: [jira] Resolved:
(HTTPCORE-60) Transport appears to be hanging because an unchecked
exception caused the I/O dispatch thread to terminate
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2007-03-23 at 08:51 +0000, ant elder wrote:
> I've tried on several JDKs now and _always_ get similar intermittent
> I/O related errors. I can use JMeter directly against Axis2-1.1.1
> without any problems at all, so this does look like some issue with
> the NIO transport. Be really good to hear from other Windows users to
> see if this is just my specific environment or a more general problem
> problem.
>
> To recreate:
>
> 1) build Synapse server sample by running 'ant' in the samples
> \axis2Server\src\SimpleStockQuoteService directory
> 2) start the sample service by running samples\axis2Server
> \axis2server.bat
> 3) get the Synapse config (either 8 or 501) from
> http://people.apache.org/~antelder/temp/, put in repository\conf
> \sample and start syanps: bin\synapse.bat -sample=8
> 4) get the JMeter config test1.jmx from
> http://people.apache.org/~antelder/temp/, start Jmeter and File ->
> Open and point to the test1.jmx file
> 5) JMeter Run -> Start and after not to long IO errors should appear
> in the Syanpse console
>
> ...ant
>
Anthony, Asankha and all
It _suspect_ Windows specific implementation of NIO pipes is the
culprit. By itself HttpCore can take a fair amount of load. I ran a few
tests just to give you a rough indication as to where HttpCore stands in
term of performance and stability. Full disclaimer: I run Linux, not
Windows.
The NIO transport can process 200000 HTTP GET requests over 500
simultaneous connections delivering raw throughput of 1883 messages per
second. This is not stellar but okayish enough for a start
======================================================================
Server Software: Jakarta-HttpComponents-NIO/1.1
Server Hostname: localhost
Server Port: 8080
Document Path: /index.html
Document Length: 12926 bytes
Concurrency Level: 500
Time taken for tests: 106.191969 seconds
Complete requests: 200000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 200000
Total transferred: -1674510608 bytes
HTML transferred: -1708134128 bytes
Requests per second: 1883.38 [#/sec] (mean)
Time per request: 265.480 [ms] (mean)
Time per request: 0.531 [ms] (mean, across all concurrent
requests)
Transfer rate: -15399.13 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 44.8 0 3136
Processing: 215 262 26.4 250 418
Waiting: 6 131 74.5 130 410
Total: 215 264 53.0 250 3383
Percentage of the requests served within a certain time (ms)
50% 250
66% 286
75% 289
80% 291
90% 297
95% 300
98% 311
99% 326
100% 3383 (longest request)
======================================================================
For comparison HttpCore transport based on classic I/O performs twice as
fast. I never was a big fan of NIO and these numbers go to show why. NIO
starts paying off only when dealing with several thousands of
simultaneous connections, but I am digressing.
======================================================================
Server Software: Jakarta-HttpComponents/1.1
Server Hostname: localhost
Server Port: 8080
Document Path: /index.html
Document Length: 12926 bytes
Concurrency Level: 500
Time taken for tests: 56.582312 seconds
Complete requests: 200000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 200000
Total transferred: -1676941116 bytes
HTML transferred: -1709741444 bytes
Requests per second: 3534.67 [#/sec] (mean)
Time per request: 141.456 [ms] (mean)
Time per request: 0.283 [ms] (mean, across all concurrent
requests)
Transfer rate: -28942.56 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 29 829.7 0 45069
Processing: 0 93 632.8 43 31867
Waiting: 0 88 632.3 40 31867
Total: 0 123 1363.5 43 56477
Percentage of the requests served within a certain time (ms)
50% 43
66% 90
75% 111
80% 121
90% 157
95% 199
98% 275
99% 356
100% 56477 (longest request)
======================================================================
So, bottom line, it can well be NIO pipes on Windows simply do not scale
well. Maybe. So, at some point you might look at an option of replacing
NIO pipes with some custom code based on shared buffers provided in
HttpCore NIO extensions.
Cheers
Oleg
> ---------- Forwarded message ----------
> From: Asankha C. Perera <as...@wso2.com>
> Date: Mar 22, 2007 4:58 PM
> Subject: Re: [jira] Resolved: (HTTPCORE-60) Transport appears to be
> hanging because an unchecked exception caused the I/O dispatch thread
> to terminate
> To: HttpComponents Project <ht...@jakarta.apache.org>
>
> Oleg/Ant
>
> I am guessing this is something to do with Windows or the JDK you
> use.. But I am unable to test this week, so will try to my best to try
> this sometime next week. As I said, on Linux I have run the system
> through thousands of messages and multiple threads concurrently and
> have fixed all the issues I came across.
>
> So Oleg, I do not see this as a blocker for the HttpCore release - but
> I will use your latest snapshots in Synapse to check on this in future
> if it occurs again
>
> thanks
> asankha
>
> Oleg Kalnichevski (JIRA) wrote:
> > [ https://issues.apache.org/jira/browse/HTTPCORE-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> > ]
> >
> > Oleg Kalnichevski resolved HTTPCORE-60.
> > ---------------------------------------
> >
> > Resolution: Fixed
> >
> > Anthony
> > It turned out ClosedChannelException is a checked I/O exception so it cannot kill the I/O dispatch thread. So, apparently I was wrong in my initial assertion about the cause of the Synapse I/O transport lockup. I tweaked HttpCore code a little and changed the IOSessionImpl to catch all ChannelClosedException-s thrown by the underlying byte channel just in case.
> >
> >
> > Please review the changes and let me know if it is okay to proceed with the release
> >
> > Oleg
> >
> >
> > > Transport appears to be hanging because an unchecked exception caused the I/O dispatch thread to terminate
> > > ----------------------------------------------------------------------------------------------------------
> > >
> > >
> > > Key: HTTPCORE-60
> > > URL: https://issues.apache.org/jira/browse/HTTPCORE-60
> > >
> > > Project: HttpComponents Core
> > > Issue Type: Bug
> > > Affects Versions: 4.0-alpha4
> > > Reporter: ant elder
> > > Assigned To: Oleg Kalnichevski
> > > Fix For: 4.0-alpha4
> > >
> > >
> > >
> > > See discussion on synapse-dev mailing list: http://www.nabble.com/Intermittent-IO-Errors-using-Synapse-tf3439957.html
> > >
> > > The transport appears to be hanging because an unchecked exception
> > > caused the I/O dispatch thread to terminate. I believe there are several
> > > different types of problems (at least two) that we are seeing here.
> > >
> > > [I/O reactor worker thread 5] ERROR ServerHandler - I/O Error : null
> > >
> > > > java.nio.channels.ClosedChannelException
> > > > at
> > > > sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:112)
> > > > at
> > > > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:139)
> > > >
> > > >
> >
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> httpcomponents-dev-unsubscribe@jakarta.apache.org For additional
> commands, e-mail: httpcomponents-dev-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: synapse-dev-help@ws.apache.org
Re: Transport appears to be hanging because an unchecked exception
caused the I/O dispatch thread to terminate
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2007-03-26 at 13:16 +0530, Asankha C. Perera wrote:
> Hi Oleg
>
> Oleg Kalnichevski wrote:
> > The problem appears to be caused by Synapse opening an I/O pipe per
> > *every* incoming and outgoing HTTP message. On some platforms this can
> > be a very expensive operation both in terms of performance and system
> > resources. On Windows opening a I/O pipe apparently requires a local IP
> > port to be allocated. No wonder Synapse chokes only after a few thousand
> > of requests.
> >
> > I see absolutely no reason why Synapse should make use of I/O pipes.
> > Essentially pipes are being used to bridge event-driven NIO and stream
> > based classic IO. There are other ways to get the job done. A trivial
> > shared buffer with synchronized access should perfectly suffice. I'll
> > happily lend you a helping hand if necessary.
> >
> Could you help me a bit here.. The Pipe class seemed to let us do what
> we wanted - i.e. bridge streams to channels - without having to write
> our own code. Could you elaborate more on how you propose to get around
> using Pipes? or would you have a pointer to any code?
>
Hi Asankha,
This does require writing some custom code, but it think it is well
worth the trouble.
Basically all you need is an object with synchronized access to its
internal buffer, so it could be used by the I/O dispatch thread to
produce data and by the worker thread to consume data or the other way
around.
HttpCore NIO provides two interfaces for that end, which you may want to
take a starting point:
http://svn.apache.org/repos/asf/jakarta/httpcomponents/httpcore/trunk/module-nio/src/main/java/org/apache/http/nio/util/ContentInputBuffer.java
http://svn.apache.org/repos/asf/jakarta/httpcomponents/httpcore/trunk/module-nio/src/main/java/org/apache/http/nio/util/ContentOutputBuffer.java
There are also some concrete implementations of those interfaces
provided out of the box by HttpCore NIO.
http://svn.apache.org/repos/asf/jakarta/httpcomponents/httpcore/trunk/module-nio/src/main/java/org/apache/http/nio/util/SharedInputBuffer.java
http://svn.apache.org/repos/asf/jakarta/httpcomponents/httpcore/trunk/module-nio/src/main/java/org/apache/http/nio/util/SharedOutputBuffer.java
These shared buffer classes are pretty advanced, as they are capable of
throttling the frequency of I/O events to make sure the internal buffer
does not get overflown, that is, the worker thread can temporarily
suspend data input / output on the socket channel if it cannot keep up
with the I/O rate and take the time needed to do data processing and
free up more space in the shared buffers. This can help ensure that the
transport operates with a nearly constant memory footprint, so once the
connection is established and is fully initialized (content buffers
allocated and all) it will never go down due to out of memory condition.
There is hardly anything worse for an HTTP transport then dropping
connections while streaming out response body after already having sent
HTTP 200 OK back to the client.
You can take a look at the throttling version of the HTTP service
handler for an example of shared buffers in action.
http://svn.apache.org/repos/asf/jakarta/httpcomponents/httpcore/trunk/module-nio/src/main/java/org/apache/http/nio/protocol/ThrottlingHttpServiceHandler.java
BUT the bad news is there is nearly no test coverage for these classes
yet as I wanted to spend more time working on them during ALPHA5. This
code can certainly benefit from more testing.
So, you might want to start with a somewhat simpler custom
implementation for Synapse 1.0 that always expands the buffers whenever
more input / output is made available, and then consider making it more
sophisticated for 1.1. Just an idea.
Oleg
> thanks
> asankha
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: synapse-dev-help@ws.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: synapse-dev-help@ws.apache.org
Re: Transport appears to be hanging because an unchecked exception
caused the I/O dispatch thread to terminate
Posted by "Asankha C. Perera" <as...@wso2.com>.
Hi Oleg
Oleg Kalnichevski wrote:
> The problem appears to be caused by Synapse opening an I/O pipe per
> *every* incoming and outgoing HTTP message. On some platforms this can
> be a very expensive operation both in terms of performance and system
> resources. On Windows opening a I/O pipe apparently requires a local IP
> port to be allocated. No wonder Synapse chokes only after a few thousand
> of requests.
>
> I see absolutely no reason why Synapse should make use of I/O pipes.
> Essentially pipes are being used to bridge event-driven NIO and stream
> based classic IO. There are other ways to get the job done. A trivial
> shared buffer with synchronized access should perfectly suffice. I'll
> happily lend you a helping hand if necessary.
>
Could you help me a bit here.. The Pipe class seemed to let us do what
we wanted - i.e. bridge streams to channels - without having to write
our own code. Could you elaborate more on how you propose to get around
using Pipes? or would you have a pointer to any code?
thanks
asankha
---------------------------------------------------------------------
To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: synapse-dev-help@ws.apache.org
Re: Transport appears to be hanging because an unchecked exception
caused the I/O dispatch thread to terminate
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2007-03-25 at 09:43 +0000, ant elder wrote:
> The symptoms I get do seem to match what you describe. There's still
> two problems with that though which I'd like to understand better.
>
> 1) Why don't I see this with the non-NIO transport? For example I can
> run the Synapse server samples in either the Synapse sample server
> which uses the NIO transport, or I can just use a separate axis2-1.1.1
> distro with the non-NIO transport. When using JMeter against
> axis2-1.1.1 it works fine and i can send tens of thousands of requests
> without any errors. Whats different here, the underlying TCP stack and
> config is the same isn't it?
Anthony, Asankha, at al
The problem appears to be caused by Synapse opening an I/O pipe per
*every* incoming and outgoing HTTP message. On some platforms this can
be a very expensive operation both in terms of performance and system
resources. On Windows opening a I/O pipe apparently requires a local IP
port to be allocated. No wonder Synapse chokes only after a few thousand
of requests.
I see absolutely no reason why Synapse should make use of I/O pipes.
Essentially pipes are being used to bridge event-driven NIO and stream
based classic IO. There are other ways to get the job done. A trivial
shared buffer with synchronized access should perfectly suffice. I'll
happily lend you a helping hand if necessary.
> 2) Synapse often hangs after the IO error and needs to be restarted.
> Is there any way we can make it recover from this without requiring a
> restart? By handling the exception differently or something?
>
Please let me know if you see any unchecked exceptions thrown by I/O
reactors, as those exceptions cause I/O dispatch threads to terminate,
effectively locking up the I/O reactor.
Oleg
> ...ant
>
> On 3/24/07, Asankha C. Perera < asankha@wso2.com> wrote:
> Ant
>
> This is the same error seen by Indika on Windows.. and I think
> my analysis is correct. If you run the test for the first time
> or after a few minutes of running the test last, you should be
> able to go to around 1000 iterations. After you start to hit
> this issue, even 200 iterations would give you the error. At
> this time, doing a netstat -na should show you that most of
> the tcp ports are in TIME_WAIT state. Usually it could take at
> least one minute till a port is cleared up by the OS. The
> tuning parameters I specified for Linux tells the OS to use
> the full port range for applications, and to set the tcp fin
> timeout to 30 secs - to clear up the ports as quickly as
> possible. Without *any* OS tuning and on a Windows XP system -
> you definitely will encounter this issue.
>
>
> asankha
>
> ant elder wrote:
> > I've tried again with the latest Synapse and HTTP components
> > code and several JVMs. The results feel slightly different
> > than before but the end result is still always the root
> > exception included below. Sometime it doesn't occur till
> > around 1000 requests, but sometimes it happens after not
> > many requests at all.
> >
> > ...ant
> >
> > java.io.IOException: Unable to establish loopback connection
> > at sun.nio.ch.PipeImpl$Initializer.run(Unknown
> > Source)
> > at
> > java.security.AccessController.doPrivileged(Native Method)
> > at sun.nio.ch.PipeImpl.<init>(Unknown Source)
> > at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown
> > Source)
> > at java.nio.channels.Pipe.open(Unknown Source)
> > at
> > org.apache.axis2.transport.nhttp.ServerHandler.requestReceived (ServerHandler.java:108)
> > at
> > org.apache.axis2.transport.nhttp.LoggingNHttpServiceHandler.requestReceived(LoggingNHttpServiceHandler.java:83)
> > at
> > org.apache.http.impl.nio.DefaultNHttpServerConnection.consumeInput (DefaultNHttpServerConnection.java:96)
> > at
> > org.apache.axis2.transport.nhttp.PlainServerIOEventDispatch.inputReady(PlainServerIOEventDispatch.java:67)
> > at
> > org.apache.http.impl.nio.reactor.BaseIOReactor.readable
> > (BaseIOReactor.java:68)
> > at
> > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:160)
> > at
> > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java :145)
> > at
> > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:127)
> > at
> > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java :153)
> > at java.lang.Thread.run(Unknown Source)
> > Caused by: java.net.BindException: Address already in use:
> > connect
> > at sun.nio.ch.Net.connect(Native Method)
> > at sun.nio.ch.SocketChannelImpl.connect (Unknown
> > Source)
> > at java.nio.channels.SocketChannel.open(Unknown
> > Source)
> >
> > On 3/23/07, Asankha C. Perera <as...@wso2.com> wrote:
> > Ant
> >
> > I am quite sure that the problem seen by Indika now
> > was related to the ports being exhausted - see the
> > following articles and esp. the "MaxUserPort" and
> > "TcpTimedWaitDelay" parameters that could tweaked -
> > to be consistent with what I am using before running
> > a load test on Linux. I will ask Indika to check
> > these on Monday - but you may try this in the
> > meantime if you get a chance
> >
> > http://www.microsoft.com/technet/network/deploy/depovg/tcpip2k.mspx
> > http://www.microsoft.com/technet/community/columns/cableguy/cg1205.mspx
> > http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html
> >
> >
> > asankha
> >
> > Asankha C. Perera wrote:
> > > Hi Ant
> > >
> > > I fixed this for Linux and JDK 1.5 - I am
> > > confident of this fix as I was able to first
> > > recreate the issue consistently and then see the
> > > fix in action using 5 concurrent users sending a
> > > total of 5000 messages multiple times. However
> > > Indika is still seeing a 'similar' issue in
> > > Windows using JDK 1.4. We will try to see if its
> > > related to JDK 1.4 or Windows. If you get the
> > > latest nhttp code and build the nhttp JAR you
> > > could verify this fix - and let me know.
> > >
> > > I am listing some of the linux commands that came
> > > in handy for the resolution incase someone wants
> > > to check this.
> > >
> > > lsof -p 7426 => lists the open files for the pid
> > > given after the -p option
> > >
> > > ls -l /proc/9976/fd | wc -l => for each process
> > > the /proc filesystem lists the files used and thus
> > > you could count the open files with this command
> > >
> > > asankha
> > >
> > > Asankha C. Perera wrote:
> > > > Ant / Oleg
> > > >
> > > > I can recreate this issue on both Windows and
> > > > Linux and think its caused by my code related to
> > > > use of Pipes.. and I am actively looking into
> > > > this right now.. will get back to you on what I
> > > > find.
> > > >
> > > > asankha
> > > >
> > > > ant elder wrote:
> > > > > I've tried on several JDKs now and _always_
> > > > > get similar intermittent I/O related errors. I
> > > > > can use JMeter directly against Axis2-1.1.1
> > > > > without any problems at all, so this does look
> > > > > like some issue with the NIO transport. Be
> > > > > really good to hear from other Windows users
> > > > > to see if this is just my specific environment
> > > > > or a more general problem problem.
> > > > >
> > > > > To recreate:
> > > > >
> > > > > 1) build Synapse server sample by running
> > > > > 'ant' in the samples\axis2Server\src
> > > > > \SimpleStockQuoteService directory
> > > > > 2) start the sample service by running samples
> > > > > \axis2Server\axis2server.bat
> > > > > 3) get the Synapse config (either 8 or 501)
> > > > > from http://people.apache.org/~antelder/temp/,
> > > > > put in repository\conf\sample and start
> > > > > syanps: bin\synapse.bat -sample=8
> > > > > 4) get the JMeter config test1.jmx from
> > > > > http://people.apache.org/~antelder/temp/,
> > > > > start Jmeter and File -> Open and point to the
> > > > > test1.jmx file
> > > > > 5) JMeter Run -> Start and after not to long
> > > > > IO errors should appear in the Syanpse
> > > > > console
> > > > >
> > > > > ...ant
> > > > >
> > > > > ---------- Forwarded message ----------
> > > > > From: Asankha C. Perera <as...@wso2.com>
> > > > > Date: Mar 22, 2007 4:58 PM
> > > > > Subject: Re: [jira] Resolved: (HTTPCORE-60)
> > > > > Transport appears to be hanging because an
> > > > > unchecked exception caused the I/O dispatch
> > > > > thread to terminate
> > > > > To: HttpComponents Project
> > > > > <ht...@jakarta.apache.org>
> > > > >
> > > > > Oleg/Ant
> > > > >
> > > > > I am guessing this is something to do with
> > > > > Windows or the JDK you use.. But I am unable
> > > > > to test this week, so will try to my best to
> > > > > try this sometime next week. As I said, on
> > > > > Linux I have run the system through thousands
> > > > > of messages and multiple threads concurrently
> > > > > and have fixed all the issues I came across.
> > > > >
> > > > > So Oleg, I do not see this as a blocker for
> > > > > the HttpCore release - but I will use your
> > > > > latest snapshots in Synapse to check on this
> > > > > in future if it occurs again
> > > > >
> > > > > thanks
> > > > > asankha
> > > > >
> > > > > Oleg Kalnichevski (JIRA) wrote:
> > > > > > [
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/HTTPCORE-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> > > > > >
> > > > > > ]
> > > > > >
> > > > > > Oleg Kalnichevski resolved HTTPCORE-60.
> > > > > > ---------------------------------------
> > > > > >
> > > > > >
> > > > > >
> > > > > > Resolution: Fixed
> > > > > >
> > > > > > Anthony
> > > > > > It turned out ClosedChannelException is a checked I/O exception so it cannot kill the I/O dispatch thread. So, apparently I was wrong in my initial assertion about the cause of the Synapse I/O transport lockup. I tweaked HttpCore code a little and changed the IOSessionImpl to catch all ChannelClosedException-s thrown by the underlying byte channel just in case.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Please review the changes and let me know if it is okay to proceed with the release
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > >
> > > > > > > Transport appears to be hanging because an unchecked exception caused the I/O dispatch thread to terminate
> > > > > > > ----------------------------------------------------------------------------------------------------------
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Key: HTTPCORE-60
> > > > > > > URL: https://issues.apache.org/jira/browse/HTTPCORE-60
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Project: HttpComponents Core
> > > > > > > Issue Type: Bug
> > > > > > > Affects Versions: 4.0-alpha4
> > > > > > > Reporter: ant elder
> > > > > > > Assigned To: Oleg Kalnichevski
> > > > > > > Fix For: 4.0-alpha4
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > See discussion on synapse-dev mailing list: http://www.nabble.com/Intermittent-IO-Errors-using-Synapse-tf3439957.html
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The transport appears to be hanging because an unchecked exception
> > > > > > > caused the I/O dispatch thread to terminate. I believe there are several
> > > > > > > different types of problems (at least two) that we are seeing here.
> > > > > > >
> > > > > > > [I/O reactor worker thread 5] ERROR ServerHandler - I/O Error : null
> > > > > > >
> > > > > > > > java.nio.channels.ClosedChannelException
> > > > > > > > at
> > > > > > > > sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:112)
> > > > > > > > at
> > > > > > > > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java
> > > > > > > >
> > > > > > > >
> > > > > > > > :139)
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.org For additional commands, e-mail: httpcomponents-dev-help@jakarta.apache.org
> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For additional commands, e-mail: synapse-dev-help@ws.apache.org
> > > --------------------------------------------------------------------- To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For additional commands, e-mail: synapse-dev-help@ws.apache.org
> > --------------------------------------------------------------------- To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For additional commands, e-mail: synapse-dev-help@ws.apache.org
> >
> --------------------------------------------------------------------- To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For additional commands, e-mail: synapse-dev-help@ws.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: synapse-dev-help@ws.apache.org
Re: Transport appears to be hanging because an unchecked exception caused the I/O dispatch thread to terminate
Posted by ant elder <an...@gmail.com>.
The symptoms I get do seem to match what you describe. There's still two
problems with that though which I'd like to understand better.
1) Why don't I see this with the non-NIO transport? For example I can run
the Synapse server samples in either the Synapse sample server which uses
the NIO transport, or I can just use a separate axis2-1.1.1 distro with the
non-NIO transport. When using JMeter against axis2-1.1.1 it works fine and i
can send tens of thousands of requests without any errors. Whats different
here, the underlying TCP stack and config is the same isn't it?
2) Synapse often hangs after the IO error and needs to be restarted. Is
there any way we can make it recover from this without requiring a restart?
By handling the exception differently or something?
...ant
On 3/24/07, Asankha C. Perera < asankha@wso2.com> wrote:
>
> Ant
>
> This is the same error seen by Indika on Windows.. and I think my analysis
> is correct. If you run the test for the first time or after a few minutes of
> running the test last, you should be able to go to around 1000 iterations.
> After you start to hit this issue, even 200 iterations would give you the
> error. At this time, doing a netstat -na should show you that most of the
> tcp ports are in TIME_WAIT state. Usually it could take at least one minute
> till a port is cleared up by the OS. The tuning parameters I specified for
> Linux tells the OS to use the full port range for applications, and to set
> the tcp fin timeout to 30 secs - to clear up the ports as quickly as
> possible. Without *any* OS tuning and on a Windows XP system - you
> definitely will encounter this issue.
>
> asankha
>
> ant elder wrote:
>
> I've tried again with the latest Synapse and HTTP components code and
> several JVMs. The results feel slightly different than before but the end
> result is still always the root exception included below. Sometime it
> doesn't occur till around 1000 requests, but sometimes it happens after not
> many requests at all.
>
> ...ant
>
> java.io.IOException: Unable to establish loopback connection
> at sun.nio.ch.PipeImpl$Initializer.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.nio.ch.PipeImpl.<init>(Unknown Source)
> at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown Source)
> at java.nio.channels.Pipe.open(Unknown Source)
> at org.apache.axis2.transport.nhttp.ServerHandler.requestReceived(
> ServerHandler.java:108)
> at
> org.apache.axis2.transport.nhttp.LoggingNHttpServiceHandler.requestReceived
> (LoggingNHttpServiceHandler.java:83)
> at
> org.apache.http.impl.nio.DefaultNHttpServerConnection.consumeInput (
> DefaultNHttpServerConnection.java:96)
> at
> org.apache.axis2.transport.nhttp.PlainServerIOEventDispatch.inputReady(
> PlainServerIOEventDispatch.java:67)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable (
> BaseIOReactor.java:68)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent
> (AbstractIOReactor.java:160)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(
> AbstractIOReactor.java :145)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(
> AbstractIOReactor.java:127)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(
> AbstractMultiworkerIOReactor.java :153)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.net.BindException: Address already in use: connect
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect (Unknown Source)
> at java.nio.channels.SocketChannel.open(Unknown Source)
>
> On 3/23/07, Asankha C. Perera < asankha@wso2.com> wrote:
> >
> > Ant
> >
> > I am quite sure that the problem seen by Indika now was related to the
> > ports being exhausted - see the following articles and esp. the
> > "MaxUserPort" and "TcpTimedWaitDelay" parameters that could tweaked - to be
> > consistent with what I am using before running a load test on Linux. I will
> > ask Indika to check these on Monday - but you may try this in the meantime
> > if you get a chance
> >
> > http://www.microsoft.com/technet/network/deploy/depovg/tcpip2k.mspx
> > http://www.microsoft.com/technet/community/columns/cableguy/cg1205.mspx
> > http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html
> >
> >
> > asankha
> >
> > Asankha C. Perera wrote:
> >
> > Hi Ant
> >
> > I fixed this for Linux and JDK 1.5 - I am confident of this fix as I was
> > able to first recreate the issue consistently and then see the fix in action
> > using 5 concurrent users sending a total of 5000 messages multiple times.
> > However Indika is still seeing a 'similar' issue in Windows using JDK
> > 1.4. We will try to see if its related to JDK 1.4 or Windows. If you get
> > the latest nhttp code and build the nhttp JAR you could verify this fix -
> > and let me know.
> >
> > I am listing some of the linux commands that came in handy for the
> > resolution incase someone wants to check this.
> >
> > lsof -p 7426 => lists the open files for the pid given after the -p
> > option
> >
> > ls -l /proc/9976/fd | wc -l => for each process the /proc filesystem
> > lists the files used and thus you could count the open files with this
> > command
> >
> > asankha
> >
> > Asankha C. Perera wrote:
> >
> > Ant / Oleg
> >
> > I can recreate this issue on both Windows and Linux and think its caused
> > by my code related to use of Pipes.. and I am actively looking into this
> > right now.. will get back to you on what I find.
> >
> > asankha
> >
> > ant elder wrote:
> >
> > I've tried on several JDKs now and _always_ get similar intermittent I/O
> > related errors. I can use JMeter directly against Axis2-1.1.1 without
> > any problems at all, so this does look like some issue with the NIO
> > transport. Be really good to hear from other Windows users to see if this is
> > just my specific environment or a more general problem problem.
> >
> > To recreate:
> >
> > 1) build Synapse server sample by running 'ant' in the
> > samples\axis2Server\src\SimpleStockQuoteService directory
> > 2) start the sample service by running
> > samples\axis2Server\axis2server.bat
> > 3) get the Synapse config (either 8 or 501) from
> > http://people.apache.org/~antelder/temp/<http://people.apache.org/%7Eantelder/temp/>,
> > put in repository\conf\sample and start syanps: bin\synapse.bat -sample=8
> > 4) get the JMeter config test1.jmx from
> > http://people.apache.org/~antelder/temp/<http://people.apache.org/%7Eantelder/temp/>,
> > start Jmeter and File -> Open and point to the test1.jmx file
> > 5) JMeter Run -> Start and after not to long IO errors should appear in
> > the Syanpse console
> >
> > ...ant
> >
> > ---------- Forwarded message ----------
> > From: Asankha C. Perera <as...@wso2.com>
> > Date: Mar 22, 2007 4:58 PM
> > Subject: Re: [jira] Resolved: (HTTPCORE-60) Transport appears to be
> > hanging because an unchecked exception caused the I/O dispatch thread to
> > terminate
> > To: HttpComponents Project < httpcomponents-dev@jakarta.apache.org>
> >
> > Oleg/Ant
> >
> > I am guessing this is something to do with Windows or the JDK you use..
> > But I am unable to test this week, so will try to my best to try this
> > sometime next week. As I said, on Linux I have run the system through
> > thousands of messages and multiple threads concurrently and have fixed all
> > the issues I came across.
> >
> > So Oleg, I do not see this as a blocker for the HttpCore release - but I
> > will use your latest snapshots in Synapse to check on this in future if it
> > occurs again
> >
> > thanks
> > asankha
> >
> > Oleg Kalnichevski (JIRA) wrote:
> >
> > [
> > https://issues.apache.org/jira/browse/HTTPCORE-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> >
> > ]
> >
> > Oleg Kalnichevski resolved HTTPCORE-60.
> > ---------------------------------------
> >
> > Resolution: Fixed
> >
> > Anthony
> > It turned out ClosedChannelException is a checked I/O exception so it cannot kill the I/O dispatch thread. So, apparently I was wrong in my initial assertion about the cause of the Synapse I/O transport lockup. I tweaked HttpCore code a little and changed the IOSessionImpl to catch all ChannelClosedException-s thrown by the underlying byte channel just in case.
> >
> >
> >
> >
> > Please review the changes and let me know if it is okay to proceed with the release
> >
> > Oleg
> >
> > Transport appears to be hanging because an unchecked exception caused the I/O dispatch thread to terminate
> > ----------------------------------------------------------------------------------------------------------
> >
> >
> >
> >
> > Key: HTTPCORE-60
> > URL: https://issues.apache.org/jira/browse/HTTPCORE-60
> >
> >
> >
> > Project: HttpComponents Core
> > Issue Type: Bug
> > Affects Versions: 4.0-alpha4
> > Reporter: ant elder
> > Assigned To: Oleg Kalnichevski
> > Fix For: 4.0-alpha4
> >
> >
> >
> > See discussion on synapse-dev mailing list: http://www.nabble.com/Intermittent-IO-Errors-using-Synapse-tf3439957.html
> >
> >
> >
> > The transport appears to be hanging because an unchecked exception
> > caused the I/O dispatch thread to terminate. I believe there are several
> > different types of problems (at least two) that we are seeing here.
> >
> > [I/O reactor worker thread 5] ERROR ServerHandler - I/O Error : null
> >
> >
> > java.nio.channels.ClosedChannelException
> > at
> > sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:112)
> > at
> > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java
> >
> > :139)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> > httpcomponents-dev-unsubscribe@jakarta.apache.org For additional
> > commands, e-mail: httpcomponents-dev-help@jakarta.apache.org
> >
> > --------------------------------------------------------------------- To
> > unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For
> > additional commands, e-mail: synapse-dev-help@ws.apache.org
> >
> > --------------------------------------------------------------------- To
> > unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For
> > additional commands, e-mail: synapse-dev-help@ws.apache.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For
> > additional commands, e-mail: synapse-dev-help@ws.apache.org
> >
>
> --------------------------------------------------------------------- To
> unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For additional
> commands, e-mail: synapse-dev-help@ws.apache.org
>
Re: Transport appears to be hanging because an unchecked exception caused the I/O dispatch thread to terminate
Posted by ant elder <an...@gmail.com>.
I've tried again with the latest Synapse and HTTP components code and
several JVMs. The results feel slightly different than before but the end
result is still always the root exception included below. Sometime it
doesn't occur till around 1000 requests, but sometimes it happens after not
many requests at all.
...ant
java.io.IOException: Unable to establish loopback connection
at sun.nio.ch.PipeImpl$Initializer.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.<init>(Unknown Source)
at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown Source)
at java.nio.channels.Pipe.open(Unknown Source)
at org.apache.axis2.transport.nhttp.ServerHandler.requestReceived(
ServerHandler.java:108)
at
org.apache.axis2.transport.nhttp.LoggingNHttpServiceHandler.requestReceived(
LoggingNHttpServiceHandler.java:83)
at
org.apache.http.impl.nio.DefaultNHttpServerConnection.consumeInput(
DefaultNHttpServerConnection.java:96)
at
org.apache.axis2.transport.nhttp.PlainServerIOEventDispatch.inputReady(
PlainServerIOEventDispatch.java:67)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(
BaseIOReactor.java:68)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(
AbstractIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(
AbstractIOReactor.java:145)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(
AbstractIOReactor.java:127)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(
AbstractMultiworkerIOReactor.java:153)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.BindException: Address already in use: connect
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at java.nio.channels.SocketChannel.open(Unknown Source)
On 3/23/07, Asankha C. Perera <as...@wso2.com> wrote:
>
> Ant
>
> I am quite sure that the problem seen by Indika now was related to the
> ports being exhausted - see the following articles and esp. the
> "MaxUserPort" and "TcpTimedWaitDelay" parameters that could tweaked - to be
> consistent with what I am using before running a load test on Linux. I will
> ask Indika to check these on Monday - but you may try this in the meantime
> if you get a chance**
>
> http://www.microsoft.com/technet/network/deploy/depovg/tcpip2k.mspx
> http://www.microsoft.com/technet/community/columns/cableguy/cg1205.mspx
>
> http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html
>
> asankha
>
> Asankha C. Perera wrote:
>
> Hi Ant
>
> I fixed this for Linux and JDK 1.5 - I am confident of this fix as I was
> able to first recreate the issue consistently and then see the fix in action
> using 5 concurrent users sending a total of 5000 messages multiple times.
> However Indika is still seeing a 'similar' issue in Windows using JDK 1.4.
> We will try to see if its related to JDK 1.4 or Windows. If you get the
> latest nhttp code and build the nhttp JAR you could verify this fix - and
> let me know.
>
> I am listing some of the linux commands that came in handy for the
> resolution incase someone wants to check this.
>
> lsof -p 7426 => lists the open files for the pid given after the -p option
>
> ls -l /proc/9976/fd | wc -l => for each process the /proc filesystem lists
> the files used and thus you could count the open files with this command
>
> asankha
>
> Asankha C. Perera wrote:
>
> Ant / Oleg
>
> I can recreate this issue on both Windows and Linux and think its caused
> by my code related to use of Pipes.. and I am actively looking into this
> right now.. will get back to you on what I find.
>
> asankha
>
> ant elder wrote:
>
> I've tried on several JDKs now and _always_ get similar intermittent I/O
> related errors. I can use JMeter directly against Axis2-1.1.1 without any
> problems at all, so this does look like some issue with the NIO transport.
> Be really good to hear from other Windows users to see if this is just my
> specific environment or a more general problem problem.
>
> To recreate:
>
> 1) build Synapse server sample by running 'ant' in the
> samples\axis2Server\src\SimpleStockQuoteService directory
> 2) start the sample service by running samples\axis2Server\axis2server.bat
>
> 3) get the Synapse config (either 8 or 501) from
> http://people.apache.org/~antelder/temp/<http://people.apache.org/%7Eantelder/temp/>,
> put in repository\conf\sample and start syanps: bin\synapse.bat -sample=8
> 4) get the JMeter config test1.jmx from
> http://people.apache.org/~antelder/temp/<http://people.apache.org/%7Eantelder/temp/>,
> start Jmeter and File -> Open and point to the test1.jmx file
> 5) JMeter Run -> Start and after not to long IO errors should appear in
> the Syanpse console
>
> ...ant
>
> ---------- Forwarded message ----------
> From: Asankha C. Perera <as...@wso2.com>
> Date: Mar 22, 2007 4:58 PM
> Subject: Re: [jira] Resolved: (HTTPCORE-60) Transport appears to be
> hanging because an unchecked exception caused the I/O dispatch thread to
> terminate
> To: HttpComponents Project < httpcomponents-dev@jakarta.apache.org>
>
> Oleg/Ant
>
> I am guessing this is something to do with Windows or the JDK you use..
> But I am unable to test this week, so will try to my best to try this
> sometime next week. As I said, on Linux I have run the system through
> thousands of messages and multiple threads concurrently and have fixed all
> the issues I came across.
>
> So Oleg, I do not see this as a blocker for the HttpCore release - but I
> will use your latest snapshots in Synapse to check on this in future if it
> occurs again
>
> thanks
> asankha
>
> Oleg Kalnichevski (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/HTTPCORE-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Oleg Kalnichevski resolved HTTPCORE-60.
> ---------------------------------------
>
> Resolution: Fixed
>
> Anthony
> It turned out ClosedChannelException is a checked I/O exception so it cannot kill the I/O dispatch thread. So, apparently I was wrong in my initial assertion about the cause of the Synapse I/O transport lockup. I tweaked HttpCore code a little and changed the IOSessionImpl to catch all ChannelClosedException-s thrown by the underlying byte channel just in case.
>
>
> Please review the changes and let me know if it is okay to proceed with the release
>
> Oleg
>
> Transport appears to be hanging because an unchecked exception caused the I/O dispatch thread to terminate
> ----------------------------------------------------------------------------------------------------------
>
>
> Key: HTTPCORE-60
> URL: https://issues.apache.org/jira/browse/HTTPCORE-60
>
> Project: HttpComponents Core
> Issue Type: Bug
> Affects Versions: 4.0-alpha4
> Reporter: ant elder
> Assigned To: Oleg Kalnichevski
> Fix For: 4.0-alpha4
>
>
>
> See discussion on synapse-dev mailing list: http://www.nabble.com/Intermittent-IO-Errors-using-Synapse-tf3439957.html
>
> The transport appears to be hanging because an unchecked exception
> caused the I/O dispatch thread to terminate. I believe there are several
> different types of problems (at least two) that we are seeing here.
>
> [I/O reactor worker thread 5] ERROR ServerHandler - I/O Error : null
>
>
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:112)
> at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:139)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jakarta.apache.orgFor additional commands, e-mail:
> httpcomponents-dev-help@jakarta.apache.org
>
> --------------------------------------------------------------------- To
> unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For additional
> commands, e-mail: synapse-dev-help@ws.apache.org
>
> --------------------------------------------------------------------- To
> unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For additional
> commands, e-mail: synapse-dev-help@ws.apache.org
>
> --------------------------------------------------------------------- To
> unsubscribe, e-mail: synapse-dev-unsubscribe@ws.apache.org For additional
> commands, e-mail: synapse-dev-help@ws.apache.org
>