You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@synapse.apache.org by Hiranya Jayathilaka <hi...@gmail.com> on 2013/07/23 02:23:16 UTC
Migrating Synapse to Latest HTTP Core
Hi Folks,
Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Thread.java:680)
I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
Thanks,
Hiranya
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-07-24 at 11:04 -0700, Hiranya Jayathilaka wrote:
> Hi Oleg,
>
> Thanks for your great insights. Now I'm able to get all the unit tests and integration tests to run successfully.
>
> BTW do you have any idea as to what changed between 4.1.x and 4.2.x to cause this behavior? Our test suites work fine on current Synapse trunk which is based on HTTP Core 4.1.4.
>
Hi Hiranya
Prior to 4.2 HttpCore did not explicitly set SO_REUSEADDRESS parameter
on server sockets and relied on platform specific default behavior.
Oleg
> Also do you think it's a good idea to make SoReuseAddress=true the default in Synapse? Do you see any potential issues with that?
>
> Thanks,
> Hiranya
>
> On Jul 24, 2013, at 5:49 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > On Wed, 2013-07-24 at 13:45 +0200, Oleg Kalnichevski wrote:
> >> On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
> >>> Hi again,
> >>>
> >>> On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
> >>> <hi...@gmail.com> wrote:
> >
> > ...
> >
> >>>
> >>> I've also noticed that the problem doesn't occur if we do not send any
> >>> messages in the 1st round. I verified this behavior with Synapse too.
> >>> If a test case doesn't send any messages to Synapse, the test case
> >>> that follows will run without any issues.
> >>>
> >>
> >> Hiranya
> >>
> >> (1) The socket is CLOSE_WAIT state is from the client connection. It has
> >> nothing to do with the problem and can be gotten rid of by shutting down
> >> the client connection pool.
> >>
> >> client.getConnectionManager().shutdown();
> >>
> >> (2) The 'Address already in use' problem goes away if the server is
> >> configured to re-use socket address
> >>
> >> reactorConfig.setSoReuseAddress(true);
> >>
> >> I think this option should be activated for integration tests running
> >> inside the same JVM anyway, but I'll investigate why the address remains
> >> bound even after the listener has been shut down. I am not yet sure
> >> whether this is a bug or an expected behavior.
> >>
> >> Oleg
> >>
> >>
> >
> > Just a quick follow-up. The problem appears to be Linux specific. The
> > same test app works fine for me on Windows XP.
> >
> > I also modified slightly your code in order to have a finer control over
> > the client side.
> > ---
> > System.out.println("Round 1");
> > initServer();
> >
> > DefaultHttpClientConnection conn = new DefaultHttpClientConnection();
> > try {
> > Socket socket = new Socket("localhost", 8280);
> > conn.bind(socket, new BasicHttpParams());
> > conn.setSocketTimeout(1000);
> > conn.sendRequestHeader(new BasicHttpRequest("GET", "/"));
> > conn.flush();
> > conn.receiveResponseHeader();
> > } catch (SocketTimeoutException ex) {
> > } finally {
> > conn.close();
> > // this closes the socket the connection is bound to
> > }
> >
> > System.out.println("Waiting - Shutdown in 5 secs");
> > Thread.sleep(5000);
> > shutdown();
> >
> > System.out.println("\n\nRound 2");
> > initServer();
> > System.out.println("Waiting - Shutdown in 5 secs");
> > Thread.sleep(5000);
> > shutdown();
> > ---
> >
> > If you just comment out the conn#receiveResponseHeader() statement the
> > client side socket will never be put into read mode, and (surprise,
> > surprise) the listener address will no longer be stuck in the bound
> > state.
> >
> > I am neither a Linux nor a TCP/IP specialist. I cannot tell why this is
> > happening. However, by the look of it, one must use 'SO_REUSE_ADDRESS'
> > option on Linux in order to be able to shut down and re-open the
> > listener on the same address if the socket bound to that address had
> > some data passing through it.
> >
> > Anyone happens to have an idea?
> >
> > Oleg
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> > For additional commands, e-mail: dev-help@synapse.apache.org
> >
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-07-24 at 11:04 -0700, Hiranya Jayathilaka wrote:
> Hi Oleg,
>
> Thanks for your great insights. Now I'm able to get all the unit tests and integration tests to run successfully.
>
> BTW do you have any idea as to what changed between 4.1.x and 4.2.x to cause this behavior? Our test suites work fine on current Synapse trunk which is based on HTTP Core 4.1.4.
>
Hi Hiranya
Prior to 4.2 HttpCore did not explicitly set SO_REUSEADDRESS parameter
on server sockets and relied on platform specific default behavior.
Oleg
> Also do you think it's a good idea to make SoReuseAddress=true the default in Synapse? Do you see any potential issues with that?
>
> Thanks,
> Hiranya
>
> On Jul 24, 2013, at 5:49 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > On Wed, 2013-07-24 at 13:45 +0200, Oleg Kalnichevski wrote:
> >> On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
> >>> Hi again,
> >>>
> >>> On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
> >>> <hi...@gmail.com> wrote:
> >
> > ...
> >
> >>>
> >>> I've also noticed that the problem doesn't occur if we do not send any
> >>> messages in the 1st round. I verified this behavior with Synapse too.
> >>> If a test case doesn't send any messages to Synapse, the test case
> >>> that follows will run without any issues.
> >>>
> >>
> >> Hiranya
> >>
> >> (1) The socket is CLOSE_WAIT state is from the client connection. It has
> >> nothing to do with the problem and can be gotten rid of by shutting down
> >> the client connection pool.
> >>
> >> client.getConnectionManager().shutdown();
> >>
> >> (2) The 'Address already in use' problem goes away if the server is
> >> configured to re-use socket address
> >>
> >> reactorConfig.setSoReuseAddress(true);
> >>
> >> I think this option should be activated for integration tests running
> >> inside the same JVM anyway, but I'll investigate why the address remains
> >> bound even after the listener has been shut down. I am not yet sure
> >> whether this is a bug or an expected behavior.
> >>
> >> Oleg
> >>
> >>
> >
> > Just a quick follow-up. The problem appears to be Linux specific. The
> > same test app works fine for me on Windows XP.
> >
> > I also modified slightly your code in order to have a finer control over
> > the client side.
> > ---
> > System.out.println("Round 1");
> > initServer();
> >
> > DefaultHttpClientConnection conn = new DefaultHttpClientConnection();
> > try {
> > Socket socket = new Socket("localhost", 8280);
> > conn.bind(socket, new BasicHttpParams());
> > conn.setSocketTimeout(1000);
> > conn.sendRequestHeader(new BasicHttpRequest("GET", "/"));
> > conn.flush();
> > conn.receiveResponseHeader();
> > } catch (SocketTimeoutException ex) {
> > } finally {
> > conn.close();
> > // this closes the socket the connection is bound to
> > }
> >
> > System.out.println("Waiting - Shutdown in 5 secs");
> > Thread.sleep(5000);
> > shutdown();
> >
> > System.out.println("\n\nRound 2");
> > initServer();
> > System.out.println("Waiting - Shutdown in 5 secs");
> > Thread.sleep(5000);
> > shutdown();
> > ---
> >
> > If you just comment out the conn#receiveResponseHeader() statement the
> > client side socket will never be put into read mode, and (surprise,
> > surprise) the listener address will no longer be stuck in the bound
> > state.
> >
> > I am neither a Linux nor a TCP/IP specialist. I cannot tell why this is
> > happening. However, by the look of it, one must use 'SO_REUSE_ADDRESS'
> > option on Linux in order to be able to shut down and re-open the
> > listener on the same address if the socket bound to that address had
> > some data passing through it.
> >
> > Anyone happens to have an idea?
> >
> > Oleg
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> > For additional commands, e-mail: dev-help@synapse.apache.org
> >
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi Oleg,
Thanks for your great insights. Now I'm able to get all the unit tests and integration tests to run successfully.
BTW do you have any idea as to what changed between 4.1.x and 4.2.x to cause this behavior? Our test suites work fine on current Synapse trunk which is based on HTTP Core 4.1.4.
Also do you think it's a good idea to make SoReuseAddress=true the default in Synapse? Do you see any potential issues with that?
Thanks,
Hiranya
On Jul 24, 2013, at 5:49 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2013-07-24 at 13:45 +0200, Oleg Kalnichevski wrote:
>> On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
>>> Hi again,
>>>
>>> On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
>>> <hi...@gmail.com> wrote:
>
> ...
>
>>>
>>> I've also noticed that the problem doesn't occur if we do not send any
>>> messages in the 1st round. I verified this behavior with Synapse too.
>>> If a test case doesn't send any messages to Synapse, the test case
>>> that follows will run without any issues.
>>>
>>
>> Hiranya
>>
>> (1) The socket is CLOSE_WAIT state is from the client connection. It has
>> nothing to do with the problem and can be gotten rid of by shutting down
>> the client connection pool.
>>
>> client.getConnectionManager().shutdown();
>>
>> (2) The 'Address already in use' problem goes away if the server is
>> configured to re-use socket address
>>
>> reactorConfig.setSoReuseAddress(true);
>>
>> I think this option should be activated for integration tests running
>> inside the same JVM anyway, but I'll investigate why the address remains
>> bound even after the listener has been shut down. I am not yet sure
>> whether this is a bug or an expected behavior.
>>
>> Oleg
>>
>>
>
> Just a quick follow-up. The problem appears to be Linux specific. The
> same test app works fine for me on Windows XP.
>
> I also modified slightly your code in order to have a finer control over
> the client side.
> ---
> System.out.println("Round 1");
> initServer();
>
> DefaultHttpClientConnection conn = new DefaultHttpClientConnection();
> try {
> Socket socket = new Socket("localhost", 8280);
> conn.bind(socket, new BasicHttpParams());
> conn.setSocketTimeout(1000);
> conn.sendRequestHeader(new BasicHttpRequest("GET", "/"));
> conn.flush();
> conn.receiveResponseHeader();
> } catch (SocketTimeoutException ex) {
> } finally {
> conn.close();
> // this closes the socket the connection is bound to
> }
>
> System.out.println("Waiting - Shutdown in 5 secs");
> Thread.sleep(5000);
> shutdown();
>
> System.out.println("\n\nRound 2");
> initServer();
> System.out.println("Waiting - Shutdown in 5 secs");
> Thread.sleep(5000);
> shutdown();
> ---
>
> If you just comment out the conn#receiveResponseHeader() statement the
> client side socket will never be put into read mode, and (surprise,
> surprise) the listener address will no longer be stuck in the bound
> state.
>
> I am neither a Linux nor a TCP/IP specialist. I cannot tell why this is
> happening. However, by the look of it, one must use 'SO_REUSE_ADDRESS'
> option on Linux in order to be able to shut down and re-open the
> listener on the same address if the socket bound to that address had
> some data passing through it.
>
> Anyone happens to have an idea?
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
On Jul 24, 2013, at 5:49 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2013-07-24 at 13:45 +0200, Oleg Kalnichevski wrote:
>> On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
>>> Hi again,
>>>
>>> On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
>>> <hi...@gmail.com> wrote:
>
> ...
>
>>>
>>> I've also noticed that the problem doesn't occur if we do not send any
>>> messages in the 1st round. I verified this behavior with Synapse too.
>>> If a test case doesn't send any messages to Synapse, the test case
>>> that follows will run without any issues.
>>>
>>
>> Hiranya
>>
>> (1) The socket is CLOSE_WAIT state is from the client connection. It has
>> nothing to do with the problem and can be gotten rid of by shutting down
>> the client connection pool.
>>
>> client.getConnectionManager().shutdown();
>>
>> (2) The 'Address already in use' problem goes away if the server is
>> configured to re-use socket address
>>
>> reactorConfig.setSoReuseAddress(true);
>>
>> I think this option should be activated for integration tests running
>> inside the same JVM anyway, but I'll investigate why the address remains
>> bound even after the listener has been shut down. I am not yet sure
>> whether this is a bug or an expected behavior.
>>
>> Oleg
>>
>>
>
> Just a quick follow-up. The problem appears to be Linux specific. The
> same test app works fine for me on Windows XP.
>
> I also modified slightly your code in order to have a finer control over
> the client side.
> ---
> System.out.println("Round 1");
> initServer();
>
> DefaultHttpClientConnection conn = new DefaultHttpClientConnection();
> try {
> Socket socket = new Socket("localhost", 8280);
> conn.bind(socket, new BasicHttpParams());
> conn.setSocketTimeout(1000);
> conn.sendRequestHeader(new BasicHttpRequest("GET", "/"));
> conn.flush();
> conn.receiveResponseHeader();
> } catch (SocketTimeoutException ex) {
> } finally {
> conn.close();
> // this closes the socket the connection is bound to
> }
>
> System.out.println("Waiting - Shutdown in 5 secs");
> Thread.sleep(5000);
> shutdown();
>
> System.out.println("\n\nRound 2");
> initServer();
> System.out.println("Waiting - Shutdown in 5 secs");
> Thread.sleep(5000);
> shutdown();
> ---
>
> If you just comment out the conn#receiveResponseHeader() statement the
> client side socket will never be put into read mode, and (surprise,
> surprise) the listener address will no longer be stuck in the bound
> state.
>
> I am neither a Linux nor a TCP/IP specialist. I cannot tell why this is
> happening. However, by the look of it, one must use 'SO_REUSE_ADDRESS'
> option on Linux in order to be able to shut down and re-open the
> listener on the same address if the socket bound to that address had
> some data passing through it.
>
> Anyone happens to have an idea?
I came across [1] which describes the semantics of SO_REUSEADDR in great detail. And yes, the behavior is very much OS dependent (the article explains why our test case works in Windows without any issues).
Thanks,
Hiranya
[1] - http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
On Jul 24, 2013, at 5:49 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2013-07-24 at 13:45 +0200, Oleg Kalnichevski wrote:
>> On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
>>> Hi again,
>>>
>>> On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
>>> <hi...@gmail.com> wrote:
>
> ...
>
>>>
>>> I've also noticed that the problem doesn't occur if we do not send any
>>> messages in the 1st round. I verified this behavior with Synapse too.
>>> If a test case doesn't send any messages to Synapse, the test case
>>> that follows will run without any issues.
>>>
>>
>> Hiranya
>>
>> (1) The socket is CLOSE_WAIT state is from the client connection. It has
>> nothing to do with the problem and can be gotten rid of by shutting down
>> the client connection pool.
>>
>> client.getConnectionManager().shutdown();
>>
>> (2) The 'Address already in use' problem goes away if the server is
>> configured to re-use socket address
>>
>> reactorConfig.setSoReuseAddress(true);
>>
>> I think this option should be activated for integration tests running
>> inside the same JVM anyway, but I'll investigate why the address remains
>> bound even after the listener has been shut down. I am not yet sure
>> whether this is a bug or an expected behavior.
>>
>> Oleg
>>
>>
>
> Just a quick follow-up. The problem appears to be Linux specific. The
> same test app works fine for me on Windows XP.
>
> I also modified slightly your code in order to have a finer control over
> the client side.
> ---
> System.out.println("Round 1");
> initServer();
>
> DefaultHttpClientConnection conn = new DefaultHttpClientConnection();
> try {
> Socket socket = new Socket("localhost", 8280);
> conn.bind(socket, new BasicHttpParams());
> conn.setSocketTimeout(1000);
> conn.sendRequestHeader(new BasicHttpRequest("GET", "/"));
> conn.flush();
> conn.receiveResponseHeader();
> } catch (SocketTimeoutException ex) {
> } finally {
> conn.close();
> // this closes the socket the connection is bound to
> }
>
> System.out.println("Waiting - Shutdown in 5 secs");
> Thread.sleep(5000);
> shutdown();
>
> System.out.println("\n\nRound 2");
> initServer();
> System.out.println("Waiting - Shutdown in 5 secs");
> Thread.sleep(5000);
> shutdown();
> ---
>
> If you just comment out the conn#receiveResponseHeader() statement the
> client side socket will never be put into read mode, and (surprise,
> surprise) the listener address will no longer be stuck in the bound
> state.
>
> I am neither a Linux nor a TCP/IP specialist. I cannot tell why this is
> happening. However, by the look of it, one must use 'SO_REUSE_ADDRESS'
> option on Linux in order to be able to shut down and re-open the
> listener on the same address if the socket bound to that address had
> some data passing through it.
>
> Anyone happens to have an idea?
I came across [1] which describes the semantics of SO_REUSEADDR in great detail. And yes, the behavior is very much OS dependent (the article explains why our test case works in Windows without any issues).
Thanks,
Hiranya
[1] - http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi Oleg,
Thanks for your great insights. Now I'm able to get all the unit tests and integration tests to run successfully.
BTW do you have any idea as to what changed between 4.1.x and 4.2.x to cause this behavior? Our test suites work fine on current Synapse trunk which is based on HTTP Core 4.1.4.
Also do you think it's a good idea to make SoReuseAddress=true the default in Synapse? Do you see any potential issues with that?
Thanks,
Hiranya
On Jul 24, 2013, at 5:49 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2013-07-24 at 13:45 +0200, Oleg Kalnichevski wrote:
>> On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
>>> Hi again,
>>>
>>> On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
>>> <hi...@gmail.com> wrote:
>
> ...
>
>>>
>>> I've also noticed that the problem doesn't occur if we do not send any
>>> messages in the 1st round. I verified this behavior with Synapse too.
>>> If a test case doesn't send any messages to Synapse, the test case
>>> that follows will run without any issues.
>>>
>>
>> Hiranya
>>
>> (1) The socket is CLOSE_WAIT state is from the client connection. It has
>> nothing to do with the problem and can be gotten rid of by shutting down
>> the client connection pool.
>>
>> client.getConnectionManager().shutdown();
>>
>> (2) The 'Address already in use' problem goes away if the server is
>> configured to re-use socket address
>>
>> reactorConfig.setSoReuseAddress(true);
>>
>> I think this option should be activated for integration tests running
>> inside the same JVM anyway, but I'll investigate why the address remains
>> bound even after the listener has been shut down. I am not yet sure
>> whether this is a bug or an expected behavior.
>>
>> Oleg
>>
>>
>
> Just a quick follow-up. The problem appears to be Linux specific. The
> same test app works fine for me on Windows XP.
>
> I also modified slightly your code in order to have a finer control over
> the client side.
> ---
> System.out.println("Round 1");
> initServer();
>
> DefaultHttpClientConnection conn = new DefaultHttpClientConnection();
> try {
> Socket socket = new Socket("localhost", 8280);
> conn.bind(socket, new BasicHttpParams());
> conn.setSocketTimeout(1000);
> conn.sendRequestHeader(new BasicHttpRequest("GET", "/"));
> conn.flush();
> conn.receiveResponseHeader();
> } catch (SocketTimeoutException ex) {
> } finally {
> conn.close();
> // this closes the socket the connection is bound to
> }
>
> System.out.println("Waiting - Shutdown in 5 secs");
> Thread.sleep(5000);
> shutdown();
>
> System.out.println("\n\nRound 2");
> initServer();
> System.out.println("Waiting - Shutdown in 5 secs");
> Thread.sleep(5000);
> shutdown();
> ---
>
> If you just comment out the conn#receiveResponseHeader() statement the
> client side socket will never be put into read mode, and (surprise,
> surprise) the listener address will no longer be stuck in the bound
> state.
>
> I am neither a Linux nor a TCP/IP specialist. I cannot tell why this is
> happening. However, by the look of it, one must use 'SO_REUSE_ADDRESS'
> option on Linux in order to be able to shut down and re-open the
> listener on the same address if the socket bound to that address had
> some data passing through it.
>
> Anyone happens to have an idea?
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-07-24 at 13:45 +0200, Oleg Kalnichevski wrote:
> On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
> > Hi again,
> >
> > On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
> > <hi...@gmail.com> wrote:
...
> >
> > I've also noticed that the problem doesn't occur if we do not send any
> > messages in the 1st round. I verified this behavior with Synapse too.
> > If a test case doesn't send any messages to Synapse, the test case
> > that follows will run without any issues.
> >
>
> Hiranya
>
> (1) The socket is CLOSE_WAIT state is from the client connection. It has
> nothing to do with the problem and can be gotten rid of by shutting down
> the client connection pool.
>
> client.getConnectionManager().shutdown();
>
> (2) The 'Address already in use' problem goes away if the server is
> configured to re-use socket address
>
> reactorConfig.setSoReuseAddress(true);
>
> I think this option should be activated for integration tests running
> inside the same JVM anyway, but I'll investigate why the address remains
> bound even after the listener has been shut down. I am not yet sure
> whether this is a bug or an expected behavior.
>
> Oleg
>
>
Just a quick follow-up. The problem appears to be Linux specific. The
same test app works fine for me on Windows XP.
I also modified slightly your code in order to have a finer control over
the client side.
---
System.out.println("Round 1");
initServer();
DefaultHttpClientConnection conn = new DefaultHttpClientConnection();
try {
Socket socket = new Socket("localhost", 8280);
conn.bind(socket, new BasicHttpParams());
conn.setSocketTimeout(1000);
conn.sendRequestHeader(new BasicHttpRequest("GET", "/"));
conn.flush();
conn.receiveResponseHeader();
} catch (SocketTimeoutException ex) {
} finally {
conn.close();
// this closes the socket the connection is bound to
}
System.out.println("Waiting - Shutdown in 5 secs");
Thread.sleep(5000);
shutdown();
System.out.println("\n\nRound 2");
initServer();
System.out.println("Waiting - Shutdown in 5 secs");
Thread.sleep(5000);
shutdown();
---
If you just comment out the conn#receiveResponseHeader() statement the
client side socket will never be put into read mode, and (surprise,
surprise) the listener address will no longer be stuck in the bound
state.
I am neither a Linux nor a TCP/IP specialist. I cannot tell why this is
happening. However, by the look of it, one must use 'SO_REUSE_ADDRESS'
option on Linux in order to be able to shut down and re-open the
listener on the same address if the socket bound to that address had
some data passing through it.
Anyone happens to have an idea?
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-07-24 at 13:45 +0200, Oleg Kalnichevski wrote:
> On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
> > Hi again,
> >
> > On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
> > <hi...@gmail.com> wrote:
...
> >
> > I've also noticed that the problem doesn't occur if we do not send any
> > messages in the 1st round. I verified this behavior with Synapse too.
> > If a test case doesn't send any messages to Synapse, the test case
> > that follows will run without any issues.
> >
>
> Hiranya
>
> (1) The socket is CLOSE_WAIT state is from the client connection. It has
> nothing to do with the problem and can be gotten rid of by shutting down
> the client connection pool.
>
> client.getConnectionManager().shutdown();
>
> (2) The 'Address already in use' problem goes away if the server is
> configured to re-use socket address
>
> reactorConfig.setSoReuseAddress(true);
>
> I think this option should be activated for integration tests running
> inside the same JVM anyway, but I'll investigate why the address remains
> bound even after the listener has been shut down. I am not yet sure
> whether this is a bug or an expected behavior.
>
> Oleg
>
>
Just a quick follow-up. The problem appears to be Linux specific. The
same test app works fine for me on Windows XP.
I also modified slightly your code in order to have a finer control over
the client side.
---
System.out.println("Round 1");
initServer();
DefaultHttpClientConnection conn = new DefaultHttpClientConnection();
try {
Socket socket = new Socket("localhost", 8280);
conn.bind(socket, new BasicHttpParams());
conn.setSocketTimeout(1000);
conn.sendRequestHeader(new BasicHttpRequest("GET", "/"));
conn.flush();
conn.receiveResponseHeader();
} catch (SocketTimeoutException ex) {
} finally {
conn.close();
// this closes the socket the connection is bound to
}
System.out.println("Waiting - Shutdown in 5 secs");
Thread.sleep(5000);
shutdown();
System.out.println("\n\nRound 2");
initServer();
System.out.println("Waiting - Shutdown in 5 secs");
Thread.sleep(5000);
shutdown();
---
If you just comment out the conn#receiveResponseHeader() statement the
client side socket will never be put into read mode, and (surprise,
surprise) the listener address will no longer be stuck in the bound
state.
I am neither a Linux nor a TCP/IP specialist. I cannot tell why this is
happening. However, by the look of it, one must use 'SO_REUSE_ADDRESS'
option on Linux in order to be able to shut down and re-open the
listener on the same address if the socket bound to that address had
some data passing through it.
Anyone happens to have an idea?
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
> Hi again,
>
> On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
> <hi...@gmail.com> wrote:
>
> > Hi Oleg,
> >
> >
> > I can reproduce the problem with the attached JUnit test case. This
> > test case doesn't use any Axis2/Synapse related code. It only
> > requires HTTP Core and HTTP client. I used HTTP Core 4.2.4 for
> > testing.
> >
> >
> > The test case starts a ListeningIOReactor, does a simple HTTP
> > request-response exchange and then terminates the reactor. Then
> > after a couple of seconds, it tries to start the IO reactor again
> > and the exception occurs. Here's the full output:
> >
> >
> > Round 1
> > Listener started on port: 8280
> > Connected
> > Request received
> > Response submitted
> > Waiting - Shutdown in 5 secs
> > Shutdown
> >
> >
> > … wait 2 seconds...
> >
> >
> > Round 2
> > Listener started on port: 8280
> > Waiting - Shutdown in 5 secs
> > java.net.BindException: Address already in use
> > at sun.nio.ch.Net.bind(Native Method)
> > at
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
> > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
> > at
> > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
> > at
> > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
> > at
> > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
> > at org.apache.synapse.transport.nhttp.HttpCoreTest
> > $3.run(HttpCoreTest.java:101)
> > Shutdown
>
>
> I've also noticed that the problem doesn't occur if we do not send any
> messages in the 1st round. I verified this behavior with Synapse too.
> If a test case doesn't send any messages to Synapse, the test case
> that follows will run without any issues.
>
Hiranya
(1) The socket is CLOSE_WAIT state is from the client connection. It has
nothing to do with the problem and can be gotten rid of by shutting down
the client connection pool.
client.getConnectionManager().shutdown();
(2) The 'Address already in use' problem goes away if the server is
configured to re-use socket address
reactorConfig.setSoReuseAddress(true);
I think this option should be activated for integration tests running
inside the same JVM anyway, but I'll investigate why the address remains
bound even after the listener has been shut down. I am not yet sure
whether this is a bug or an expected behavior.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-07-24 at 01:40 -0700, Hiranya Jayathilaka wrote:
> Hi again,
>
> On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka
> <hi...@gmail.com> wrote:
>
> > Hi Oleg,
> >
> >
> > I can reproduce the problem with the attached JUnit test case. This
> > test case doesn't use any Axis2/Synapse related code. It only
> > requires HTTP Core and HTTP client. I used HTTP Core 4.2.4 for
> > testing.
> >
> >
> > The test case starts a ListeningIOReactor, does a simple HTTP
> > request-response exchange and then terminates the reactor. Then
> > after a couple of seconds, it tries to start the IO reactor again
> > and the exception occurs. Here's the full output:
> >
> >
> > Round 1
> > Listener started on port: 8280
> > Connected
> > Request received
> > Response submitted
> > Waiting - Shutdown in 5 secs
> > Shutdown
> >
> >
> > … wait 2 seconds...
> >
> >
> > Round 2
> > Listener started on port: 8280
> > Waiting - Shutdown in 5 secs
> > java.net.BindException: Address already in use
> > at sun.nio.ch.Net.bind(Native Method)
> > at
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
> > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
> > at
> > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
> > at
> > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
> > at
> > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
> > at org.apache.synapse.transport.nhttp.HttpCoreTest
> > $3.run(HttpCoreTest.java:101)
> > Shutdown
>
>
> I've also noticed that the problem doesn't occur if we do not send any
> messages in the 1st round. I verified this behavior with Synapse too.
> If a test case doesn't send any messages to Synapse, the test case
> that follows will run without any issues.
>
Hiranya
(1) The socket is CLOSE_WAIT state is from the client connection. It has
nothing to do with the problem and can be gotten rid of by shutting down
the client connection pool.
client.getConnectionManager().shutdown();
(2) The 'Address already in use' problem goes away if the server is
configured to re-use socket address
reactorConfig.setSoReuseAddress(true);
I think this option should be activated for integration tests running
inside the same JVM anyway, but I'll investigate why the address remains
bound even after the listener has been shut down. I am not yet sure
whether this is a bug or an expected behavior.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi again,
On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> Hi Oleg,
>
> I can reproduce the problem with the attached JUnit test case. This test case doesn't use any Axis2/Synapse related code. It only requires HTTP Core and HTTP client. I used HTTP Core 4.2.4 for testing.
>
> The test case starts a ListeningIOReactor, does a simple HTTP request-response exchange and then terminates the reactor. Then after a couple of seconds, it tries to start the IO reactor again and the exception occurs. Here's the full output:
>
> Round 1
> Listener started on port: 8280
> Connected
> Request received
> Response submitted
> Waiting - Shutdown in 5 secs
> Shutdown
>
> … wait 2 seconds...
>
> Round 2
> Listener started on port: 8280
> Waiting - Shutdown in 5 secs
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
> at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
> at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
> at org.apache.synapse.transport.nhttp.HttpCoreTest$3.run(HttpCoreTest.java:101)
> Shutdown
I've also noticed that the problem doesn't occur if we do not send any messages in the 1st round. I verified this behavior with Synapse too. If a test case doesn't send any messages to Synapse, the test case that follows will run without any issues.
Thanks,
Hiranya
>
> Thanks,
> Hiranya
>
> <HttpCoreTest.java>
>
> On Jul 24, 2013, at 12:17 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
>> On Tue, 2013-07-23 at 20:24 -0700, Hiranya Jayathilaka wrote:
>>> It looks like the socket goes into the CLOSE_WAIT state after shutting
>>> down the IO reactor and remains there until the JVM it self is killed.
>>> Since all automated test cases are executed within a single JVM
>>> instance, this issue continues to occur. Looks like something is not
>>> getting cleaned up properly with the latest HTTP Core.
>>>
>>>
>>> Thanks,
>>> Hiranya
>>>
>>
>> Hiranya
>>
>> I need a more specific evidence of HttpCore doing something wrong. Could
>> you please reproduce the issue with a test app, preferably not dependent
>> on Synapse?
>>
>> Oleg
>>
>>> On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka
>>> <hi...@gmail.com> wrote:
>>>
>>>> Hi Oleg,
>>>>
>>>>
>>>> I've done the necessary code changes locally and it seems to work
>>>> fine. The HTTPS sender now works as expected. But when running
>>>> automated tests, I frequently get the following exception:
>>>>
>>>>
>>>> 34 [HttpCoreNIOListener] WARN
>>>> org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may
>>>> be unstable: IOReactor encountered a checked exception : Address
>>>> already in use
>>>> java.net.BindException: Address already in use
>>>> at sun.nio.ch.Net.bind(Native Method)
>>>> at
>>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
>>>> at
>>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>>>> at
>>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
>>>> at
>>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
>>>> at
>>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
>>>> at org.apache.synapse.transport.nhttp.HttpCoreNIOListener
>>>> $2.run(HttpCoreNIOListener.java:254)
>>>> at java.lang.Thread.run(Thread.java:680)
>>>>
>>>>
>>>> This is seen in both NHTTP unit tests and Synapse integration tests.
>>>> Any idea what is going on here? Synapse cleans up the IO reactor by
>>>> calling ioReactor.shutdown(). Is there anything else we need to do
>>>> to clean up the reactor properly?
>>>>
>>>>
>>>> Thanks,
>>>> Hiranya
>>>>
>>>>
>>>> On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka
>>>> <hi...@gmail.com> wrote:
>>>>
>>>>> HI Oleg,
>>>>>
>>>>> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>>>>>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri
>>>>>>> <ra...@gmail.com> wrote:
>>>>>>>
>>>>>>>> May be we should enable the SSl debug logs and see what's
>>>>>>>> going on ?
>>>>>>>
>>>>>>> I ran some more tests and it looks like the HTTPS sender
>>>>>>> doesn't work with HTTP Core 4.2. It just sends out the
>>>>>>> messages without any SSL security. I can even monitor the
>>>>>>> message in plain text using TCPMon. Looks like some SSL
>>>>>>> related code in the Synapse HTTPS sender doesn't work properly
>>>>>>> with the new HTTP Core.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hiranya
>>>>>>>
>>>>>>
>>>>>> Hiranya
>>>>>>
>>>>>> Have you migrated off the deprecated APIs to new 4.2 APIs?
>>>>>
>>>>>
>>>>> No not yet. I just tried changing the HTTP Core dependency version
>>>>> to begin with. I guess migrating off the deprecated APIs is the
>>>>> next logical step.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Hiranya
>>>>>
>>>>>>
>>>>>> Oleg
>>>>>>
>>>>>> PS: By the way, you should probably consider migrating straight
>>>>>> to 4.3
>>>>>> at this point.
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> Rajika
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka
>>>>>>>> <hi...@gmail.com> wrote:
>>>>>>>> Hi Folks,
>>>>>>>>
>>>>>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried
>>>>>>>> using 4.2 instead (just changed the version of the maven
>>>>>>>> dependency). Almost everything worked fine, except for a
>>>>>>>> couple of integration test failures. In both these tests
>>>>>>>> Synapse uses HTTPS to contact the backend services, so I
>>>>>>>> suppose that's where things went wrong. The backend server
>>>>>>>> threw the following exception in the process:
>>>>>>>>
>>>>>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O
>>>>>>>> dispatcher-1] ERROR ServerHandler I/O error: Unrecognized
>>>>>>>> SSL message, plaintext connection?
>>>>>>>> javax.net.ssl.SSLException: Unrecognized SSL message,
>>>>>>>> plaintext connection?
>>>>>>>> at
>>>>>>>> com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>>>>>>> at
>>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>>>>>>> at
>>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>>>>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>>>>>>> at
>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>>>>>>> at
>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>>>>>>> at
>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>>>>>> at java.lang.Thread.run(Thread.java:680)
>>>>>>>>
>>>>>>>> I also tried upgrading to the latest HTTP Core version
>>>>>>>> (4.2.4). With this version NHTTP unit tests failed and most
>>>>>>>> of the integration tests failed too. It turns out only the
>>>>>>>> very first integration test ran successfully. Everything
>>>>>>>> that followed threw a bind exception. Looks like something
>>>>>>>> isn't getting cleaned up properly (although the logs
>>>>>>>> indicate that Synapse transport listeners are shutting down
>>>>>>>> cleanly after each test case).
>>>>>>>>
>>>>>>>> Does anybody got an idea what's going on? Do we need to do
>>>>>>>> any code changes to migrate to the latest HTTP Core?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hiranya
>>>>>>>> --
>>>>>>>> Hiranya Jayathilaka
>>>>>>>> Mayhem Lab/RACE Lab;
>>>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Hiranya Jayathilaka
>>>>>>> Mayhem Lab/RACE Lab;
>>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>>>>>> For additional commands, e-mail: dev-help@hc.apache.org
>>>>>>
>>>>>
>>>>> --
>>>>> Hiranya Jayathilaka
>>>>> Mayhem Lab/RACE Lab;
>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>
>>>>>
>>>>
>>>> --
>>>> Hiranya Jayathilaka
>>>> Mayhem Lab/RACE Lab;
>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>
>>>>
>>>
>>> --
>>> Hiranya Jayathilaka
>>> Mayhem Lab/RACE Lab;
>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>> Blog: http://techfeast-hiranya.blogspot.com
>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi again,
On Jul 24, 2013, at 1:36 AM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> Hi Oleg,
>
> I can reproduce the problem with the attached JUnit test case. This test case doesn't use any Axis2/Synapse related code. It only requires HTTP Core and HTTP client. I used HTTP Core 4.2.4 for testing.
>
> The test case starts a ListeningIOReactor, does a simple HTTP request-response exchange and then terminates the reactor. Then after a couple of seconds, it tries to start the IO reactor again and the exception occurs. Here's the full output:
>
> Round 1
> Listener started on port: 8280
> Connected
> Request received
> Response submitted
> Waiting - Shutdown in 5 secs
> Shutdown
>
> … wait 2 seconds...
>
> Round 2
> Listener started on port: 8280
> Waiting - Shutdown in 5 secs
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
> at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
> at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
> at org.apache.synapse.transport.nhttp.HttpCoreTest$3.run(HttpCoreTest.java:101)
> Shutdown
I've also noticed that the problem doesn't occur if we do not send any messages in the 1st round. I verified this behavior with Synapse too. If a test case doesn't send any messages to Synapse, the test case that follows will run without any issues.
Thanks,
Hiranya
>
> Thanks,
> Hiranya
>
> <HttpCoreTest.java>
>
> On Jul 24, 2013, at 12:17 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
>> On Tue, 2013-07-23 at 20:24 -0700, Hiranya Jayathilaka wrote:
>>> It looks like the socket goes into the CLOSE_WAIT state after shutting
>>> down the IO reactor and remains there until the JVM it self is killed.
>>> Since all automated test cases are executed within a single JVM
>>> instance, this issue continues to occur. Looks like something is not
>>> getting cleaned up properly with the latest HTTP Core.
>>>
>>>
>>> Thanks,
>>> Hiranya
>>>
>>
>> Hiranya
>>
>> I need a more specific evidence of HttpCore doing something wrong. Could
>> you please reproduce the issue with a test app, preferably not dependent
>> on Synapse?
>>
>> Oleg
>>
>>> On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka
>>> <hi...@gmail.com> wrote:
>>>
>>>> Hi Oleg,
>>>>
>>>>
>>>> I've done the necessary code changes locally and it seems to work
>>>> fine. The HTTPS sender now works as expected. But when running
>>>> automated tests, I frequently get the following exception:
>>>>
>>>>
>>>> 34 [HttpCoreNIOListener] WARN
>>>> org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may
>>>> be unstable: IOReactor encountered a checked exception : Address
>>>> already in use
>>>> java.net.BindException: Address already in use
>>>> at sun.nio.ch.Net.bind(Native Method)
>>>> at
>>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
>>>> at
>>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>>>> at
>>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
>>>> at
>>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
>>>> at
>>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
>>>> at org.apache.synapse.transport.nhttp.HttpCoreNIOListener
>>>> $2.run(HttpCoreNIOListener.java:254)
>>>> at java.lang.Thread.run(Thread.java:680)
>>>>
>>>>
>>>> This is seen in both NHTTP unit tests and Synapse integration tests.
>>>> Any idea what is going on here? Synapse cleans up the IO reactor by
>>>> calling ioReactor.shutdown(). Is there anything else we need to do
>>>> to clean up the reactor properly?
>>>>
>>>>
>>>> Thanks,
>>>> Hiranya
>>>>
>>>>
>>>> On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka
>>>> <hi...@gmail.com> wrote:
>>>>
>>>>> HI Oleg,
>>>>>
>>>>> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>>>>>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri
>>>>>>> <ra...@gmail.com> wrote:
>>>>>>>
>>>>>>>> May be we should enable the SSl debug logs and see what's
>>>>>>>> going on ?
>>>>>>>
>>>>>>> I ran some more tests and it looks like the HTTPS sender
>>>>>>> doesn't work with HTTP Core 4.2. It just sends out the
>>>>>>> messages without any SSL security. I can even monitor the
>>>>>>> message in plain text using TCPMon. Looks like some SSL
>>>>>>> related code in the Synapse HTTPS sender doesn't work properly
>>>>>>> with the new HTTP Core.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hiranya
>>>>>>>
>>>>>>
>>>>>> Hiranya
>>>>>>
>>>>>> Have you migrated off the deprecated APIs to new 4.2 APIs?
>>>>>
>>>>>
>>>>> No not yet. I just tried changing the HTTP Core dependency version
>>>>> to begin with. I guess migrating off the deprecated APIs is the
>>>>> next logical step.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Hiranya
>>>>>
>>>>>>
>>>>>> Oleg
>>>>>>
>>>>>> PS: By the way, you should probably consider migrating straight
>>>>>> to 4.3
>>>>>> at this point.
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> Rajika
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka
>>>>>>>> <hi...@gmail.com> wrote:
>>>>>>>> Hi Folks,
>>>>>>>>
>>>>>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried
>>>>>>>> using 4.2 instead (just changed the version of the maven
>>>>>>>> dependency). Almost everything worked fine, except for a
>>>>>>>> couple of integration test failures. In both these tests
>>>>>>>> Synapse uses HTTPS to contact the backend services, so I
>>>>>>>> suppose that's where things went wrong. The backend server
>>>>>>>> threw the following exception in the process:
>>>>>>>>
>>>>>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O
>>>>>>>> dispatcher-1] ERROR ServerHandler I/O error: Unrecognized
>>>>>>>> SSL message, plaintext connection?
>>>>>>>> javax.net.ssl.SSLException: Unrecognized SSL message,
>>>>>>>> plaintext connection?
>>>>>>>> at
>>>>>>>> com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>>>>>>> at
>>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>>>>>>> at
>>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>>>>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>>>>>>> at
>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>>>>>>> at
>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>>>>>>> at
>>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>>>>>> at
>>>>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>>>>>> at java.lang.Thread.run(Thread.java:680)
>>>>>>>>
>>>>>>>> I also tried upgrading to the latest HTTP Core version
>>>>>>>> (4.2.4). With this version NHTTP unit tests failed and most
>>>>>>>> of the integration tests failed too. It turns out only the
>>>>>>>> very first integration test ran successfully. Everything
>>>>>>>> that followed threw a bind exception. Looks like something
>>>>>>>> isn't getting cleaned up properly (although the logs
>>>>>>>> indicate that Synapse transport listeners are shutting down
>>>>>>>> cleanly after each test case).
>>>>>>>>
>>>>>>>> Does anybody got an idea what's going on? Do we need to do
>>>>>>>> any code changes to migrate to the latest HTTP Core?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Hiranya
>>>>>>>> --
>>>>>>>> Hiranya Jayathilaka
>>>>>>>> Mayhem Lab/RACE Lab;
>>>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Hiranya Jayathilaka
>>>>>>> Mayhem Lab/RACE Lab;
>>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>>>>>> For additional commands, e-mail: dev-help@hc.apache.org
>>>>>>
>>>>>
>>>>> --
>>>>> Hiranya Jayathilaka
>>>>> Mayhem Lab/RACE Lab;
>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>
>>>>>
>>>>
>>>> --
>>>> Hiranya Jayathilaka
>>>> Mayhem Lab/RACE Lab;
>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>
>>>>
>>>
>>> --
>>> Hiranya Jayathilaka
>>> Mayhem Lab/RACE Lab;
>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>> Blog: http://techfeast-hiranya.blogspot.com
>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi Oleg,
I can reproduce the problem with the attached JUnit test case. This test case doesn't use any Axis2/Synapse related code. It only requires HTTP Core and HTTP client. I used HTTP Core 4.2.4 for testing.
The test case starts a ListeningIOReactor, does a simple HTTP request-response exchange and then terminates the reactor. Then after a couple of seconds, it tries to start the IO reactor again and the exception occurs. Here's the full output:
Round 1
Listener started on port: 8280
Connected
Request received
Response submitted
Waiting - Shutdown in 5 secs
Shutdown
� wait 2 seconds...
Round 2
Listener started on port: 8280
Waiting - Shutdown in 5 secs
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
at org.apache.synapse.transport.nhttp.HttpCoreTest$3.run(HttpCoreTest.java:101)
Shutdown
Thanks,
Hiranya
On Jul 24, 2013, at 12:17 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Tue, 2013-07-23 at 20:24 -0700, Hiranya Jayathilaka wrote:
>> It looks like the socket goes into the CLOSE_WAIT state after shutting
>> down the IO reactor and remains there until the JVM it self is killed.
>> Since all automated test cases are executed within a single JVM
>> instance, this issue continues to occur. Looks like something is not
>> getting cleaned up properly with the latest HTTP Core.
>>
>>
>> Thanks,
>> Hiranya
>>
>
> Hiranya
>
> I need a more specific evidence of HttpCore doing something wrong. Could
> you please reproduce the issue with a test app, preferably not dependent
> on Synapse?
>
> Oleg
>
>> On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka
>> <hi...@gmail.com> wrote:
>>
>>> Hi Oleg,
>>>
>>>
>>> I've done the necessary code changes locally and it seems to work
>>> fine. The HTTPS sender now works as expected. But when running
>>> automated tests, I frequently get the following exception:
>>>
>>>
>>> 34 [HttpCoreNIOListener] WARN
>>> org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may
>>> be unstable: IOReactor encountered a checked exception : Address
>>> already in use
>>> java.net.BindException: Address already in use
>>> at sun.nio.ch.Net.bind(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
>>> at
>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>>> at
>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
>>> at
>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
>>> at
>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
>>> at
>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
>>> at org.apache.synapse.transport.nhttp.HttpCoreNIOListener
>>> $2.run(HttpCoreNIOListener.java:254)
>>> at java.lang.Thread.run(Thread.java:680)
>>>
>>>
>>> This is seen in both NHTTP unit tests and Synapse integration tests.
>>> Any idea what is going on here? Synapse cleans up the IO reactor by
>>> calling ioReactor.shutdown(). Is there anything else we need to do
>>> to clean up the reactor properly?
>>>
>>>
>>> Thanks,
>>> Hiranya
>>>
>>>
>>> On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka
>>> <hi...@gmail.com> wrote:
>>>
>>>> HI Oleg,
>>>>
>>>> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org>
>>>> wrote:
>>>>
>>>>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>>>>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri
>>>>>> <ra...@gmail.com> wrote:
>>>>>>
>>>>>>> May be we should enable the SSl debug logs and see what's
>>>>>>> going on ?
>>>>>>
>>>>>> I ran some more tests and it looks like the HTTPS sender
>>>>>> doesn't work with HTTP Core 4.2. It just sends out the
>>>>>> messages without any SSL security. I can even monitor the
>>>>>> message in plain text using TCPMon. Looks like some SSL
>>>>>> related code in the Synapse HTTPS sender doesn't work properly
>>>>>> with the new HTTP Core.
>>>>>>
>>>>>> Thanks,
>>>>>> Hiranya
>>>>>>
>>>>>
>>>>> Hiranya
>>>>>
>>>>> Have you migrated off the deprecated APIs to new 4.2 APIs?
>>>>
>>>>
>>>> No not yet. I just tried changing the HTTP Core dependency version
>>>> to begin with. I guess migrating off the deprecated APIs is the
>>>> next logical step.
>>>>
>>>>
>>>> Thanks,
>>>> Hiranya
>>>>
>>>>>
>>>>> Oleg
>>>>>
>>>>> PS: By the way, you should probably consider migrating straight
>>>>> to 4.3
>>>>> at this point.
>>>>>
>>>>>
>>>>>>>
>>>>>>> Rajika
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka
>>>>>>> <hi...@gmail.com> wrote:
>>>>>>> Hi Folks,
>>>>>>>
>>>>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried
>>>>>>> using 4.2 instead (just changed the version of the maven
>>>>>>> dependency). Almost everything worked fine, except for a
>>>>>>> couple of integration test failures. In both these tests
>>>>>>> Synapse uses HTTPS to contact the backend services, so I
>>>>>>> suppose that's where things went wrong. The backend server
>>>>>>> threw the following exception in the process:
>>>>>>>
>>>>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O
>>>>>>> dispatcher-1] ERROR ServerHandler I/O error: Unrecognized
>>>>>>> SSL message, plaintext connection?
>>>>>>> javax.net.ssl.SSLException: Unrecognized SSL message,
>>>>>>> plaintext connection?
>>>>>>> at
>>>>>>> com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>>>>>> at
>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>>>>>> at
>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>>>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>>>>>> at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>>>>>> at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>>>>>> at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>>>>> at java.lang.Thread.run(Thread.java:680)
>>>>>>>
>>>>>>> I also tried upgrading to the latest HTTP Core version
>>>>>>> (4.2.4). With this version NHTTP unit tests failed and most
>>>>>>> of the integration tests failed too. It turns out only the
>>>>>>> very first integration test ran successfully. Everything
>>>>>>> that followed threw a bind exception. Looks like something
>>>>>>> isn't getting cleaned up properly (although the logs
>>>>>>> indicate that Synapse transport listeners are shutting down
>>>>>>> cleanly after each test case).
>>>>>>>
>>>>>>> Does anybody got an idea what's going on? Do we need to do
>>>>>>> any code changes to migrate to the latest HTTP Core?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hiranya
>>>>>>> --
>>>>>>> Hiranya Jayathilaka
>>>>>>> Mayhem Lab/RACE Lab;
>>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Hiranya Jayathilaka
>>>>>> Mayhem Lab/RACE Lab;
>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>>>>> For additional commands, e-mail: dev-help@hc.apache.org
>>>>>
>>>>
>>>> --
>>>> Hiranya Jayathilaka
>>>> Mayhem Lab/RACE Lab;
>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>
>>>>
>>>
>>> --
>>> Hiranya Jayathilaka
>>> Mayhem Lab/RACE Lab;
>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>> Blog: http://techfeast-hiranya.blogspot.com
>>>
>>>
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi Oleg,
I can reproduce the problem with the attached JUnit test case. This test case doesn't use any Axis2/Synapse related code. It only requires HTTP Core and HTTP client. I used HTTP Core 4.2.4 for testing.
The test case starts a ListeningIOReactor, does a simple HTTP request-response exchange and then terminates the reactor. Then after a couple of seconds, it tries to start the IO reactor again and the exception occurs. Here's the full output:
Round 1
Listener started on port: 8280
Connected
Request received
Response submitted
Waiting - Shutdown in 5 secs
Shutdown
… wait 2 seconds...
Round 2
Listener started on port: 8280
Waiting - Shutdown in 5 secs
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
at org.apache.synapse.transport.nhttp.HttpCoreTest$3.run(HttpCoreTest.java:101)
Shutdown
Thanks,
Hiranya
On Jul 24, 2013, at 12:17 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Tue, 2013-07-23 at 20:24 -0700, Hiranya Jayathilaka wrote:
>> It looks like the socket goes into the CLOSE_WAIT state after shutting
>> down the IO reactor and remains there until the JVM it self is killed.
>> Since all automated test cases are executed within a single JVM
>> instance, this issue continues to occur. Looks like something is not
>> getting cleaned up properly with the latest HTTP Core.
>>
>>
>> Thanks,
>> Hiranya
>>
>
> Hiranya
>
> I need a more specific evidence of HttpCore doing something wrong. Could
> you please reproduce the issue with a test app, preferably not dependent
> on Synapse?
>
> Oleg
>
>> On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka
>> <hi...@gmail.com> wrote:
>>
>>> Hi Oleg,
>>>
>>>
>>> I've done the necessary code changes locally and it seems to work
>>> fine. The HTTPS sender now works as expected. But when running
>>> automated tests, I frequently get the following exception:
>>>
>>>
>>> 34 [HttpCoreNIOListener] WARN
>>> org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may
>>> be unstable: IOReactor encountered a checked exception : Address
>>> already in use
>>> java.net.BindException: Address already in use
>>> at sun.nio.ch.Net.bind(Native Method)
>>> at
>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
>>> at
>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>>> at
>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
>>> at
>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
>>> at
>>> org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
>>> at
>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
>>> at org.apache.synapse.transport.nhttp.HttpCoreNIOListener
>>> $2.run(HttpCoreNIOListener.java:254)
>>> at java.lang.Thread.run(Thread.java:680)
>>>
>>>
>>> This is seen in both NHTTP unit tests and Synapse integration tests.
>>> Any idea what is going on here? Synapse cleans up the IO reactor by
>>> calling ioReactor.shutdown(). Is there anything else we need to do
>>> to clean up the reactor properly?
>>>
>>>
>>> Thanks,
>>> Hiranya
>>>
>>>
>>> On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka
>>> <hi...@gmail.com> wrote:
>>>
>>>> HI Oleg,
>>>>
>>>> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org>
>>>> wrote:
>>>>
>>>>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>>>>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri
>>>>>> <ra...@gmail.com> wrote:
>>>>>>
>>>>>>> May be we should enable the SSl debug logs and see what's
>>>>>>> going on ?
>>>>>>
>>>>>> I ran some more tests and it looks like the HTTPS sender
>>>>>> doesn't work with HTTP Core 4.2. It just sends out the
>>>>>> messages without any SSL security. I can even monitor the
>>>>>> message in plain text using TCPMon. Looks like some SSL
>>>>>> related code in the Synapse HTTPS sender doesn't work properly
>>>>>> with the new HTTP Core.
>>>>>>
>>>>>> Thanks,
>>>>>> Hiranya
>>>>>>
>>>>>
>>>>> Hiranya
>>>>>
>>>>> Have you migrated off the deprecated APIs to new 4.2 APIs?
>>>>
>>>>
>>>> No not yet. I just tried changing the HTTP Core dependency version
>>>> to begin with. I guess migrating off the deprecated APIs is the
>>>> next logical step.
>>>>
>>>>
>>>> Thanks,
>>>> Hiranya
>>>>
>>>>>
>>>>> Oleg
>>>>>
>>>>> PS: By the way, you should probably consider migrating straight
>>>>> to 4.3
>>>>> at this point.
>>>>>
>>>>>
>>>>>>>
>>>>>>> Rajika
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka
>>>>>>> <hi...@gmail.com> wrote:
>>>>>>> Hi Folks,
>>>>>>>
>>>>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried
>>>>>>> using 4.2 instead (just changed the version of the maven
>>>>>>> dependency). Almost everything worked fine, except for a
>>>>>>> couple of integration test failures. In both these tests
>>>>>>> Synapse uses HTTPS to contact the backend services, so I
>>>>>>> suppose that's where things went wrong. The backend server
>>>>>>> threw the following exception in the process:
>>>>>>>
>>>>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O
>>>>>>> dispatcher-1] ERROR ServerHandler I/O error: Unrecognized
>>>>>>> SSL message, plaintext connection?
>>>>>>> javax.net.ssl.SSLException: Unrecognized SSL message,
>>>>>>> plaintext connection?
>>>>>>> at
>>>>>>> com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>>>>>> at
>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>>>>>> at
>>>>>>> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>>>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>>>>>> at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>>>>>> at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>>>>>> at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>>>>> at
>>>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>>>>> at java.lang.Thread.run(Thread.java:680)
>>>>>>>
>>>>>>> I also tried upgrading to the latest HTTP Core version
>>>>>>> (4.2.4). With this version NHTTP unit tests failed and most
>>>>>>> of the integration tests failed too. It turns out only the
>>>>>>> very first integration test ran successfully. Everything
>>>>>>> that followed threw a bind exception. Looks like something
>>>>>>> isn't getting cleaned up properly (although the logs
>>>>>>> indicate that Synapse transport listeners are shutting down
>>>>>>> cleanly after each test case).
>>>>>>>
>>>>>>> Does anybody got an idea what's going on? Do we need to do
>>>>>>> any code changes to migrate to the latest HTTP Core?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Hiranya
>>>>>>> --
>>>>>>> Hiranya Jayathilaka
>>>>>>> Mayhem Lab/RACE Lab;
>>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Hiranya Jayathilaka
>>>>>> Mayhem Lab/RACE Lab;
>>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>>>>> For additional commands, e-mail: dev-help@hc.apache.org
>>>>>
>>>>
>>>> --
>>>> Hiranya Jayathilaka
>>>> Mayhem Lab/RACE Lab;
>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>
>>>>
>>>
>>> --
>>> Hiranya Jayathilaka
>>> Mayhem Lab/RACE Lab;
>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>> Blog: http://techfeast-hiranya.blogspot.com
>>>
>>>
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2013-07-23 at 20:24 -0700, Hiranya Jayathilaka wrote:
> It looks like the socket goes into the CLOSE_WAIT state after shutting
> down the IO reactor and remains there until the JVM it self is killed.
> Since all automated test cases are executed within a single JVM
> instance, this issue continues to occur. Looks like something is not
> getting cleaned up properly with the latest HTTP Core.
>
>
> Thanks,
> Hiranya
>
Hiranya
I need a more specific evidence of HttpCore doing something wrong. Could
you please reproduce the issue with a test app, preferably not dependent
on Synapse?
Oleg
> On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka
> <hi...@gmail.com> wrote:
>
> > Hi Oleg,
> >
> >
> > I've done the necessary code changes locally and it seems to work
> > fine. The HTTPS sender now works as expected. But when running
> > automated tests, I frequently get the following exception:
> >
> >
> > 34 [HttpCoreNIOListener] WARN
> > org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may
> > be unstable: IOReactor encountered a checked exception : Address
> > already in use
> > java.net.BindException: Address already in use
> > at sun.nio.ch.Net.bind(Native Method)
> > at
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
> > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
> > at
> > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
> > at
> > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
> > at
> > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
> > at org.apache.synapse.transport.nhttp.HttpCoreNIOListener
> > $2.run(HttpCoreNIOListener.java:254)
> > at java.lang.Thread.run(Thread.java:680)
> >
> >
> > This is seen in both NHTTP unit tests and Synapse integration tests.
> > Any idea what is going on here? Synapse cleans up the IO reactor by
> > calling ioReactor.shutdown(). Is there anything else we need to do
> > to clean up the reactor properly?
> >
> >
> > Thanks,
> > Hiranya
> >
> >
> > On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka
> > <hi...@gmail.com> wrote:
> >
> > > HI Oleg,
> > >
> > > On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org>
> > > wrote:
> > >
> > > > On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
> > > > > On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri
> > > > > <ra...@gmail.com> wrote:
> > > > >
> > > > > > May be we should enable the SSl debug logs and see what's
> > > > > > going on ?
> > > > >
> > > > > I ran some more tests and it looks like the HTTPS sender
> > > > > doesn't work with HTTP Core 4.2. It just sends out the
> > > > > messages without any SSL security. I can even monitor the
> > > > > message in plain text using TCPMon. Looks like some SSL
> > > > > related code in the Synapse HTTPS sender doesn't work properly
> > > > > with the new HTTP Core.
> > > > >
> > > > > Thanks,
> > > > > Hiranya
> > > > >
> > > >
> > > > Hiranya
> > > >
> > > > Have you migrated off the deprecated APIs to new 4.2 APIs?
> > >
> > >
> > > No not yet. I just tried changing the HTTP Core dependency version
> > > to begin with. I guess migrating off the deprecated APIs is the
> > > next logical step.
> > >
> > >
> > > Thanks,
> > > Hiranya
> > >
> > > >
> > > > Oleg
> > > >
> > > > PS: By the way, you should probably consider migrating straight
> > > > to 4.3
> > > > at this point.
> > > >
> > > >
> > > > > >
> > > > > > Rajika
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka
> > > > > > <hi...@gmail.com> wrote:
> > > > > > Hi Folks,
> > > > > >
> > > > > > Currently Synapse is based on HTTP Core 4.1.4. I just tried
> > > > > > using 4.2 instead (just changed the version of the maven
> > > > > > dependency). Almost everything worked fine, except for a
> > > > > > couple of integration test failures. In both these tests
> > > > > > Synapse uses HTTPS to contact the backend services, so I
> > > > > > suppose that's where things went wrong. The backend server
> > > > > > threw the following exception in the process:
> > > > > >
> > > > > > 2013-07-22 17:06:36,970 [-] [https-Listener I/O
> > > > > > dispatcher-1] ERROR ServerHandler I/O error: Unrecognized
> > > > > > SSL message, plaintext connection?
> > > > > > javax.net.ssl.SSLException: Unrecognized SSL message,
> > > > > > plaintext connection?
> > > > > > at
> > > > > > com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
> > > > > > at
> > > > > > com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
> > > > > > at
> > > > > > com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
> > > > > > at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> > > > > > at
> > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
> > > > > > at
> > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
> > > > > > at
> > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
> > > > > > at
> > > > > > org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> > > > > > at java.lang.Thread.run(Thread.java:680)
> > > > > >
> > > > > > I also tried upgrading to the latest HTTP Core version
> > > > > > (4.2.4). With this version NHTTP unit tests failed and most
> > > > > > of the integration tests failed too. It turns out only the
> > > > > > very first integration test ran successfully. Everything
> > > > > > that followed threw a bind exception. Looks like something
> > > > > > isn't getting cleaned up properly (although the logs
> > > > > > indicate that Synapse transport listeners are shutting down
> > > > > > cleanly after each test case).
> > > > > >
> > > > > > Does anybody got an idea what's going on? Do we need to do
> > > > > > any code changes to migrate to the latest HTTP Core?
> > > > > >
> > > > > > Thanks,
> > > > > > Hiranya
> > > > > > --
> > > > > > Hiranya Jayathilaka
> > > > > > Mayhem Lab/RACE Lab;
> > > > > > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > > > > > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > > > > > Blog: http://techfeast-hiranya.blogspot.com
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Hiranya Jayathilaka
> > > > > Mayhem Lab/RACE Lab;
> > > > > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > > > > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > > > > Blog: http://techfeast-hiranya.blogspot.com
> > > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> > >
> > > --
> > > Hiranya Jayathilaka
> > > Mayhem Lab/RACE Lab;
> > > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > > Blog: http://techfeast-hiranya.blogspot.com
> > >
> > >
> >
> > --
> > Hiranya Jayathilaka
> > Mayhem Lab/RACE Lab;
> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > Blog: http://techfeast-hiranya.blogspot.com
> >
> >
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2013-07-23 at 20:24 -0700, Hiranya Jayathilaka wrote:
> It looks like the socket goes into the CLOSE_WAIT state after shutting
> down the IO reactor and remains there until the JVM it self is killed.
> Since all automated test cases are executed within a single JVM
> instance, this issue continues to occur. Looks like something is not
> getting cleaned up properly with the latest HTTP Core.
>
>
> Thanks,
> Hiranya
>
Hiranya
I need a more specific evidence of HttpCore doing something wrong. Could
you please reproduce the issue with a test app, preferably not dependent
on Synapse?
Oleg
> On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka
> <hi...@gmail.com> wrote:
>
> > Hi Oleg,
> >
> >
> > I've done the necessary code changes locally and it seems to work
> > fine. The HTTPS sender now works as expected. But when running
> > automated tests, I frequently get the following exception:
> >
> >
> > 34 [HttpCoreNIOListener] WARN
> > org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may
> > be unstable: IOReactor encountered a checked exception : Address
> > already in use
> > java.net.BindException: Address already in use
> > at sun.nio.ch.Net.bind(Native Method)
> > at
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
> > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
> > at
> > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
> > at
> > org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
> > at
> > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
> > at org.apache.synapse.transport.nhttp.HttpCoreNIOListener
> > $2.run(HttpCoreNIOListener.java:254)
> > at java.lang.Thread.run(Thread.java:680)
> >
> >
> > This is seen in both NHTTP unit tests and Synapse integration tests.
> > Any idea what is going on here? Synapse cleans up the IO reactor by
> > calling ioReactor.shutdown(). Is there anything else we need to do
> > to clean up the reactor properly?
> >
> >
> > Thanks,
> > Hiranya
> >
> >
> > On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka
> > <hi...@gmail.com> wrote:
> >
> > > HI Oleg,
> > >
> > > On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org>
> > > wrote:
> > >
> > > > On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
> > > > > On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri
> > > > > <ra...@gmail.com> wrote:
> > > > >
> > > > > > May be we should enable the SSl debug logs and see what's
> > > > > > going on ?
> > > > >
> > > > > I ran some more tests and it looks like the HTTPS sender
> > > > > doesn't work with HTTP Core 4.2. It just sends out the
> > > > > messages without any SSL security. I can even monitor the
> > > > > message in plain text using TCPMon. Looks like some SSL
> > > > > related code in the Synapse HTTPS sender doesn't work properly
> > > > > with the new HTTP Core.
> > > > >
> > > > > Thanks,
> > > > > Hiranya
> > > > >
> > > >
> > > > Hiranya
> > > >
> > > > Have you migrated off the deprecated APIs to new 4.2 APIs?
> > >
> > >
> > > No not yet. I just tried changing the HTTP Core dependency version
> > > to begin with. I guess migrating off the deprecated APIs is the
> > > next logical step.
> > >
> > >
> > > Thanks,
> > > Hiranya
> > >
> > > >
> > > > Oleg
> > > >
> > > > PS: By the way, you should probably consider migrating straight
> > > > to 4.3
> > > > at this point.
> > > >
> > > >
> > > > > >
> > > > > > Rajika
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka
> > > > > > <hi...@gmail.com> wrote:
> > > > > > Hi Folks,
> > > > > >
> > > > > > Currently Synapse is based on HTTP Core 4.1.4. I just tried
> > > > > > using 4.2 instead (just changed the version of the maven
> > > > > > dependency). Almost everything worked fine, except for a
> > > > > > couple of integration test failures. In both these tests
> > > > > > Synapse uses HTTPS to contact the backend services, so I
> > > > > > suppose that's where things went wrong. The backend server
> > > > > > threw the following exception in the process:
> > > > > >
> > > > > > 2013-07-22 17:06:36,970 [-] [https-Listener I/O
> > > > > > dispatcher-1] ERROR ServerHandler I/O error: Unrecognized
> > > > > > SSL message, plaintext connection?
> > > > > > javax.net.ssl.SSLException: Unrecognized SSL message,
> > > > > > plaintext connection?
> > > > > > at
> > > > > > com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
> > > > > > at
> > > > > > com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
> > > > > > at
> > > > > > com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
> > > > > > at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> > > > > > at
> > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
> > > > > > at
> > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
> > > > > > at
> > > > > > org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
> > > > > > at
> > > > > > org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> > > > > > at
> > > > > > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> > > > > > at java.lang.Thread.run(Thread.java:680)
> > > > > >
> > > > > > I also tried upgrading to the latest HTTP Core version
> > > > > > (4.2.4). With this version NHTTP unit tests failed and most
> > > > > > of the integration tests failed too. It turns out only the
> > > > > > very first integration test ran successfully. Everything
> > > > > > that followed threw a bind exception. Looks like something
> > > > > > isn't getting cleaned up properly (although the logs
> > > > > > indicate that Synapse transport listeners are shutting down
> > > > > > cleanly after each test case).
> > > > > >
> > > > > > Does anybody got an idea what's going on? Do we need to do
> > > > > > any code changes to migrate to the latest HTTP Core?
> > > > > >
> > > > > > Thanks,
> > > > > > Hiranya
> > > > > > --
> > > > > > Hiranya Jayathilaka
> > > > > > Mayhem Lab/RACE Lab;
> > > > > > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > > > > > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > > > > > Blog: http://techfeast-hiranya.blogspot.com
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Hiranya Jayathilaka
> > > > > Mayhem Lab/RACE Lab;
> > > > > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > > > > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > > > > Blog: http://techfeast-hiranya.blogspot.com
> > > > >
> > > >
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail: dev-help@hc.apache.org
> > > >
> > >
> > > --
> > > Hiranya Jayathilaka
> > > Mayhem Lab/RACE Lab;
> > > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > > Blog: http://techfeast-hiranya.blogspot.com
> > >
> > >
> >
> > --
> > Hiranya Jayathilaka
> > Mayhem Lab/RACE Lab;
> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > Blog: http://techfeast-hiranya.blogspot.com
> >
> >
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
It looks like the socket goes into the CLOSE_WAIT state after shutting down the IO reactor and remains there until the JVM it self is killed. Since all automated test cases are executed within a single JVM instance, this issue continues to occur. Looks like something is not getting cleaned up properly with the latest HTTP Core.
Thanks,
Hiranya
On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> Hi Oleg,
>
> I've done the necessary code changes locally and it seems to work fine. The HTTPS sender now works as expected. But when running automated tests, I frequently get the following exception:
>
> 34 [HttpCoreNIOListener] WARN org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may be unstable: IOReactor encountered a checked exception : Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
> at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
> at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
> at org.apache.synapse.transport.nhttp.HttpCoreNIOListener$2.run(HttpCoreNIOListener.java:254)
> at java.lang.Thread.run(Thread.java:680)
>
> This is seen in both NHTTP unit tests and Synapse integration tests. Any idea what is going on here? Synapse cleans up the IO reactor by calling ioReactor.shutdown(). Is there anything else we need to do to clean up the reactor properly?
>
> Thanks,
> Hiranya
>
> On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
>
>> HI Oleg,
>>
>> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>>
>>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
>>>>
>>>>> May be we should enable the SSl debug logs and see what's going on ?
>>>>
>>>> I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
>>>>
>>>> Thanks,
>>>> Hiranya
>>>>
>>>
>>> Hiranya
>>>
>>> Have you migrated off the deprecated APIs to new 4.2 APIs?
>>
>> No not yet. I just tried changing the HTTP Core dependency version to begin with. I guess migrating off the deprecated APIs is the next logical step.
>>
>> Thanks,
>> Hiranya
>>
>>>
>>> Oleg
>>>
>>> PS: By the way, you should probably consider migrating straight to 4.3
>>> at this point.
>>>
>>>
>>>>>
>>>>> Rajika
>>>>>
>>>>>
>>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
>>>>> Hi Folks,
>>>>>
>>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
>>>>>
>>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
>>>>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>>>> at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>>>> at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>>> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>>> at java.lang.Thread.run(Thread.java:680)
>>>>>
>>>>> I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
>>>>>
>>>>> Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
>>>>>
>>>>> Thanks,
>>>>> Hiranya
>>>>> --
>>>>> Hiranya Jayathilaka
>>>>> Mayhem Lab/RACE Lab;
>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>
>>>>>
>>>>
>>>> --
>>>> Hiranya Jayathilaka
>>>> Mayhem Lab/RACE Lab;
>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: dev-help@hc.apache.org
>>>
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
It looks like the socket goes into the CLOSE_WAIT state after shutting down the IO reactor and remains there until the JVM it self is killed. Since all automated test cases are executed within a single JVM instance, this issue continues to occur. Looks like something is not getting cleaned up properly with the latest HTTP Core.
Thanks,
Hiranya
On Jul 23, 2013, at 3:54 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> Hi Oleg,
>
> I've done the necessary code changes locally and it seems to work fine. The HTTPS sender now works as expected. But when running automated tests, I frequently get the following exception:
>
> 34 [HttpCoreNIOListener] WARN org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may be unstable: IOReactor encountered a checked exception : Address already in use
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
> at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
> at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
> at org.apache.synapse.transport.nhttp.HttpCoreNIOListener$2.run(HttpCoreNIOListener.java:254)
> at java.lang.Thread.run(Thread.java:680)
>
> This is seen in both NHTTP unit tests and Synapse integration tests. Any idea what is going on here? Synapse cleans up the IO reactor by calling ioReactor.shutdown(). Is there anything else we need to do to clean up the reactor properly?
>
> Thanks,
> Hiranya
>
> On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
>
>> HI Oleg,
>>
>> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>>
>>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
>>>>
>>>>> May be we should enable the SSl debug logs and see what's going on ?
>>>>
>>>> I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
>>>>
>>>> Thanks,
>>>> Hiranya
>>>>
>>>
>>> Hiranya
>>>
>>> Have you migrated off the deprecated APIs to new 4.2 APIs?
>>
>> No not yet. I just tried changing the HTTP Core dependency version to begin with. I guess migrating off the deprecated APIs is the next logical step.
>>
>> Thanks,
>> Hiranya
>>
>>>
>>> Oleg
>>>
>>> PS: By the way, you should probably consider migrating straight to 4.3
>>> at this point.
>>>
>>>
>>>>>
>>>>> Rajika
>>>>>
>>>>>
>>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
>>>>> Hi Folks,
>>>>>
>>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
>>>>>
>>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
>>>>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>>>> at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>>>> at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>>> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>>> at java.lang.Thread.run(Thread.java:680)
>>>>>
>>>>> I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
>>>>>
>>>>> Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
>>>>>
>>>>> Thanks,
>>>>> Hiranya
>>>>> --
>>>>> Hiranya Jayathilaka
>>>>> Mayhem Lab/RACE Lab;
>>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>>
>>>>>
>>>>
>>>> --
>>>> Hiranya Jayathilaka
>>>> Mayhem Lab/RACE Lab;
>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: dev-help@hc.apache.org
>>>
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi Oleg,
I've done the necessary code changes locally and it seems to work fine. The HTTPS sender now works as expected. But when running automated tests, I frequently get the following exception:
34 [HttpCoreNIOListener] WARN org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may be unstable: IOReactor encountered a checked exception : Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
at org.apache.synapse.transport.nhttp.HttpCoreNIOListener$2.run(HttpCoreNIOListener.java:254)
at java.lang.Thread.run(Thread.java:680)
This is seen in both NHTTP unit tests and Synapse integration tests. Any idea what is going on here? Synapse cleans up the IO reactor by calling ioReactor.shutdown(). Is there anything else we need to do to clean up the reactor properly?
Thanks,
Hiranya
On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> HI Oleg,
>
> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
>>>
>>>> May be we should enable the SSl debug logs and see what's going on ?
>>>
>>> I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
>>>
>>> Thanks,
>>> Hiranya
>>>
>>
>> Hiranya
>>
>> Have you migrated off the deprecated APIs to new 4.2 APIs?
>
> No not yet. I just tried changing the HTTP Core dependency version to begin with. I guess migrating off the deprecated APIs is the next logical step.
>
> Thanks,
> Hiranya
>
>>
>> Oleg
>>
>> PS: By the way, you should probably consider migrating straight to 4.3
>> at this point.
>>
>>
>>>>
>>>> Rajika
>>>>
>>>>
>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
>>>> Hi Folks,
>>>>
>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
>>>>
>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
>>>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>>> at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>>> at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>> at java.lang.Thread.run(Thread.java:680)
>>>>
>>>> I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
>>>>
>>>> Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
>>>>
>>>> Thanks,
>>>> Hiranya
>>>> --
>>>> Hiranya Jayathilaka
>>>> Mayhem Lab/RACE Lab;
>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>
>>>>
>>>
>>> --
>>> Hiranya Jayathilaka
>>> Mayhem Lab/RACE Lab;
>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>> Blog: http://techfeast-hiranya.blogspot.com
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
Hi Oleg,
I've done the necessary code changes locally and it seems to work fine. The HTTPS sender now works as expected. But when running automated tests, I frequently get the following exception:
34 [HttpCoreNIOListener] WARN org.apache.synapse.transport.nhttp.HttpCoreNIOListener - System may be unstable: IOReactor encountered a checked exception : Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:124)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processSessionRequests(DefaultListeningIOReactor.java:238)
at org.apache.http.impl.nio.reactor.DefaultListeningIOReactor.processEvents(DefaultListeningIOReactor.java:144)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:374)
at org.apache.synapse.transport.nhttp.HttpCoreNIOListener$2.run(HttpCoreNIOListener.java:254)
at java.lang.Thread.run(Thread.java:680)
This is seen in both NHTTP unit tests and Synapse integration tests. Any idea what is going on here? Synapse cleans up the IO reactor by calling ioReactor.shutdown(). Is there anything else we need to do to clean up the reactor properly?
Thanks,
Hiranya
On Jul 23, 2013, at 10:54 AM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> HI Oleg,
>
> On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
>> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
>>>
>>>> May be we should enable the SSl debug logs and see what's going on ?
>>>
>>> I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
>>>
>>> Thanks,
>>> Hiranya
>>>
>>
>> Hiranya
>>
>> Have you migrated off the deprecated APIs to new 4.2 APIs?
>
> No not yet. I just tried changing the HTTP Core dependency version to begin with. I guess migrating off the deprecated APIs is the next logical step.
>
> Thanks,
> Hiranya
>
>>
>> Oleg
>>
>> PS: By the way, you should probably consider migrating straight to 4.3
>> at this point.
>>
>>
>>>>
>>>> Rajika
>>>>
>>>>
>>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
>>>> Hi Folks,
>>>>
>>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
>>>>
>>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
>>>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>>> at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>>> at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>> at java.lang.Thread.run(Thread.java:680)
>>>>
>>>> I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
>>>>
>>>> Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
>>>>
>>>> Thanks,
>>>> Hiranya
>>>> --
>>>> Hiranya Jayathilaka
>>>> Mayhem Lab/RACE Lab;
>>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>>> Blog: http://techfeast-hiranya.blogspot.com
>>>>
>>>>
>>>
>>> --
>>> Hiranya Jayathilaka
>>> Mayhem Lab/RACE Lab;
>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>> Blog: http://techfeast-hiranya.blogspot.com
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
HI Oleg,
On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
>>
>>> May be we should enable the SSl debug logs and see what's going on ?
>>
>> I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
>>
>> Thanks,
>> Hiranya
>>
>
> Hiranya
>
> Have you migrated off the deprecated APIs to new 4.2 APIs?
No not yet. I just tried changing the HTTP Core dependency version to begin with. I guess migrating off the deprecated APIs is the next logical step.
Thanks,
Hiranya
>
> Oleg
>
> PS: By the way, you should probably consider migrating straight to 4.3
> at this point.
>
>
>>>
>>> Rajika
>>>
>>>
>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
>>> Hi Folks,
>>>
>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
>>>
>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
>>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>> at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>> at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>> at java.lang.Thread.run(Thread.java:680)
>>>
>>> I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
>>>
>>> Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
>>>
>>> Thanks,
>>> Hiranya
>>> --
>>> Hiranya Jayathilaka
>>> Mayhem Lab/RACE Lab;
>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>> Blog: http://techfeast-hiranya.blogspot.com
>>>
>>>
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
HI Oleg,
On Jul 23, 2013, at 12:32 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
>> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
>>
>>> May be we should enable the SSl debug logs and see what's going on ?
>>
>> I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
>>
>> Thanks,
>> Hiranya
>>
>
> Hiranya
>
> Have you migrated off the deprecated APIs to new 4.2 APIs?
No not yet. I just tried changing the HTTP Core dependency version to begin with. I guess migrating off the deprecated APIs is the next logical step.
Thanks,
Hiranya
>
> Oleg
>
> PS: By the way, you should probably consider migrating straight to 4.3
> at this point.
>
>
>>>
>>> Rajika
>>>
>>>
>>> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
>>> Hi Folks,
>>>
>>> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
>>>
>>> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
>>> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
>>> at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
>>> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
>>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
>>> at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>> at java.lang.Thread.run(Thread.java:680)
>>>
>>> I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
>>>
>>> Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
>>>
>>> Thanks,
>>> Hiranya
>>> --
>>> Hiranya Jayathilaka
>>> Mayhem Lab/RACE Lab;
>>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>>> Blog: http://techfeast-hiranya.blogspot.com
>>>
>>>
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
>
> > May be we should enable the SSl debug logs and see what's going on ?
>
> I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
>
> Thanks,
> Hiranya
>
Hiranya
Have you migrated off the deprecated APIs to new 4.2 APIs?
Oleg
PS: By the way, you should probably consider migrating straight to 4.3
at this point.
> >
> > Rajika
> >
> >
> > On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> > Hi Folks,
> >
> > Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
> >
> > 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
> > javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
> > at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
> > at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
> > at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
> > at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
> > at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
> > at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
> > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
> > at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> > at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> > at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
> >
> > Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
> >
> > Thanks,
> > Hiranya
> > --
> > Hiranya Jayathilaka
> > Mayhem Lab/RACE Lab;
> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > Blog: http://techfeast-hiranya.blogspot.com
> >
> >
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2013-07-22 at 20:51 -0700, Hiranya Jayathilaka wrote:
> On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
>
> > May be we should enable the SSl debug logs and see what's going on ?
>
> I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
>
> Thanks,
> Hiranya
>
Hiranya
Have you migrated off the deprecated APIs to new 4.2 APIs?
Oleg
PS: By the way, you should probably consider migrating straight to 4.3
at this point.
> >
> > Rajika
> >
> >
> > On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> > Hi Folks,
> >
> > Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
> >
> > 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
> > javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
> > at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
> > at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
> > at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
> > at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
> > at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
> > at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
> > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
> > at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> > at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> > at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> > at java.lang.Thread.run(Thread.java:680)
> >
> > I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
> >
> > Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
> >
> > Thanks,
> > Hiranya
> > --
> > Hiranya Jayathilaka
> > Mayhem Lab/RACE Lab;
> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > Blog: http://techfeast-hiranya.blogspot.com
> >
> >
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
> May be we should enable the SSl debug logs and see what's going on ?
I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
Thanks,
Hiranya
>
> Rajika
>
>
> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> Hi Folks,
>
> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
>
> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
> at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
> at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Thread.java:680)
>
> I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
>
> Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
>
> Thanks,
> Hiranya
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
On Jul 22, 2013, at 7:53 PM, Rajika Kumarasiri <ra...@gmail.com> wrote:
> May be we should enable the SSl debug logs and see what's going on ?
I ran some more tests and it looks like the HTTPS sender doesn't work with HTTP Core 4.2. It just sends out the messages without any SSL security. I can even monitor the message in plain text using TCPMon. Looks like some SSL related code in the Synapse HTTPS sender doesn't work properly with the new HTTP Core.
Thanks,
Hiranya
>
> Rajika
>
>
> On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka <hi...@gmail.com> wrote:
> Hi Folks,
>
> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2 instead (just changed the version of the maven dependency). Almost everything worked fine, except for a couple of integration test failures. In both these tests Synapse uses HTTPS to contact the backend services, so I suppose that's where things went wrong. The backend server threw the following exception in the process:
>
> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
> at com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
> at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
> at org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Thread.java:680)
>
> I also tried upgrading to the latest HTTP Core version (4.2.4). With this version NHTTP unit tests failed and most of the integration tests failed too. It turns out only the very first integration test ran successfully. Everything that followed threw a bind exception. Looks like something isn't getting cleaned up properly (although the logs indicate that Synapse transport listeners are shutting down cleanly after each test case).
>
> Does anybody got an idea what's going on? Do we need to do any code changes to migrate to the latest HTTP Core?
>
> Thanks,
> Hiranya
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
>
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Migrating Synapse to Latest HTTP Core
Posted by Rajika Kumarasiri <ra...@gmail.com>.
May be we should enable the SSl debug logs and see what's going on ?
Rajika
On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka
<hi...@gmail.com>wrote:
> Hi Folks,
>
> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2
> instead (just changed the version of the maven dependency). Almost
> everything worked fine, except for a couple of integration test failures.
> In both these tests Synapse uses HTTPS to contact the backend services, so
> I suppose that's where things went wrong. The backend server threw the
> following exception in the process:
>
> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR
> ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
> at
> com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
> at
> com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
> at
> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
> at
> org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Thread.java:680)
>
> I also tried upgrading to the latest HTTP Core version (4.2.4). With this
> version NHTTP unit tests failed and most of the integration tests failed
> too. It turns out only the very first integration test ran successfully.
> Everything that followed threw a bind exception. Looks like something isn't
> getting cleaned up properly (although the logs indicate that Synapse
> transport listeners are shutting down cleanly after each test case).
>
> Does anybody got an idea what's going on? Do we need to do any code
> changes to migrate to the latest HTTP Core?
>
> Thanks,
> Hiranya
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
>
Re: Migrating Synapse to Latest HTTP Core
Posted by Rajika Kumarasiri <ra...@gmail.com>.
May be we should enable the SSl debug logs and see what's going on ?
Rajika
On Mon, Jul 22, 2013 at 8:23 PM, Hiranya Jayathilaka
<hi...@gmail.com>wrote:
> Hi Folks,
>
> Currently Synapse is based on HTTP Core 4.1.4. I just tried using 4.2
> instead (just changed the version of the maven dependency). Almost
> everything worked fine, except for a couple of integration test failures.
> In both these tests Synapse uses HTTPS to contact the backend services, so
> I suppose that's where things went wrong. The backend server threw the
> following exception in the process:
>
> 2013-07-22 17:06:36,970 [-] [https-Listener I/O dispatcher-1] ERROR
> ServerHandler I/O error: Unrecognized SSL message, plaintext connection?
> javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
> at
> com.sun.net.ssl.internal.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:152)
> at
> com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:806)
> at
> com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:228)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:263)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:370)
> at
> org.apache.http.impl.nio.SSLServerIOEventDispatch.inputReady(SSLServerIOEventDispatch.java:229)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
> at java.lang.Thread.run(Thread.java:680)
>
> I also tried upgrading to the latest HTTP Core version (4.2.4). With this
> version NHTTP unit tests failed and most of the integration tests failed
> too. It turns out only the very first integration test ran successfully.
> Everything that followed threw a bind exception. Looks like something isn't
> getting cleaned up properly (although the logs indicate that Synapse
> transport listeners are shutting down cleanly after each test case).
>
> Does anybody got an idea what's going on? Do we need to do any code
> changes to migrate to the latest HTTP Core?
>
> Thanks,
> Hiranya
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
>