You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@synapse.apache.org by Ruwan Linton <ru...@gmail.com> on 2009/09/24 02:03:38 UTC
Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null
Hi Daniel,
This should be fixed now on the trunk, were you trying this on the trunk?
Can you please verify this on a SNAPSHOT build?
This will be available with the 1.3 release.
Thanks,
Ruwan
On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel <Da...@intuit.com>wrote:
> Hi,
>
> I've found the same problem as the one listed below - every time a
> malformed request is given to Synapse, an infinite loop will be
> triggered making the Synapse server sluggish, and consuming the entire
> log file (in case the output is redirected). I've noticed that the
> problem might get fixed. Could you please let me know if the problem was
> fixes, and if yes in which release.
>
> Thanks,
> Daniel
>
>
> Hi All
>
> I guess this boils down to how we deal with broken,
> partial, and even
> possibly malicious requests in the NIO transport.. I
> will fix the issue
> pointed to by Andreas and Murali ASAP as I consider this
> critical.
>
> Murali - I believe you should plan on updating to
> Synapse 1.3 in the
> near future. There are a few issues we have fixed over
> the past year
> that should be very important for any production
> deployment. As I
> believe your environment / configuration is
> straightforward, so the
> migration should be quite trivial too - Could you also
> raise an issue on
> the JIRA?
>
> This runtime exception is caught in
> BaseIOReactor#writable and delegated to the
> registered exception handler (in our case an anonymous
> inner class of
> HttpCoreNIOListener with the following implementation:
>
> ioReactor.setExceptionHandler(new
> IOReactorExceptionHandler() {
> public boolean handle(IOException
> ioException) {
> log.warn("System may be unstable:
> IOReactor encountered a
> checked exception : "
> + ioException.getMessage(),
> ioException);
> return true;
> }
>
> public boolean handle(RuntimeException
> runtimeException) {
> log.warn("System may be unstable:
> IOReactor encountered a
> runtime exception : "
> +
> runtimeException.getMessage(), runtimeException);
> return true;
> }
> });
>
> I wondering a bit about the return value. Taken from the
> JavaDoc of the
> interface IOReactorExceptionHandler#handle
>
> True if it is safe to ignore the exception and continue
> execution of the I/O
> reactor; if the I/O reactor must throw {@link
> RuntimeException} and terminate
>
> Hmm, is it really safe to proceed here? We do not test
> any detail of the
> RuntimeException. To me this looks wrong.
>
> Yes, we return true since the later versions of HttpCore
> dump some
> information and try to continue in the face of RT
> exceptions. What I
> feel is that we need to enhance the NIO transport to be
> stable in the
> face of broken, partial and malicious requests so that
> RT exceptions
> will not be thrown in such cases, and we will drop the
> connection with a
> WARN
>
> cheers
> asankha
>
> --
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
>
> http://esbmagic.blogspot.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-...@synapse.apache.org
> For additional commands, e-mail:
> dev-...@synapse.apache.org
>
>
--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@wso2.com; cell: +94 77 341 3097
blog: http://ruwansblog.blogspot.com
Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered
a runtime exception : null
Posted by "Asankha C. Perera" <as...@apache.org>.
Hi Mike
> Asankha - Thanks for committing the fix. The 'other' means by which this
> error comes up, is when I deploy to a server that has active keepalive
> requests hitting synapse. Unfortunately I don't know enough about these
> requests to link the problem conclusively to that activity, and also
> unfortunately, I was unable to run the tests in that particular
> environment this weekend. I'll see if I can get on the system tomorrow
> night and run tcpdump from there.
>
The possibility I could think of is a new and valid request coming over
a keepalive connection, which is just being closed by Synapse.. the
probability of which is low, unless the system is under load. The
current fix should work even in that case, but the client will get an
error, which may not be perfect, but I think possible options we have is
limited.
If possible, leave the tcpdump running and gzip the file once this
happens - else you may end up with a huge log file.. You may send the
file directly to me as the tcp dump may reveal message data if posted
publicly.
cheers
asankha
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered a runtime exception : null
Posted by "Obendorf, Mike" <Mi...@intuit.com>.
Asankha - Thanks for committing the fix. The 'other' means by which this
error comes up, is when I deploy to a server that has active keepalive
requests hitting synapse. Unfortunately I don't know enough about these
requests to link the problem conclusively to that activity, and also
unfortunately, I was unable to run the tests in that particular
environment this weekend. I'll see if I can get on the system tomorrow
night and run tcpdump from there.
Mike
-----Original Message-----
From: Asankha Perera [mailto:asankha.apache@gmail.com] On Behalf Of
Asankha C. Perera
Sent: Sunday, September 27, 2009 10:21 PM
To: user@synapse.apache.org
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null
Hi Mike/Daniel
> Thanks for the tcpdump reminder... Here is log output (nohup) and
tcdump
> (dump) from 1.2 and 1.3. You can see the infinite loop in the 1.3 log
(I
> truncated for brevity), whereas the 1.2 log just shows the error once.
>
Thanks for the log file. However, it shows that you probably telnet into
the system:
13:03:56.256981 IP 172.19.52.21.19472 > 172.16.204.44.http: P 4:6(2) ack
1 win 64390
0x0000: 4500 002a b869 4000 7c06 edfe ac13 3415 E..*.i@.|.....4.
0x0010: ac10 cc2c 4c10 0050 1b8e 4eb1 1be7 5f7a ...,L..P..N..._z
0x0020: 5018 fb86 1cd3 0000 *0d0a * P.........
I was wondering if you see this by any "other" means, during supposedly
normal interactions with a client/backend system, in which case the
issue may still need more investigation. Since the suggested fix is
trivial, and serves the purpose for now, I have commited it, but I would
still like to know if this occurs under any normal circumstances.
I've filed an enhancement for HttpCore-208 after which we could improve
Synapse to better recover from generally any unexpected error. Eric has
also suggested that its better to re-start the IOReactors on fatal
exceptions, which we will incoporate with Synapse-584
cheers
asankha
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered
a runtime exception : null
Posted by "Asankha C. Perera" <as...@apache.org>.
Hi Mike/Daniel
> Thanks for the tcpdump reminder... Here is log output (nohup) and tcdump
> (dump) from 1.2 and 1.3. You can see the infinite loop in the 1.3 log (I
> truncated for brevity), whereas the 1.2 log just shows the error once.
>
Thanks for the log file. However, it shows that you probably telnet into
the system:
13:03:56.256981 IP 172.19.52.21.19472 > 172.16.204.44.http: P 4:6(2) ack
1 win 64390
0x0000: 4500 002a b869 4000 7c06 edfe ac13 3415 E..*.i@.|.....4.
0x0010: ac10 cc2c 4c10 0050 1b8e 4eb1 1be7 5f7a ...,L..P..N..._z
0x0020: 5018 fb86 1cd3 0000 *0d0a * P.........
I was wondering if you see this by any "other" means, during supposedly
normal interactions with a client/backend system, in which case the
issue may still need more investigation. Since the suggested fix is
trivial, and serves the purpose for now, I have commited it, but I would
still like to know if this occurs under any normal circumstances.
I've filed an enhancement for HttpCore-208 after which we could improve
Synapse to better recover from generally any unexpected error. Eric has
also suggested that its better to re-start the IOReactors on fatal
exceptions, which we will incoporate with Synapse-584
cheers
asankha
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered a runtime exception : null
Posted by "Obendorf, Mike" <Mi...@intuit.com>.
Thanks for the tcpdump reminder... Here is log output (nohup) and tcdump
(dump) from 1.2 and 1.3. You can see the infinite loop in the 1.3 log (I
truncated for brevity), whereas the 1.2 log just shows the error once.
Thanks,
Mike
-----Original Message-----
From: Asankha Perera [mailto:asankha.apache@gmail.com] On Behalf Of
Asankha C. Perera
Sent: Saturday, September 26, 2009 7:47 PM
To: user@synapse.apache.org
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null
Hi Mike
> Looking at the differences between 1.2 and 1.3 it appears that
keepalive
> connection optimizations were put in place that perhaps caused this
> regression. I say regression, because I don't see the infinite loop
when
> running 1.2 synapse with the simple telnet 8280 test. Also, I did not
> see this error in any of my testing until I deployed to a box which
was
> subject to keepalive processing. I'm definitely no expert in this
area,
> so this is all just based on anecdotal evidence - I'm just throwing
> these observations out there for you to chew on...
>
There was a proposal from one of the users that we should recover from
fatal exceptions by auto-restarting the IOReactors. I think thats a good
thing as certain severe network level failures etc could otherwise put
Synapse into a non-recoverable state, in which case shutting down and
restarting automatically is better than trying to log the error and
continue in an infinite loop.
However, I cannot think why this could happen in 1.3, instead of 1.2 in
"normal" processing.. if you could provide a tcp dump (taken on Linux as
"sudo tcpdump -i eth0 -X -n -s 4096 port 8280 > dump.txt" would be very
valuable to find out the cause of this. Will it be possible to get this
log?
cheers
asankha
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered
a runtime exception : null
Posted by "Asankha C. Perera" <as...@apache.org>.
Hi Mike
> Looking at the differences between 1.2 and 1.3 it appears that keepalive
> connection optimizations were put in place that perhaps caused this
> regression. I say regression, because I don't see the infinite loop when
> running 1.2 synapse with the simple telnet 8280 test. Also, I did not
> see this error in any of my testing until I deployed to a box which was
> subject to keepalive processing. I'm definitely no expert in this area,
> so this is all just based on anecdotal evidence - I'm just throwing
> these observations out there for you to chew on...
>
There was a proposal from one of the users that we should recover from
fatal exceptions by auto-restarting the IOReactors. I think thats a good
thing as certain severe network level failures etc could otherwise put
Synapse into a non-recoverable state, in which case shutting down and
restarting automatically is better than trying to log the error and
continue in an infinite loop.
However, I cannot think why this could happen in 1.3, instead of 1.2 in
"normal" processing.. if you could provide a tcp dump (taken on Linux as
"sudo tcpdump -i eth0 -X -n -s 4096 port 8280 > dump.txt" would be very
valuable to find out the cause of this. Will it be possible to get this log?
cheers
asankha
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered a runtime exception : null
Posted by "Obendorf, Mike" <Mi...@intuit.com>.
Looking at the differences between 1.2 and 1.3 it appears that keepalive
connection optimizations were put in place that perhaps caused this
regression. I say regression, because I don't see the infinite loop when
running 1.2 synapse with the simple telnet 8280 test. Also, I did not
see this error in any of my testing until I deployed to a box which was
subject to keepalive processing. I'm definitely no expert in this area,
so this is all just based on anecdotal evidence - I'm just throwing
these observations out there for you to chew on...
Thanks,
Mike
-----Original Message-----
From: Asankha Perera [mailto:asankha.apache@gmail.com] On Behalf Of
Asankha C. Perera
Sent: Friday, September 25, 2009 8:28 PM
To: user@synapse.apache.org
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null
Hi Ruwan / Daniel
> I had a look at the issue and I think Daniels suggestion will work. I
will
> fix this soon on the 1.3 branch.
>
>
>> The file that contains the error is
>> synapse-nhttp-transport/ServerHandler.java:209
>> int bytesWritten = outBuf.produceContent(encoder);
>> where the outBuf is
>> ContentOutputBuffer outBuf = (ContentOutputBuffer)
>> context.getAttribute(RESPONSE_SOURCE_BUFFER);
>>
>> In this case when the outBuf is null, I think a possible solution to
fix
>> this is to just close the connection.
>> What do you think?
>>
Could you create a JIRA and attach a log you see? Ensure that it
captures the first time the runtime exception occurs. Is it also
possible to get a tcpdump of the communication that causes this during
normal operation - unless someone is trying to intentionally break
Synapse. Simply closing the connection here would not be the proper
solution - but something that will just cover up something else that
happened. We should use this opportunity to understand why this happens
during normal/abnormal operations and ensure that other possibilities
which may lead to this is also analyzed
cheers
asankha
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered
a runtime exception : null
Posted by "Asankha C. Perera" <as...@apache.org>.
Hi Ruwan / Daniel
> I had a look at the issue and I think Daniels suggestion will work. I will
> fix this soon on the 1.3 branch.
>
>
>> The file that contains the error is
>> synapse-nhttp-transport/ServerHandler.java:209
>> int bytesWritten = outBuf.produceContent(encoder);
>> where the outBuf is
>> ContentOutputBuffer outBuf = (ContentOutputBuffer)
>> context.getAttribute(RESPONSE_SOURCE_BUFFER);
>>
>> In this case when the outBuf is null, I think a possible solution to fix
>> this is to just close the connection.
>> What do you think?
>>
Could you create a JIRA and attach a log you see? Ensure that it
captures the first time the runtime exception occurs. Is it also
possible to get a tcpdump of the communication that causes this during
normal operation - unless someone is trying to intentionally break
Synapse. Simply closing the connection here would not be the proper
solution - but something that will just cover up something else that
happened. We should use this opportunity to understand why this happens
during normal/abnormal operations and ensure that other possibilities
which may lead to this is also analyzed
cheers
asankha
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null
Posted by Ruwan Linton <ru...@gmail.com>.
Hi Danile/Mike,
I had a look at the issue and I think Daniels suggestion will work. I will
fix this soon on the 1.3 branch.
Thanks,
Ruwan
On Sat, Sep 26, 2009 at 4:38 AM, Obendorf, Mike <Michael_Obendorf@intuit.com
> wrote:
> Daniel -
>
> Did you get a fix from Ruwan for this? We're now seeing the same error in
> one of our deployments and would like to patch that build until 1.3 is
> officially released. BTW, I just downloaded the latest 1.3 source and get
> the same error as well...
>
> Thanks,
> Mike
>
> -----Original Message-----
> From: Moise, Daniel [mailto:Daniel_Moise@Intuit.com]
> Sent: Thursday, September 24, 2009 11:02 AM
> To: user@synapse.apache.org
> Subject: RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> The file that contains the error is
> synapse-nhttp-transport/ServerHandler.java:209
> int bytesWritten = outBuf.produceContent(encoder);
> where the outBuf is
> ContentOutputBuffer outBuf = (ContentOutputBuffer)
> context.getAttribute(RESPONSE_SOURCE_BUFFER);
>
> In this case when the outBuf is null, I think a possible solution to fix
> this is to just close the connection.
> What do you think?
>
> Thanks,
> Daniel
>
>
> -----Original Message-----
> From: Ruwan Linton [mailto:ruwan.linton@gmail.com]
> Sent: Thursday, September 24, 2009 11:53 AM
> To: user@synapse.apache.org
> Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> Let me get back to you after testing this scenario. Thanks for the input
> Daniel. You may also raise a JIRA issue for this.
>
> Thanks,
> Ruwan
>
> On Thu, Sep 24, 2009 at 10:31 PM, Moise, Daniel
> <Da...@intuit.com>wrote:
>
> > Hi Ruwan,
> >
> > I've tried the last changes in the 1.3 branch, and the problem is
> still
> > there. To test it, we've just run a 'telnet localhost 8280'.
> > Based on your answer, it seems that you guys have a solution for this.
> > Could you send me a patch or the changes for that. You don't have to
> > necessarily check them in the branch for now.
> >
> > Thanks,
> > Daniel
> >
> > -----Original Message-----
> > From: Ruwan Linton [mailto:ruwan.linton@gmail.com]
> > Sent: Wednesday, September 23, 2009 6:04 PM
> > To: user@synapse.apache.org
> > Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> > encountered a runtime exception : null
> >
> > Hi Daniel,
> >
> > This should be fixed now on the trunk, were you trying this on the
> > trunk?
> > Can you please verify this on a SNAPSHOT build?
> >
> > This will be available with the 1.3 release.
> >
> > Thanks,
> > Ruwan
> >
> > On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
> > <Da...@intuit.com>wrote:
> >
> > > Hi,
> > >
> > > I've found the same problem as the one listed below - every time a
> > > malformed request is given to Synapse, an infinite loop will be
> > > triggered making the Synapse server sluggish, and consuming the
> entire
> > > log file (in case the output is redirected). I've noticed that the
> > > problem might get fixed. Could you please let me know if the problem
> > was
> > > fixes, and if yes in which release.
> > >
> > > Thanks,
> > > Daniel
> > >
> > >
> > > Hi All
> > >
> > > I guess this boils down to how we deal with broken,
> > > partial, and even
> > > possibly malicious requests in the NIO transport.. I
> > > will fix the issue
> > > pointed to by Andreas and Murali ASAP as I consider
> > this
> > > critical.
> > >
> > > Murali - I believe you should plan on updating to
> > > Synapse 1.3 in the
> > > near future. There are a few issues we have fixed
> over
> > > the past year
> > > that should be very important for any production
> > > deployment. As I
> > > believe your environment / configuration is
> > > straightforward, so the
> > > migration should be quite trivial too - Could you
> also
> > > raise an issue on
> > > the JIRA?
> > >
> > > This runtime exception is caught in
> > > BaseIOReactor#writable and delegated to the
> > > registered exception handler (in our case an
> anonymous
> > > inner class of
> > > HttpCoreNIOListener with the following
> implementation:
> > >
> > > ioReactor.setExceptionHandler(new
> > > IOReactorExceptionHandler() {
> > > public boolean handle(IOException
> > > ioException) {
> > > log.warn("System may be unstable:
> > > IOReactor encountered a
> > > checked exception : "
> > > +
> ioException.getMessage(),
> > > ioException);
> > > return true;
> > > }
> > >
> > > public boolean
> handle(RuntimeException
> > > runtimeException) {
> > > log.warn("System may be unstable:
> > > IOReactor encountered a
> > > runtime exception : "
> > > +
> > > runtimeException.getMessage(), runtimeException);
> > > return true;
> > > }
> > > });
> > >
> > > I wondering a bit about the return value. Taken from
> > the
> > > JavaDoc of the
> > > interface IOReactorExceptionHandler#handle
> > >
> > > True if it is safe to ignore the exception and
> continue
> > > execution of the I/O
> > > reactor; if the I/O reactor must throw {@link
> > > RuntimeException} and terminate
> > >
> > > Hmm, is it really safe to proceed here? We do not
> test
> > > any detail of the
> > > RuntimeException. To me this looks wrong.
> > >
> > > Yes, we return true since the later versions of
> > HttpCore
> > > dump some
> > > information and try to continue in the face of RT
> > > exceptions. What I
> > > feel is that we need to enhance the NIO transport to
> be
> > > stable in the
> > > face of broken, partial and malicious requests so
> that
> > > RT exceptions
> > > will not be thrown in such cases, and we will drop
> the
> > > connection with a
> > > WARN
> > >
> > > cheers
> > > asankha
> > >
> > > --
> > > Asankha C. Perera
> > > AdroitLogic, http://adroitlogic.org
> > >
> > > http://esbmagic.blogspot.com
> > >
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-...@synapse.apache.org
> > > For additional commands, e-mail:
> > > dev-...@synapse.apache.org
> > >
> > >
> >
> >
> > --
> > Ruwan Linton
> > Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> > WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> > email: ruwan@wso2.com; cell: +94 77 341 3097
> > blog: http://ruwansblog.blogspot.com
> >
>
>
>
> --
> Ruwan Linton
> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> email: ruwan@wso2.com; cell: +94 77 341 3097
> blog: http://ruwansblog.blogspot.com
>
--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@wso2.com; cell: +94 77 341 3097
blog: http://ruwansblog.blogspot.com
RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered a runtime exception : null
Posted by "Obendorf, Mike" <Mi...@intuit.com>.
Daniel -
Did you get a fix from Ruwan for this? We're now seeing the same error in one of our deployments and would like to patch that build until 1.3 is officially released. BTW, I just downloaded the latest 1.3 source and get the same error as well...
Thanks,
Mike
-----Original Message-----
From: Moise, Daniel [mailto:Daniel_Moise@Intuit.com]
Sent: Thursday, September 24, 2009 11:02 AM
To: user@synapse.apache.org
Subject: RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered a runtime exception : null
The file that contains the error is
synapse-nhttp-transport/ServerHandler.java:209
int bytesWritten = outBuf.produceContent(encoder);
where the outBuf is
ContentOutputBuffer outBuf = (ContentOutputBuffer)
context.getAttribute(RESPONSE_SOURCE_BUFFER);
In this case when the outBuf is null, I think a possible solution to fix
this is to just close the connection.
What do you think?
Thanks,
Daniel
-----Original Message-----
From: Ruwan Linton [mailto:ruwan.linton@gmail.com]
Sent: Thursday, September 24, 2009 11:53 AM
To: user@synapse.apache.org
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null
Let me get back to you after testing this scenario. Thanks for the input
Daniel. You may also raise a JIRA issue for this.
Thanks,
Ruwan
On Thu, Sep 24, 2009 at 10:31 PM, Moise, Daniel
<Da...@intuit.com>wrote:
> Hi Ruwan,
>
> I've tried the last changes in the 1.3 branch, and the problem is
still
> there. To test it, we've just run a 'telnet localhost 8280'.
> Based on your answer, it seems that you guys have a solution for this.
> Could you send me a patch or the changes for that. You don't have to
> necessarily check them in the branch for now.
>
> Thanks,
> Daniel
>
> -----Original Message-----
> From: Ruwan Linton [mailto:ruwan.linton@gmail.com]
> Sent: Wednesday, September 23, 2009 6:04 PM
> To: user@synapse.apache.org
> Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> Hi Daniel,
>
> This should be fixed now on the trunk, were you trying this on the
> trunk?
> Can you please verify this on a SNAPSHOT build?
>
> This will be available with the 1.3 release.
>
> Thanks,
> Ruwan
>
> On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
> <Da...@intuit.com>wrote:
>
> > Hi,
> >
> > I've found the same problem as the one listed below - every time a
> > malformed request is given to Synapse, an infinite loop will be
> > triggered making the Synapse server sluggish, and consuming the
entire
> > log file (in case the output is redirected). I've noticed that the
> > problem might get fixed. Could you please let me know if the problem
> was
> > fixes, and if yes in which release.
> >
> > Thanks,
> > Daniel
> >
> >
> > Hi All
> >
> > I guess this boils down to how we deal with broken,
> > partial, and even
> > possibly malicious requests in the NIO transport.. I
> > will fix the issue
> > pointed to by Andreas and Murali ASAP as I consider
> this
> > critical.
> >
> > Murali - I believe you should plan on updating to
> > Synapse 1.3 in the
> > near future. There are a few issues we have fixed
over
> > the past year
> > that should be very important for any production
> > deployment. As I
> > believe your environment / configuration is
> > straightforward, so the
> > migration should be quite trivial too - Could you
also
> > raise an issue on
> > the JIRA?
> >
> > This runtime exception is caught in
> > BaseIOReactor#writable and delegated to the
> > registered exception handler (in our case an
anonymous
> > inner class of
> > HttpCoreNIOListener with the following
implementation:
> >
> > ioReactor.setExceptionHandler(new
> > IOReactorExceptionHandler() {
> > public boolean handle(IOException
> > ioException) {
> > log.warn("System may be unstable:
> > IOReactor encountered a
> > checked exception : "
> > +
ioException.getMessage(),
> > ioException);
> > return true;
> > }
> >
> > public boolean
handle(RuntimeException
> > runtimeException) {
> > log.warn("System may be unstable:
> > IOReactor encountered a
> > runtime exception : "
> > +
> > runtimeException.getMessage(), runtimeException);
> > return true;
> > }
> > });
> >
> > I wondering a bit about the return value. Taken from
> the
> > JavaDoc of the
> > interface IOReactorExceptionHandler#handle
> >
> > True if it is safe to ignore the exception and
continue
> > execution of the I/O
> > reactor; if the I/O reactor must throw {@link
> > RuntimeException} and terminate
> >
> > Hmm, is it really safe to proceed here? We do not
test
> > any detail of the
> > RuntimeException. To me this looks wrong.
> >
> > Yes, we return true since the later versions of
> HttpCore
> > dump some
> > information and try to continue in the face of RT
> > exceptions. What I
> > feel is that we need to enhance the NIO transport to
be
> > stable in the
> > face of broken, partial and malicious requests so
that
> > RT exceptions
> > will not be thrown in such cases, and we will drop
the
> > connection with a
> > WARN
> >
> > cheers
> > asankha
> >
> > --
> > Asankha C. Perera
> > AdroitLogic, http://adroitlogic.org
> >
> > http://esbmagic.blogspot.com
> >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-...@synapse.apache.org
> > For additional commands, e-mail:
> > dev-...@synapse.apache.org
> >
> >
>
>
> --
> Ruwan Linton
> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> email: ruwan@wso2.com; cell: +94 77 341 3097
> blog: http://ruwansblog.blogspot.com
>
--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@wso2.com; cell: +94 77 341 3097
blog: http://ruwansblog.blogspot.com
RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered a runtime exception : null
Posted by "Moise, Daniel" <Da...@Intuit.com>.
The file that contains the error is
synapse-nhttp-transport/ServerHandler.java:209
int bytesWritten = outBuf.produceContent(encoder);
where the outBuf is
ContentOutputBuffer outBuf = (ContentOutputBuffer)
context.getAttribute(RESPONSE_SOURCE_BUFFER);
In this case when the outBuf is null, I think a possible solution to fix
this is to just close the connection.
What do you think?
Thanks,
Daniel
-----Original Message-----
From: Ruwan Linton [mailto:ruwan.linton@gmail.com]
Sent: Thursday, September 24, 2009 11:53 AM
To: user@synapse.apache.org
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null
Let me get back to you after testing this scenario. Thanks for the input
Daniel. You may also raise a JIRA issue for this.
Thanks,
Ruwan
On Thu, Sep 24, 2009 at 10:31 PM, Moise, Daniel
<Da...@intuit.com>wrote:
> Hi Ruwan,
>
> I've tried the last changes in the 1.3 branch, and the problem is
still
> there. To test it, we've just run a 'telnet localhost 8280'.
> Based on your answer, it seems that you guys have a solution for this.
> Could you send me a patch or the changes for that. You don't have to
> necessarily check them in the branch for now.
>
> Thanks,
> Daniel
>
> -----Original Message-----
> From: Ruwan Linton [mailto:ruwan.linton@gmail.com]
> Sent: Wednesday, September 23, 2009 6:04 PM
> To: user@synapse.apache.org
> Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> Hi Daniel,
>
> This should be fixed now on the trunk, were you trying this on the
> trunk?
> Can you please verify this on a SNAPSHOT build?
>
> This will be available with the 1.3 release.
>
> Thanks,
> Ruwan
>
> On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
> <Da...@intuit.com>wrote:
>
> > Hi,
> >
> > I've found the same problem as the one listed below - every time a
> > malformed request is given to Synapse, an infinite loop will be
> > triggered making the Synapse server sluggish, and consuming the
entire
> > log file (in case the output is redirected). I've noticed that the
> > problem might get fixed. Could you please let me know if the problem
> was
> > fixes, and if yes in which release.
> >
> > Thanks,
> > Daniel
> >
> >
> > Hi All
> >
> > I guess this boils down to how we deal with broken,
> > partial, and even
> > possibly malicious requests in the NIO transport.. I
> > will fix the issue
> > pointed to by Andreas and Murali ASAP as I consider
> this
> > critical.
> >
> > Murali - I believe you should plan on updating to
> > Synapse 1.3 in the
> > near future. There are a few issues we have fixed
over
> > the past year
> > that should be very important for any production
> > deployment. As I
> > believe your environment / configuration is
> > straightforward, so the
> > migration should be quite trivial too - Could you
also
> > raise an issue on
> > the JIRA?
> >
> > This runtime exception is caught in
> > BaseIOReactor#writable and delegated to the
> > registered exception handler (in our case an
anonymous
> > inner class of
> > HttpCoreNIOListener with the following
implementation:
> >
> > ioReactor.setExceptionHandler(new
> > IOReactorExceptionHandler() {
> > public boolean handle(IOException
> > ioException) {
> > log.warn("System may be unstable:
> > IOReactor encountered a
> > checked exception : "
> > +
ioException.getMessage(),
> > ioException);
> > return true;
> > }
> >
> > public boolean
handle(RuntimeException
> > runtimeException) {
> > log.warn("System may be unstable:
> > IOReactor encountered a
> > runtime exception : "
> > +
> > runtimeException.getMessage(), runtimeException);
> > return true;
> > }
> > });
> >
> > I wondering a bit about the return value. Taken from
> the
> > JavaDoc of the
> > interface IOReactorExceptionHandler#handle
> >
> > True if it is safe to ignore the exception and
continue
> > execution of the I/O
> > reactor; if the I/O reactor must throw {@link
> > RuntimeException} and terminate
> >
> > Hmm, is it really safe to proceed here? We do not
test
> > any detail of the
> > RuntimeException. To me this looks wrong.
> >
> > Yes, we return true since the later versions of
> HttpCore
> > dump some
> > information and try to continue in the face of RT
> > exceptions. What I
> > feel is that we need to enhance the NIO transport to
be
> > stable in the
> > face of broken, partial and malicious requests so
that
> > RT exceptions
> > will not be thrown in such cases, and we will drop
the
> > connection with a
> > WARN
> >
> > cheers
> > asankha
> >
> > --
> > Asankha C. Perera
> > AdroitLogic, http://adroitlogic.org
> >
> > http://esbmagic.blogspot.com
> >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-...@synapse.apache.org
> > For additional commands, e-mail:
> > dev-...@synapse.apache.org
> >
> >
>
>
> --
> Ruwan Linton
> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> email: ruwan@wso2.com; cell: +94 77 341 3097
> blog: http://ruwansblog.blogspot.com
>
--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@wso2.com; cell: +94 77 341 3097
blog: http://ruwansblog.blogspot.com
Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null
Posted by Ruwan Linton <ru...@gmail.com>.
Let me get back to you after testing this scenario. Thanks for the input
Daniel. You may also raise a JIRA issue for this.
Thanks,
Ruwan
On Thu, Sep 24, 2009 at 10:31 PM, Moise, Daniel <Da...@intuit.com>wrote:
> Hi Ruwan,
>
> I've tried the last changes in the 1.3 branch, and the problem is still
> there. To test it, we've just run a 'telnet localhost 8280'.
> Based on your answer, it seems that you guys have a solution for this.
> Could you send me a patch or the changes for that. You don't have to
> necessarily check them in the branch for now.
>
> Thanks,
> Daniel
>
> -----Original Message-----
> From: Ruwan Linton [mailto:ruwan.linton@gmail.com]
> Sent: Wednesday, September 23, 2009 6:04 PM
> To: user@synapse.apache.org
> Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
> encountered a runtime exception : null
>
> Hi Daniel,
>
> This should be fixed now on the trunk, were you trying this on the
> trunk?
> Can you please verify this on a SNAPSHOT build?
>
> This will be available with the 1.3 release.
>
> Thanks,
> Ruwan
>
> On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
> <Da...@intuit.com>wrote:
>
> > Hi,
> >
> > I've found the same problem as the one listed below - every time a
> > malformed request is given to Synapse, an infinite loop will be
> > triggered making the Synapse server sluggish, and consuming the entire
> > log file (in case the output is redirected). I've noticed that the
> > problem might get fixed. Could you please let me know if the problem
> was
> > fixes, and if yes in which release.
> >
> > Thanks,
> > Daniel
> >
> >
> > Hi All
> >
> > I guess this boils down to how we deal with broken,
> > partial, and even
> > possibly malicious requests in the NIO transport.. I
> > will fix the issue
> > pointed to by Andreas and Murali ASAP as I consider
> this
> > critical.
> >
> > Murali - I believe you should plan on updating to
> > Synapse 1.3 in the
> > near future. There are a few issues we have fixed over
> > the past year
> > that should be very important for any production
> > deployment. As I
> > believe your environment / configuration is
> > straightforward, so the
> > migration should be quite trivial too - Could you also
> > raise an issue on
> > the JIRA?
> >
> > This runtime exception is caught in
> > BaseIOReactor#writable and delegated to the
> > registered exception handler (in our case an anonymous
> > inner class of
> > HttpCoreNIOListener with the following implementation:
> >
> > ioReactor.setExceptionHandler(new
> > IOReactorExceptionHandler() {
> > public boolean handle(IOException
> > ioException) {
> > log.warn("System may be unstable:
> > IOReactor encountered a
> > checked exception : "
> > + ioException.getMessage(),
> > ioException);
> > return true;
> > }
> >
> > public boolean handle(RuntimeException
> > runtimeException) {
> > log.warn("System may be unstable:
> > IOReactor encountered a
> > runtime exception : "
> > +
> > runtimeException.getMessage(), runtimeException);
> > return true;
> > }
> > });
> >
> > I wondering a bit about the return value. Taken from
> the
> > JavaDoc of the
> > interface IOReactorExceptionHandler#handle
> >
> > True if it is safe to ignore the exception and continue
> > execution of the I/O
> > reactor; if the I/O reactor must throw {@link
> > RuntimeException} and terminate
> >
> > Hmm, is it really safe to proceed here? We do not test
> > any detail of the
> > RuntimeException. To me this looks wrong.
> >
> > Yes, we return true since the later versions of
> HttpCore
> > dump some
> > information and try to continue in the face of RT
> > exceptions. What I
> > feel is that we need to enhance the NIO transport to be
> > stable in the
> > face of broken, partial and malicious requests so that
> > RT exceptions
> > will not be thrown in such cases, and we will drop the
> > connection with a
> > WARN
> >
> > cheers
> > asankha
> >
> > --
> > Asankha C. Perera
> > AdroitLogic, http://adroitlogic.org
> >
> > http://esbmagic.blogspot.com
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-...@synapse.apache.org
> > For additional commands, e-mail:
> > dev-...@synapse.apache.org
> >
> >
>
>
> --
> Ruwan Linton
> Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
> WSO2 <http://wso2.org/esb%0AWSO2> Inc.; http://wso2.org
> email: ruwan@wso2.com; cell: +94 77 341 3097
> blog: http://ruwansblog.blogspot.com
>
--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@wso2.com; cell: +94 77 341 3097
blog: http://ruwansblog.blogspot.com
RE: HttpCoreNIOListener$1 - System may be unstable: IOReactor encountered a runtime exception : null
Posted by "Moise, Daniel" <Da...@Intuit.com>.
Hi Ruwan,
I've tried the last changes in the 1.3 branch, and the problem is still
there. To test it, we've just run a 'telnet localhost 8280'.
Based on your answer, it seems that you guys have a solution for this.
Could you send me a patch or the changes for that. You don't have to
necessarily check them in the branch for now.
Thanks,
Daniel
-----Original Message-----
From: Ruwan Linton [mailto:ruwan.linton@gmail.com]
Sent: Wednesday, September 23, 2009 6:04 PM
To: user@synapse.apache.org
Subject: Re: HttpCoreNIOListener$1 - System may be unstable: IOReactor
encountered a runtime exception : null
Hi Daniel,
This should be fixed now on the trunk, were you trying this on the
trunk?
Can you please verify this on a SNAPSHOT build?
This will be available with the 1.3 release.
Thanks,
Ruwan
On Wed, Sep 23, 2009 at 11:24 PM, Moise, Daniel
<Da...@intuit.com>wrote:
> Hi,
>
> I've found the same problem as the one listed below - every time a
> malformed request is given to Synapse, an infinite loop will be
> triggered making the Synapse server sluggish, and consuming the entire
> log file (in case the output is redirected). I've noticed that the
> problem might get fixed. Could you please let me know if the problem
was
> fixes, and if yes in which release.
>
> Thanks,
> Daniel
>
>
> Hi All
>
> I guess this boils down to how we deal with broken,
> partial, and even
> possibly malicious requests in the NIO transport.. I
> will fix the issue
> pointed to by Andreas and Murali ASAP as I consider
this
> critical.
>
> Murali - I believe you should plan on updating to
> Synapse 1.3 in the
> near future. There are a few issues we have fixed over
> the past year
> that should be very important for any production
> deployment. As I
> believe your environment / configuration is
> straightforward, so the
> migration should be quite trivial too - Could you also
> raise an issue on
> the JIRA?
>
> This runtime exception is caught in
> BaseIOReactor#writable and delegated to the
> registered exception handler (in our case an anonymous
> inner class of
> HttpCoreNIOListener with the following implementation:
>
> ioReactor.setExceptionHandler(new
> IOReactorExceptionHandler() {
> public boolean handle(IOException
> ioException) {
> log.warn("System may be unstable:
> IOReactor encountered a
> checked exception : "
> + ioException.getMessage(),
> ioException);
> return true;
> }
>
> public boolean handle(RuntimeException
> runtimeException) {
> log.warn("System may be unstable:
> IOReactor encountered a
> runtime exception : "
> +
> runtimeException.getMessage(), runtimeException);
> return true;
> }
> });
>
> I wondering a bit about the return value. Taken from
the
> JavaDoc of the
> interface IOReactorExceptionHandler#handle
>
> True if it is safe to ignore the exception and continue
> execution of the I/O
> reactor; if the I/O reactor must throw {@link
> RuntimeException} and terminate
>
> Hmm, is it really safe to proceed here? We do not test
> any detail of the
> RuntimeException. To me this looks wrong.
>
> Yes, we return true since the later versions of
HttpCore
> dump some
> information and try to continue in the face of RT
> exceptions. What I
> feel is that we need to enhance the NIO transport to be
> stable in the
> face of broken, partial and malicious requests so that
> RT exceptions
> will not be thrown in such cases, and we will drop the
> connection with a
> WARN
>
> cheers
> asankha
>
> --
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
>
> http://esbmagic.blogspot.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-...@synapse.apache.org
> For additional commands, e-mail:
> dev-...@synapse.apache.org
>
>
--
Ruwan Linton
Technical Lead & Product Manager; WSO2 ESB; http://wso2.org/esb
WSO2 Inc.; http://wso2.org
email: ruwan@wso2.com; cell: +94 77 341 3097
blog: http://ruwansblog.blogspot.com