You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@mina.apache.org by Dhruv Chopra <dh...@shephertz.co.in> on 2012/11/28 16:21:30 UTC
MINA filter chains breaking under stress
Hi,
My MINA server has a 2 filter chain (A and B - where output of encoder A is
further encoded by B and then sent out on the wire). Now everything works
fine under normal load (20 clients with incoming 20 msgs/sec and outgoing
100 msg/sec). But when I increase the load - I randomly start getting
exceptions with cause *"don't know how **to handle message of type 'X'**.
Are you missing a protocol encoder?"*. This message of type 'X' is the
output of my filter A's encoder and input for filter B's encoder. I added
a bunch of logs and see that in the failure case, my filter B never
received the output of filter A and this exception followed. I also logged
the threadids and see that when this occurred, thread 't' had encoded the
message through filter A and then started encoding another message through
filter A. Then the exception occurs and same Thread 't' continues encoding
the other message through filter B. So its like it completely forgot to
encode the earlier message through filter B. In the good cases, thread 't'
would after encoding an object through encoder A always encode that through
encoder B before doing anything else.
Another thing is that when I only have one filter - even under higher
stress, things work fine.
Sorry for the longish description - hope someone can help me identify whats
going wrong.
Thanks!
Dhruv Chopra
Here is the stack in case it helps -
java.lang.IllegalStateException: Don't know how to handle message of type
'X'. Are you missing a protocol encoder?
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.flushNow(AbstractPollingIoProcessor.java:902)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.flush(AbstractPollingIoProcessor.java:799)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$700(AbstractPollingIoProcessor.java:68)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1146)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Re: MINA filter chains breaking under stress
Posted by Max Larsson <ma...@facilityboss.biz>.
I am aware of that the filterchain is not thread safe. As long I understood
Mina the encode/decode methods can be called by various threads, but it
Will always be the same thread for the same session.
For example, we have 3 session and 2 Threads, if session 1 is supplied
to the encode method it will always be called by Thread 1. If session 2
is supplied, it will be thread 2. But a thread can be used for multiple session,
e.g. session 3 is process by thread 1 again (in this example). Am I right?
On 14.01.19, 21:13, "Jonathan Valliere" <jo...@emoten.com> wrote:
Upon a quick review of your email, I have the following notes.
The filterchain is not thread safe. You have to write threadsafe filters.
You must also make your encoded message system threadsafe. Just as with
SSL the actual encoding process must occur within a synchronized block.
As far as ProtocolCodecFilter goes, I will have to look into that.
On Mon, Jan 14, 2019 at 8:37 AM Max Larsson <ma...@facilityboss.biz>
wrote:
> I don't yet have a test case for the issues, but i have a description from
> a
> colleague in German under which circumstances this behavior occurs using
> Apache Mina 2.0.19:
>
> - There must be multiple Threads (at least two) calling nearly
> simultaneously session.write and there must be multiple ProtocolCodecFilter
> in the FilterChain, whereas the first encoder doesn't produce messages of
> type IoBuffer
>
> - For example if thread A calls session.write and the first
> ProtocolCodecFilter has processed the write request, there will be an
> encoded message in the WriteRequestQueue of the session. Lets assume that
> it
> is an byte array.
>
> - Now if another thread B even called session.write in such circumstances
> that the call of thread B has passed all FilterChain and it is in progress
> to call AbstractPollingIoProcessor$Processor.flushNow(), the encoded
> message
> from thread A (which still is in process to complete FilterChain) result in
> the known error "Don't know how to handle message of type 'xyz'. Are you
> missing a protocol encoder?'
>
> - all following calls to session.write (regardless which thread) will even
> produce the error, because the following snippet (somewhere in mina):
>
> // Check for pending writes.
> req = session.getCurrentWriteRequest();
> if (req == null) {
> req = writeRequestQueue.poll(session);
> if (req == null) {
> break;
> }
> session.setCurrentWriteRequest(req);
> }
>
> always return the same incomplete encoded message at line 'req =
> session.getCurrentWriteRequest();',
> because it isn't removed from initial first faulty processing.
>
>
> I hope someone understand whats going on, and if someone could tell me, if
> it is a bug in mina or a faulty
> usage of mina.
>
> best regards
>
> Max Larsson
>
>
>
> --
> Sent from:
> http://apache-mina.10907.n7.nabble.com/Apache-MINA-User-Forum-f31345.html
>
--
CONFIDENTIALITY NOTICE: The contents of this email message and any
attachments are intended solely for the addressee(s) and may contain
confidential and/or privileged information and may be legally protected
from disclosure.
RE: MINA filter chains breaking under stress
Posted by Dhananjay Kandari <DK...@TechMahindra.com>.
Thanks you all for your reply.
I managed to resolve it, actually mina functionality is that if it receive less bytes to read it tries to shrink the read buffer size, and when next line is longer then read buffer size at that time it reads data in two chunks
I just set the min read buffer size as max and that solve the issue
-----Original Message-----
From: Jonathan Valliere [mailto:jon.valliere@emoten.com]
Sent: Tuesday, January 15, 2019 1:43 AM
To: users@mina.apache.org
Subject: Re: MINA filter chains breaking under stress
Upon a quick review of your email, I have the following notes.
The filterchain is not thread safe. You have to write threadsafe filters.
You must also make your encoded message system threadsafe. Just as with SSL the actual encoding process must occur within a synchronized block.
As far as ProtocolCodecFilter goes, I will have to look into that.
On Mon, Jan 14, 2019 at 8:37 AM Max Larsson <ma...@facilityboss.biz>
wrote:
> I don't yet have a test case for the issues, but i have a description
> from a colleague in German under which circumstances this behavior
> occurs using Apache Mina 2.0.19:
>
> - There must be multiple Threads (at least two) calling nearly
> simultaneously session.write and there must be multiple
> ProtocolCodecFilter in the FilterChain, whereas the first encoder
> doesn't produce messages of type IoBuffer
>
> - For example if thread A calls session.write and the first
> ProtocolCodecFilter has processed the write request, there will be an
> encoded message in the WriteRequestQueue of the session. Lets assume
> that it is an byte array.
>
> - Now if another thread B even called session.write in such
> circumstances that the call of thread B has passed all FilterChain and
> it is in progress to call
> AbstractPollingIoProcessor$Processor.flushNow(), the encoded message
> from thread A (which still is in process to complete FilterChain)
> result in the known error "Don't know how to handle message of type
> 'xyz'. Are you missing a protocol encoder?'
>
> - all following calls to session.write (regardless which thread) will
> even produce the error, because the following snippet (somewhere in mina):
>
> // Check for pending writes.
> req = session.getCurrentWriteRequest();
> if (req == null) {
> req = writeRequestQueue.poll(session);
> if (req == null) {
> break;
> }
> session.setCurrentWriteRequest(req);
> }
>
> always return the same incomplete encoded message at line 'req =
> session.getCurrentWriteRequest();',
> because it isn't removed from initial first faulty processing.
>
>
> I hope someone understand whats going on, and if someone could tell
> me, if it is a bug in mina or a faulty usage of mina.
>
> best regards
>
> Max Larsson
>
>
>
> --
> Sent from:
> http://apache-mina.10907.n7.nabble.com/Apache-MINA-User-Forum-f31345.h
> tml
>
--
CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure.
============================================================================================================================
Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html <http://www.techmahindra.com/Disclaimer.html> externally http://tim.techmahindra.com/tim/disclaimer.html <http://tim.techmahindra.com/tim/disclaimer.html> internally within TechMahindra.
============================================================================================================================
Re: MINA filter chains breaking under stress
Posted by Jonathan Valliere <jo...@emoten.com>.
Upon a quick review of your email, I have the following notes.
The filterchain is not thread safe. You have to write threadsafe filters.
You must also make your encoded message system threadsafe. Just as with
SSL the actual encoding process must occur within a synchronized block.
As far as ProtocolCodecFilter goes, I will have to look into that.
On Mon, Jan 14, 2019 at 8:37 AM Max Larsson <ma...@facilityboss.biz>
wrote:
> I don't yet have a test case for the issues, but i have a description from
> a
> colleague in German under which circumstances this behavior occurs using
> Apache Mina 2.0.19:
>
> - There must be multiple Threads (at least two) calling nearly
> simultaneously session.write and there must be multiple ProtocolCodecFilter
> in the FilterChain, whereas the first encoder doesn't produce messages of
> type IoBuffer
>
> - For example if thread A calls session.write and the first
> ProtocolCodecFilter has processed the write request, there will be an
> encoded message in the WriteRequestQueue of the session. Lets assume that
> it
> is an byte array.
>
> - Now if another thread B even called session.write in such circumstances
> that the call of thread B has passed all FilterChain and it is in progress
> to call AbstractPollingIoProcessor$Processor.flushNow(), the encoded
> message
> from thread A (which still is in process to complete FilterChain) result in
> the known error "Don't know how to handle message of type 'xyz'. Are you
> missing a protocol encoder?'
>
> - all following calls to session.write (regardless which thread) will even
> produce the error, because the following snippet (somewhere in mina):
>
> // Check for pending writes.
> req = session.getCurrentWriteRequest();
> if (req == null) {
> req = writeRequestQueue.poll(session);
> if (req == null) {
> break;
> }
> session.setCurrentWriteRequest(req);
> }
>
> always return the same incomplete encoded message at line 'req =
> session.getCurrentWriteRequest();',
> because it isn't removed from initial first faulty processing.
>
>
> I hope someone understand whats going on, and if someone could tell me, if
> it is a bug in mina or a faulty
> usage of mina.
>
> best regards
>
> Max Larsson
>
>
>
> --
> Sent from:
> http://apache-mina.10907.n7.nabble.com/Apache-MINA-User-Forum-f31345.html
>
--
CONFIDENTIALITY NOTICE: The contents of this email message and any
attachments are intended solely for the addressee(s) and may contain
confidential and/or privileged information and may be legally protected
from disclosure.
Re: MINA filter chains breaking under stress
Posted by Max Larsson <ma...@facilityboss.biz>.
I don't yet have a test case for the issues, but i have a description from a
colleague in German under which circumstances this behavior occurs using
Apache Mina 2.0.19:
- There must be multiple Threads (at least two) calling nearly
simultaneously session.write and there must be multiple ProtocolCodecFilter
in the FilterChain, whereas the first encoder doesn't produce messages of
type IoBuffer
- For example if thread A calls session.write and the first
ProtocolCodecFilter has processed the write request, there will be an
encoded message in the WriteRequestQueue of the session. Lets assume that it
is an byte array.
- Now if another thread B even called session.write in such circumstances
that the call of thread B has passed all FilterChain and it is in progress
to call AbstractPollingIoProcessor$Processor.flushNow(), the encoded message
from thread A (which still is in process to complete FilterChain) result in
the known error "Don't know how to handle message of type 'xyz'. Are you
missing a protocol encoder?'
- all following calls to session.write (regardless which thread) will even
produce the error, because the following snippet (somewhere in mina):
// Check for pending writes.
req = session.getCurrentWriteRequest();
if (req == null) {
req = writeRequestQueue.poll(session);
if (req == null) {
break;
}
session.setCurrentWriteRequest(req);
}
always return the same incomplete encoded message at line 'req =
session.getCurrentWriteRequest();',
because it isn't removed from initial first faulty processing.
I hope someone understand whats going on, and if someone could tell me, if
it is a bug in mina or a faulty
usage of mina.
best regards
Max Larsson
--
Sent from: http://apache-mina.10907.n7.nabble.com/Apache-MINA-User-Forum-f31345.html
Re: MINA filter chains breaking under stress
Posted by Dhruv Chopra <dh...@shephertz.co.in>.
Hi Emmanuel,
It won't be possible for me to provide with a reproducible test setup *at
the moment*.. Unfortunately our business logic is quite tightly integrated
with MINA constructs (encoder/decoder/handler/session etc.) and it will not
be easy to separate. However once I get some time off, I will look in to it.
-Dhruv
On Thu, Nov 29, 2012 at 2:19 PM, Emmanuel Lécharny <el...@gmail.com>wrote:
> Le 11/29/12 5:16 AM, Dhruv Chopra a écrit :
> > So I tried acquiring the lock (added a new session attribute in
> > sessionCreated as suggested) when the encoder A starts encoding and
> unlock
> > after encoder B has written its output. However that didn't fix the
> > problem. In fact I then also some exceptions where a thread was trying to
> > unlock a resource it didn't own. I suppose it means that its not
> guaranteed
> > that the same thread will encode a message through all the filters.
> > Finally I synchronized the method that writes to a session in IoHandler
> > itself and the problem appears to have gone. I was able to do its since I
> > was already maintaining a wrapper around each IoSesion for some business
> > logic - and everywhere I needed to write to a session, it was going
> through
> > this wrapper.
>
> Ok, cool to see you have found a workaround. Altough it might be a
> contention point...
>
> It would be interesting for us to be able to have a reproductible
> example we can play with, if you can provide it (of course, with all the
> business part removed).
> Is that something possible ?
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: MINA filter chains breaking under stress
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/29/12 5:16 AM, Dhruv Chopra a écrit :
> So I tried acquiring the lock (added a new session attribute in
> sessionCreated as suggested) when the encoder A starts encoding and unlock
> after encoder B has written its output. However that didn't fix the
> problem. In fact I then also some exceptions where a thread was trying to
> unlock a resource it didn't own. I suppose it means that its not guaranteed
> that the same thread will encode a message through all the filters.
> Finally I synchronized the method that writes to a session in IoHandler
> itself and the problem appears to have gone. I was able to do its since I
> was already maintaining a wrapper around each IoSesion for some business
> logic - and everywhere I needed to write to a session, it was going through
> this wrapper.
Ok, cool to see you have found a workaround. Altough it might be a
contention point...
It would be interesting for us to be able to have a reproductible
example we can play with, if you can provide it (of course, with all the
business part removed).
Is that something possible ?
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: MINA filter chains breaking under stress
Posted by Dhruv Chopra <dh...@shephertz.co.in>.
So I tried acquiring the lock (added a new session attribute in
sessionCreated as suggested) when the encoder A starts encoding and unlock
after encoder B has written its output. However that didn't fix the
problem. In fact I then also some exceptions where a thread was trying to
unlock a resource it didn't own. I suppose it means that its not guaranteed
that the same thread will encode a message through all the filters.
Finally I synchronized the method that writes to a session in IoHandler
itself and the problem appears to have gone. I was able to do its since I
was already maintaining a wrapper around each IoSesion for some business
logic - and everywhere I needed to write to a session, it was going through
this wrapper.
On Wed, Nov 28, 2012 at 9:50 PM, Emmanuel Lécharny <el...@gmail.com>wrote:
> Le 11/28/12 5:10 PM, Dhruv Chopra a écrit :
> > It's difficult to tell what's going wrong without more information. Can
> > you tell us more ?
> > *- The use case is that the IoHandler upon receipt of a message,
> broadcasts
> > a message of type 'K' to a set of clients. It does so by looping through
> > all the IoSessions belonging to that set of clients and calling write on
> > them with this object of type 'K'. Encoder A accepts messages of type 'K'
> > as input. Encoder A then encodes the message into type 'X' and writes it
> to
> > the given ProtocolEncoderOutput for encoder B to consume (As described
> > earlier). Anything specific you would like me to tell you? Or what else I
> > could log that could help.*
>
> If you do that, under heavy load, it's likely that a client from another
> session sends a message, and tries to write. If those two operations
> (your own thread injecting some message in another session's write
> queue, and the session's handling an incoming message) are to happen at
> the same time, you may have some collision.
>
> May be adding a shared lock in the session's attributes which is hold by
> your thread when you write could help...
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: MINA filter chains breaking under stress
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/28/12 5:10 PM, Dhruv Chopra a écrit :
> It's difficult to tell what's going wrong without more information. Can
> you tell us more ?
> *- The use case is that the IoHandler upon receipt of a message, broadcasts
> a message of type 'K' to a set of clients. It does so by looping through
> all the IoSessions belonging to that set of clients and calling write on
> them with this object of type 'K'. Encoder A accepts messages of type 'K'
> as input. Encoder A then encodes the message into type 'X' and writes it to
> the given ProtocolEncoderOutput for encoder B to consume (As described
> earlier). Anything specific you would like me to tell you? Or what else I
> could log that could help.*
If you do that, under heavy load, it's likely that a client from another
session sends a message, and tries to write. If those two operations
(your own thread injecting some message in another session's write
queue, and the session's handling an incoming message) are to happen at
the same time, you may have some collision.
May be adding a shared lock in the session's attributes which is hold by
your thread when you write could help...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: MINA filter chains breaking under stress
Posted by Dhruv Chopra <dh...@shephertz.co.in>.
Do you have an executor filter in your chain ?
*- No, both the filters extend ProtocolEncoderAdapter*
Otherwise, it seems you have enqueued a mssage which is not an IoBuffer,
and this can't be sent. Like if encoder A directly pushes the message
instead of going through the second encoder.
*- The encoder A always writes to the ProtocolEncoderOutput out (given as
argument in the encode
method<http://www.ostools.net/uploads/apidocs/mina/org/apache/mina/filter/codec/ProtocolEncoder.html#encode(org.apache.mina.core.session.IoSession,
java.lang.Object, org.apache.mina.filter.codec.ProtocolEncoderOutput)>)*
It's difficult to tell what's going wrong without more information. Can
you tell us more ?
*- The use case is that the IoHandler upon receipt of a message, broadcasts
a message of type 'K' to a set of clients. It does so by looping through
all the IoSessions belonging to that set of clients and calling write on
them with this object of type 'K'. Encoder A accepts messages of type 'K'
as input. Encoder A then encodes the message into type 'X' and writes it to
the given ProtocolEncoderOutput for encoder B to consume (As described
earlier). Anything specific you would like me to tell you? Or what else I
could log that could help.*
*
*
-Dhruv
On Wed, Nov 28, 2012 at 9:00 PM, Emmanuel Lécharny <el...@gmail.com>wrote:
> Le 11/28/12 4:21 PM, Dhruv Chopra a écrit :
> > Hi,
> Hi,
> >
> > My MINA server has a 2 filter chain (A and B - where output of encoder A
> is
> > further encoded by B and then sent out on the wire). Now everything works
> > fine under normal load (20 clients with incoming 20 msgs/sec and outgoing
> > 100 msg/sec). But when I increase the load - I randomly start getting
> > exceptions with cause *"don't know how **to handle message of type 'X'**.
> > Are you missing a protocol encoder?"*. This message of type 'X' is the
> > output of my filter A's encoder and input for filter B's encoder. I added
> > a bunch of logs and see that in the failure case, my filter B never
> > received the output of filter A and this exception followed. I also
> logged
> > the threadids and see that when this occurred, thread 't' had encoded the
> > message through filter A and then started encoding another message
> through
> > filter A. Then the exception occurs and same Thread 't' continues
> encoding
> > the other message through filter B. So its like it completely forgot to
> > encode the earlier message through filter B. In the good cases, thread
> 't'
> > would after encoding an object through encoder A always encode that
> through
> > encoder B before doing anything else.
> >
> > Another thing is that when I only have one filter - even under higher
> > stress, things work fine.
>
> Do you have an executor filter in your chain ?
>
> Otherwise, it seems you have enqueued a mssage which is not an IoBuffer,
> and this can't be sent. Like if encoder A directly pushes the message
> instead of going through the second encoder.
>
> It's difficult to tell what's going wrong without more information. Can
> you tell us more ?
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
Re: MINA filter chains breaking under stress
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 11/28/12 4:21 PM, Dhruv Chopra a écrit :
> Hi,
Hi,
>
> My MINA server has a 2 filter chain (A and B - where output of encoder A is
> further encoded by B and then sent out on the wire). Now everything works
> fine under normal load (20 clients with incoming 20 msgs/sec and outgoing
> 100 msg/sec). But when I increase the load - I randomly start getting
> exceptions with cause *"don't know how **to handle message of type 'X'**.
> Are you missing a protocol encoder?"*. This message of type 'X' is the
> output of my filter A's encoder and input for filter B's encoder. I added
> a bunch of logs and see that in the failure case, my filter B never
> received the output of filter A and this exception followed. I also logged
> the threadids and see that when this occurred, thread 't' had encoded the
> message through filter A and then started encoding another message through
> filter A. Then the exception occurs and same Thread 't' continues encoding
> the other message through filter B. So its like it completely forgot to
> encode the earlier message through filter B. In the good cases, thread 't'
> would after encoding an object through encoder A always encode that through
> encoder B before doing anything else.
>
> Another thing is that when I only have one filter - even under higher
> stress, things work fine.
Do you have an executor filter in your chain ?
Otherwise, it seems you have enqueued a mssage which is not an IoBuffer,
and this can't be sent. Like if encoder A directly pushes the message
instead of going through the second encoder.
It's difficult to tell what's going wrong without more information. Can
you tell us more ?
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com