You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@mina.apache.org by Erinc Arikan <er...@gmail.com> on 2009/05/15 06:48:04 UTC
Re: MINA newbie question(message gets chipped 5 or 6 bytes every
time)
Thanks for the response.
I actually figured out my problem I was stuffing and unstuffing data in my
pojo, It looks like that's the problem I can't believe how I totally forgot
about that.
As you have mentioned the following code for my decoder will work ok for LAN
scenarios. It will be problematic if I receive 78 bytes one by one. How can
I handle that scenario if message reaches server in parts? There can be many
fragmented messages that might be coming from different clients. How can my
server figure out which message belongs to which historical message and
merge them correctly to make a complete message? How can the following code
be improved? Is there a working example that I can take a look at?
public class MessageDecoder extends CumulativeProtocolDecoder {
protected boolean doDecode(IoSession session, IoBuffer in,
ProtocolDecoderOutput out) throws Exception {
if (in.remaining() >= 31) {
//Has value of 78
int len = in.remaining();
byte[] stream = new byte[len];
in.get(stream);
System.out.println(len);
Message message = new Message(stream);
out.write(message);
return true;
} else {
return false;
}
}
}
On Thu, May 14, 2009 at 6:15 PM, Emmanuel Lecharny <el...@apache.org>wrote:
> Erinc Arikan wrote:
>
>> Hi,
>>
>> I just started working on MINA last week, I am trying to convert a gateway
>> application to MINA - ACTIVEMQ project. I am really impressed the
>> simplicity
>> MINA brings.
>>
>> After this irrelevant introductory part, Here's my problem:
>>
>> My clients connect to the server using a tcp connection.
>>
>> I created my own protocolfilter where I specify my encoder and decoder.
>> Decoder on the server side what I care about most, because that's what I
>> will pass into activemq and process.
>>
>> Only filters that I use are my custom protocolfilter and loggingfilter,
>> the
>> problem lies here: if my client sends a packet of length 78 bytes, decoder
>> in my protocolfilter receives a 0 padded 72 bytes of data. When I strip
>> 0's
>> and and reach end of the message delimiter byte, I realize that what I
>> receive is 72 bytes long. And it somehow matches crc16 calculation from
>> the
>> message but still some bytes are missing. Same thing happened when my
>> message has size 40 and decoder received 35 bytes. I don't know what I am
>> doing wrong but since CRC values match I feel like there is something
>> wrong
>> on my side. Unfortunately I couldn't find an answer to this question on
>> documentation.
>>
>> Any help will be greatly appreciated.
>>
>>
>
> Remember that there is no guarantee that your message will arrive in one
> chunk of exactly 78 bytes. You may as well receive 78 messages containing
> one single byte. You have to accumulate them until you get what you need.
>
>> Erinc
>>
>>
>>
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
Re: MINA newbie question(message gets chipped 5 or 6 bytes every
time)
Posted by Erinc Arikan <er...@gmail.com>.
Thanks Jim;
I figured it out after spending some time on it, You need to use
setAttribute field of the session.
On Fri, May 22, 2009 at 4:19 PM, Jim Newsham <jn...@referentia.com>wrote:
>
> I'm not a user of mina (yet, at least), but my understanding is that that's
> exactly what the session attributes are for -- storage of arbitrary data by
> the protocol.
>
> The protocol codec filter tutorial demonstrates exactly this:
>
> http://mina.apache.org/tutorial-on-protocolcodecfilter-for-mina-2x.html
>
> There are two different cases described: (1) when you know in advance how
> many bytes to wait for; and (2) when you don't know in advance. In the
> latter case, they store the partial data into the session as an attribute.
>
> Jim
>
> > -----Original Message-----
> > From: Erinc Arikan [mailto:erincarikan@gmail.com]
> > Sent: Friday, May 22, 2009 4:48 AM
> > To: users@mina.apache.org
> > Subject: Re: MINA newbie question(message gets chipped 5 or 6 bytes every
> > time)
> >
> > Hi Emmanuel;
> >
> > Thanks for the help, I think that I solved my problem about handling
> > fragmented messages. The only remaining problem for me is to store
> > incomplete messages(stream of bytes) to the session, so that next time
> > rest
> > of the message comes I can validate message integrity, and if it's
> > complete, I'll be able to process the message. I know you said simply do
> > nothing, don't read it from the buffer, but I have delimited messages
> with
> > variable message sizes and with no info about message length beforehand.
> > So
> > I need to read complete buffer every time to see my last byte is end of
> > the
> > message byte. Therefore I will need to store some bytes back into the
> > session when the last byte stream I receive includes an incomplete
> > message.
> >
> > My question is how can I accomplish storing bytes back to the session? I
> > checked session's methods there's a write method but I think that it
> > writes
> > data back to client and that's not what I need. There are some
> > setAttribute
> > methods but they don't look relevant to me too.
> >
> > Can I use IoBuffer's put method to put back what I already read from the
> > buffer? If so what's going to happen when next fragment arrives. Will it
> > simply overwrite current buffer or will it add the next fragment to what
> > Buffer contains? My intuition says that it will be added but if so
> there's
> > another thing that I wonder, do I have to reverse the order of bytes
> while
> > putting them back to IoBuffer.
> >
> > I wish that I can test and learn the answers by trial and error, but
> > regenarating this special case is little bit hard.
> >
> > Thanks so much in advance.
> >
> > Erinc
> >
> > On Fri, May 15, 2009 at 5:26 PM, Emmanuel Lecharny
> > <el...@apache.org>wrote:
> >
> > > David Rosenstrauch wrote:
> > >
> > >> Emmanuel Lecharny wrote:
> > >>
> > >>> Be aware that you not only can receive a fragment, but may be -
> > depending
> > >>> on your protocol - more than one message in the same PDU (ie you may
> > receive
> > >>> 200 bytes, 2 times 78 bytes plus some more bytes associated with
> > another
> > >>> message). You have to deal with that.
> > >>>
> > >>> If the protocol is a text based one, you may have a look to the
> > >>> documentation :
> > >>>
> > >>> http://mina.apache.org/handling-packet-fragementation.html
> > >>>
> > >>> The tutorial explain how to handle fragmented messages, using a class
> > >>> extending the CumulativeProtocolDecoder class.
> > >>>
> > >>> Hope it helps...
> > >>>
> > >>>
> > >> Sorry, getting a little confused about one aspect of this though:
> > >>
> > >> I was under the impression that if I'm using TextLineCodecFactory that
> > it
> > >> takes care of all the message fragmentation work for me, and that my
> > >> handler's messageReceived method is guaranteed to be passed one and
> > only one
> > >> text line per method call. And so consequently, you really only need
> > to
> > >> deal with fragmentation if you're not using TextLineCodecFactory.
> Have
> > I
> > >> got that correct?
> > >>
> > > Yeah, you're right, if the message is a text based one. But if it's a
> > > binary protocol, then you can't use the TextLineCodecFactory,
> obviously.
> > >
> > >
> > >
> > > --
> > > --
> > > cordialement, regards,
> > > Emmanuel Lécharny
> > > www.iktek.com
> > > directory.apache.org
> > >
> > >
> > >
>
>
>
RE: MINA newbie question(message gets chipped 5 or 6 bytes every time)
Posted by Jim Newsham <jn...@referentia.com>.
I'm not a user of mina (yet, at least), but my understanding is that that's
exactly what the session attributes are for -- storage of arbitrary data by
the protocol.
The protocol codec filter tutorial demonstrates exactly this:
http://mina.apache.org/tutorial-on-protocolcodecfilter-for-mina-2x.html
There are two different cases described: (1) when you know in advance how
many bytes to wait for; and (2) when you don't know in advance. In the
latter case, they store the partial data into the session as an attribute.
Jim
> -----Original Message-----
> From: Erinc Arikan [mailto:erincarikan@gmail.com]
> Sent: Friday, May 22, 2009 4:48 AM
> To: users@mina.apache.org
> Subject: Re: MINA newbie question(message gets chipped 5 or 6 bytes every
> time)
>
> Hi Emmanuel;
>
> Thanks for the help, I think that I solved my problem about handling
> fragmented messages. The only remaining problem for me is to store
> incomplete messages(stream of bytes) to the session, so that next time
> rest
> of the message comes I can validate message integrity, and if it's
> complete, I'll be able to process the message. I know you said simply do
> nothing, don't read it from the buffer, but I have delimited messages with
> variable message sizes and with no info about message length beforehand.
> So
> I need to read complete buffer every time to see my last byte is end of
> the
> message byte. Therefore I will need to store some bytes back into the
> session when the last byte stream I receive includes an incomplete
> message.
>
> My question is how can I accomplish storing bytes back to the session? I
> checked session's methods there's a write method but I think that it
> writes
> data back to client and that's not what I need. There are some
> setAttribute
> methods but they don't look relevant to me too.
>
> Can I use IoBuffer's put method to put back what I already read from the
> buffer? If so what's going to happen when next fragment arrives. Will it
> simply overwrite current buffer or will it add the next fragment to what
> Buffer contains? My intuition says that it will be added but if so there's
> another thing that I wonder, do I have to reverse the order of bytes while
> putting them back to IoBuffer.
>
> I wish that I can test and learn the answers by trial and error, but
> regenarating this special case is little bit hard.
>
> Thanks so much in advance.
>
> Erinc
>
> On Fri, May 15, 2009 at 5:26 PM, Emmanuel Lecharny
> <el...@apache.org>wrote:
>
> > David Rosenstrauch wrote:
> >
> >> Emmanuel Lecharny wrote:
> >>
> >>> Be aware that you not only can receive a fragment, but may be -
> depending
> >>> on your protocol - more than one message in the same PDU (ie you may
> receive
> >>> 200 bytes, 2 times 78 bytes plus some more bytes associated with
> another
> >>> message). You have to deal with that.
> >>>
> >>> If the protocol is a text based one, you may have a look to the
> >>> documentation :
> >>>
> >>> http://mina.apache.org/handling-packet-fragementation.html
> >>>
> >>> The tutorial explain how to handle fragmented messages, using a class
> >>> extending the CumulativeProtocolDecoder class.
> >>>
> >>> Hope it helps...
> >>>
> >>>
> >> Sorry, getting a little confused about one aspect of this though:
> >>
> >> I was under the impression that if I'm using TextLineCodecFactory that
> it
> >> takes care of all the message fragmentation work for me, and that my
> >> handler's messageReceived method is guaranteed to be passed one and
> only one
> >> text line per method call. And so consequently, you really only need
> to
> >> deal with fragmentation if you're not using TextLineCodecFactory. Have
> I
> >> got that correct?
> >>
> > Yeah, you're right, if the message is a text based one. But if it's a
> > binary protocol, then you can't use the TextLineCodecFactory, obviously.
> >
> >
> >
> > --
> > --
> > cordialement, regards,
> > Emmanuel Lécharny
> > www.iktek.com
> > directory.apache.org
> >
> >
> >
Re: MINA newbie question(message gets chipped 5 or 6 bytes every
time)
Posted by Erinc Arikan <er...@gmail.com>.
Hi Emmanuel;
Thanks for the help, I think that I solved my problem about handling
fragmented messages. The only remaining problem for me is to store
incomplete messages(stream of bytes) to the session, so that next time rest
of the message comes I can validate message integrity, and if it's
complete, I'll be able to process the message. I know you said simply do
nothing, don't read it from the buffer, but I have delimited messages with
variable message sizes and with no info about message length beforehand. So
I need to read complete buffer every time to see my last byte is end of the
message byte. Therefore I will need to store some bytes back into the
session when the last byte stream I receive includes an incomplete message.
My question is how can I accomplish storing bytes back to the session? I
checked session's methods there's a write method but I think that it writes
data back to client and that's not what I need. There are some setAttribute
methods but they don't look relevant to me too.
Can I use IoBuffer's put method to put back what I already read from the
buffer? If so what's going to happen when next fragment arrives. Will it
simply overwrite current buffer or will it add the next fragment to what
Buffer contains? My intuition says that it will be added but if so there's
another thing that I wonder, do I have to reverse the order of bytes while
putting them back to IoBuffer.
I wish that I can test and learn the answers by trial and error, but
regenarating this special case is little bit hard.
Thanks so much in advance.
Erinc
On Fri, May 15, 2009 at 5:26 PM, Emmanuel Lecharny <el...@apache.org>wrote:
> David Rosenstrauch wrote:
>
>> Emmanuel Lecharny wrote:
>>
>>> Be aware that you not only can receive a fragment, but may be - depending
>>> on your protocol - more than one message in the same PDU (ie you may receive
>>> 200 bytes, 2 times 78 bytes plus some more bytes associated with another
>>> message). You have to deal with that.
>>>
>>> If the protocol is a text based one, you may have a look to the
>>> documentation :
>>>
>>> http://mina.apache.org/handling-packet-fragementation.html
>>>
>>> The tutorial explain how to handle fragmented messages, using a class
>>> extending the CumulativeProtocolDecoder class.
>>>
>>> Hope it helps...
>>>
>>>
>> Sorry, getting a little confused about one aspect of this though:
>>
>> I was under the impression that if I'm using TextLineCodecFactory that it
>> takes care of all the message fragmentation work for me, and that my
>> handler's messageReceived method is guaranteed to be passed one and only one
>> text line per method call. And so consequently, you really only need to
>> deal with fragmentation if you're not using TextLineCodecFactory. Have I
>> got that correct?
>>
> Yeah, you're right, if the message is a text based one. But if it's a
> binary protocol, then you can't use the TextLineCodecFactory, obviously.
>
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
Re: MINA newbie question(message gets chipped 5 or 6 bytes every
time)
Posted by Emmanuel Lecharny <el...@apache.org>.
David Rosenstrauch wrote:
> Emmanuel Lecharny wrote:
>> Be aware that you not only can receive a fragment, but may be -
>> depending on your protocol - more than one message in the same PDU
>> (ie you may receive 200 bytes, 2 times 78 bytes plus some more bytes
>> associated with another message). You have to deal with that.
>>
>> If the protocol is a text based one, you may have a look to the
>> documentation :
>>
>> http://mina.apache.org/handling-packet-fragementation.html
>>
>> The tutorial explain how to handle fragmented messages, using a class
>> extending the CumulativeProtocolDecoder class.
>>
>> Hope it helps...
>>
>
> Sorry, getting a little confused about one aspect of this though:
>
> I was under the impression that if I'm using TextLineCodecFactory that
> it takes care of all the message fragmentation work for me, and that
> my handler's messageReceived method is guaranteed to be passed one and
> only one text line per method call. And so consequently, you really
> only need to deal with fragmentation if you're not using
> TextLineCodecFactory. Have I got that correct?
Yeah, you're right, if the message is a text based one. But if it's a
binary protocol, then you can't use the TextLineCodecFactory, obviously.
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: MINA newbie question(message gets chipped 5 or 6 bytes every
time)
Posted by David Rosenstrauch <da...@darose.net>.
Emmanuel Lecharny wrote:
> Be aware that you not only can receive a fragment, but may be -
> depending on your protocol - more than one message in the same PDU (ie
> you may receive 200 bytes, 2 times 78 bytes plus some more bytes
> associated with another message). You have to deal with that.
>
> If the protocol is a text based one, you may have a look to the
> documentation :
>
> http://mina.apache.org/handling-packet-fragementation.html
>
> The tutorial explain how to handle fragmented messages, using a class
> extending the CumulativeProtocolDecoder class.
>
> Hope it helps...
>
Sorry, getting a little confused about one aspect of this though:
I was under the impression that if I'm using TextLineCodecFactory that
it takes care of all the message fragmentation work for me, and that my
handler's messageReceived method is guaranteed to be passed one and only
one text line per method call. And so consequently, you really only
need to deal with fragmentation if you're not using
TextLineCodecFactory. Have I got that correct?
Thanks,
DR
Re: the CPU rising to 100%
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 5/22/10 7:36 AM, Allen Jiang wrote:
> Hi,all,
> When I use mina to run some case,the CPU using usually rise to 100% ,I don't know why,who could give me a answer? thank you!
Version ?
--
Regards,
Cordialement,
Emmanuel Lécharny
www.nextury.com
the CPU rising to 100%
Posted by Allen Jiang <ji...@gmail.com>.
Hi,all,
When I use mina to run some case,the CPU using usually rise to 100% ,I don't know why,who could give me a answer? thank you!
Re: MINA newbie question(message gets chipped 5 or 6 bytes every
time)
Posted by Emmanuel Lecharny <el...@apache.org>.
On Fri, May 15, 2009 at 4:38 PM, Erinc Arikan <er...@gmail.com> wrote:
> Thanks Emmanuel;
>
> Is this issue related to Nagle's Algorithm?
No.
>
> Do I have to use
>
> ((SocketSessionConfig) connector.getSessionConfig()).setTcpNoDelay(false)
No, certainly not !
>
>
> to ensure that I will receive only one complete message
You can't have such a guarantee. There is no way you can get it.
> given that I
> implement MessageDecoder correctly, but I feel like setting tcpnodelay
> false
> will negatively affect the performance. Is there any way that I can just
> prevent partial messages in the PDU?
No.
> I am happy receiving multiple messages
> but I want to be able to deal with partial messages somehow instead of
> discarding them.
Just use the cumulative decoder, but you will have to deal with message's
size. This is the frontier between MINA and your implementation on top of
it. MINA knows nothing about your messages, and your protocol. So you have
to code your codec in a way it handles all the possible messages, even if
they are fragmented.
The idea is to check if the buffer contains enough bytes to decode a
complete message, assuming you can determinate the message size, of course,
and if so, call the decoder, otherwise do nothing (the buffer is stored in
the session by the extended class).If you have more than one message in the
buffer, you will have to call the decoder until you don't have enough bytes
remaining in the buffer. Last, not least, if you can deal with messages of
different sizes, you have to determinate which kind of message you are
dealing with before calling the decoder.
In any case, MINA helps you to focus on this part of the code, alleviating
the pain about the management of sockets, connection, etc.But it's your task
to handle the protocol !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
RE: MINA newbie question(message gets chipped 5 or 6 bytes every time)
Posted by Jim Newsham <jn...@referentia.com>.
> Do I have to use
>
> ((SocketSessionConfig) connector.getSessionConfig()).setTcpNoDelay(false)
>
> to ensure that I will receive only one complete message given that I
> implement MessageDecoder correctly, but I feel like setting tcpnodelay
> false
> will negatively affect the performance.
You need to treat the incoming data as a stream of bytes, not discrete
messages, because even if you send out discrete messages from one side, the
network infrastructure (routers) may fragment your messages along the way.
Re: MINA newbie question(message gets chipped 5 or 6 bytes every
time)
Posted by Erinc Arikan <er...@gmail.com>.
Thanks Emmanuel;
Yes I believe, in your decoder if you extend CumulativeProtocolDecoder, it
takes care of fragmented messages in different PDU's (It waits till they are
complete). But you are right if there's more than one message in the
same PDU, it's little bit harder to handle given that message sizes vary for
each message type. If we talk about your scenario where server receives 1 78
byte powerup message 1 85 byte healthcheck message and some other bytes for
an incomplete message, the first two message will be sent to message handler
and will be processed. But how can I take care of the other partial package,
does cumulativeprotocoldecoder handle this incomplete message(save it
somewhere wait until we receive rest of it) or is it simply discarded? Is
this issue related to Nagle's Algorithm?
Do I have to use
((SocketSessionConfig) connector.getSessionConfig()).setTcpNoDelay(false)
to ensure that I will receive only one complete message given that I
implement MessageDecoder correctly, but I feel like setting tcpnodelay false
will negatively affect the performance. Is there any way that I can just
prevent partial messages in the PDU? I am happy receiving multiple messages
but I want to be able to deal with partial messages somehow instead of
discarding them.
Following is code of my current decoder. To process multiple messages I need
to look for message start and message end in the byte array. And each
message will be written to ProtocolDecoderOutput. But I don't know what to
do with the incomplete message.
http://mina.apache.org/handling-packet-fragementation.html
doesn't have an example for this scenario.
*
public* *class* MessageDecoder *extends* CumulativeProtocolDecoder {
*protected* *boolean* doDecode(IoSession session, IoBuffer in,
ProtocolDecoderOutput out) *throws* Exception {
*if* (in.remaining() >= 31) {
*int* len = in.remaining();
*byte*[] stream = *new* *byte*[len];
in.get(stream);
Message message = *new* Message(stream);
out.write(message);
*return* *true*;
} *else* {
*return* *false*;
}
}
}
On Fri, May 15, 2009 at 3:40 AM, Emmanuel Lecharny <el...@apache.org>wrote:
> Erinc Arikan wrote:
>
>> Thanks for the response.
>> I actually figured out my problem I was stuffing and unstuffing data in my
>> pojo, It looks like that's the problem I can't believe how I totally
>> forgot
>> about that.
>>
>>
> Cool !
>
>> As you have mentioned the following code for my decoder will work ok for
>> LAN
>> scenarios. It will be problematic if I receive 78 bytes one by one. How
>> can
>> I handle that scenario if message reaches server in parts? There can be
>> many
>> fragmented messages that might be coming from different clients. How can
>> my
>> server figure out which message belongs to which historical message and
>> merge them correctly to make a complete message? How can the following
>> code
>> be improved? Is there a working example that I can take a look at?
>>
>>
> First, every connection is associated with a session in MINA, so if a
> fragment is received, it belongs to a session. Now, the only thing you have
> to do is to accumulate the message in a temporary buffer, and store this
> buffer within this session until the message is completed.
>
> Be aware that you not only can receive a fragment, but may be - depending
> on your protocol - more than one message in the same PDU (ie you may receive
> 200 bytes, 2 times 78 bytes plus some more bytes associated with another
> message). You have to deal with that.
>
> If the protocol is a text based one, you may have a look to the
> documentation :
>
> http://mina.apache.org/handling-packet-fragementation.html
>
> The tutorial explain how to handle fragmented messages, using a class
> extending the CumulativeProtocolDecoder class.
>
> Hope it helps...
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
Re: MINA newbie question(message gets chipped 5 or 6 bytes every
time)
Posted by Emmanuel Lecharny <el...@apache.org>.
Erinc Arikan wrote:
> Thanks for the response.
> I actually figured out my problem I was stuffing and unstuffing data in my
> pojo, It looks like that's the problem I can't believe how I totally forgot
> about that.
>
Cool !
> As you have mentioned the following code for my decoder will work ok for LAN
> scenarios. It will be problematic if I receive 78 bytes one by one. How can
> I handle that scenario if message reaches server in parts? There can be many
> fragmented messages that might be coming from different clients. How can my
> server figure out which message belongs to which historical message and
> merge them correctly to make a complete message? How can the following code
> be improved? Is there a working example that I can take a look at?
>
First, every connection is associated with a session in MINA, so if a
fragment is received, it belongs to a session. Now, the only thing you
have to do is to accumulate the message in a temporary buffer, and store
this buffer within this session until the message is completed.
Be aware that you not only can receive a fragment, but may be -
depending on your protocol - more than one message in the same PDU (ie
you may receive 200 bytes, 2 times 78 bytes plus some more bytes
associated with another message). You have to deal with that.
If the protocol is a text based one, you may have a look to the
documentation :
http://mina.apache.org/handling-packet-fragementation.html
The tutorial explain how to handle fragmented messages, using a class
extending the CumulativeProtocolDecoder class.
Hope it helps...
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org