You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mina.apache.org by peter royal <pe...@pobox.com> on 2008/06/11 17:33:04 UTC
Re: How to properly synchronize external code with MINA
On May 27, 2008, at 3:27 PM, Yigal Rachman wrote:
> I asked about this in February and have managed to move ahead
> without a proper solution. However, I am seeing glitches that point
> to synchronizing problems, so I would greatly appreciate any help
> that you could offer me. To recap: how do I synchronize the
> operation of a thread external to mina so that it plays nicely with
> mina? In particular, I have no idea what to synchronize *with*. My
> first thought was that synchronizing with IoSession would be the
> thing to do, but an inspection of the mina source code does not
> support this notion.
do you need to synchronize with mina?
> The client must periodically poll the instrument for its data. I am
> using a repeating timer to do this.
this looks fine. whenever the timer fires, it will write a message on
the session. no need to do any additional synchronization.
looking back at your original error, it makes me think its an
underlying MINA bug if it still occurs..
-pete
--
(peter.royal|osi)@pobox.com - http://fotap.org/~osi
Re: How to properly synchronize external code with MINA
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 22/05/2018 à 19:31, Jonathan Valliere a écrit :
> Or you could add IoBuffer#getHexDump(start, end)?
Clearly an option.
That would raise the question : do we need an helper class then ?
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: How to properly synchronize external code with MINA
Posted by Jonathan Valliere <jo...@emoten.com>.
Or you could add IoBuffer#getHexDump(start, end)?
On Tue, May 22, 2018 at 12:55 PM Emmanuel Lécharny <el...@gmail.com>
wrote:
>
>
> Le 22/05/2018 à 18:37, Jonathan Valliere a écrit :
> > I don’t think there is any benefit of having it public. It’s only useable
> > with IoBuffer and it’s built in to IoBuffer. Unless someone added more
> > functions to it which aren’t present in IoBuffer such as hex ranges then
> > might as well leave it as is.
>
> Actually, I was thinking about adding debug code in
> NioConnector/NioProcessor in the read/write methods to get a peek of
> what is being read from a socket or written in a socket :
>
> protected int read(NioSession session, IoBuffer buf) throws Exception {
> ByteChannel channel = session.getChannel();
>
> int nbRead = channel.read(buf.buf());
>
> if (nbRead != 0) {
> IoBuffer copy = buf.duplicate();
> copy.flip();
>
> System.out.println("<<< NioSelector read: " +
> IoBufferHexDumper.getHexdump(copy, nbRead));
> }
>
> return nbRead;
> }
>
>
> (this code is just for some debug session I'm conducting atm, and I used
> the old version of the dumper. That explains the sysout that should be
> replaced by proper Log calls).
>
>
> --
> Emmanuel Lecharny
>
> Symas.com
> directory.apache.org
>
>
Re: How to properly synchronize external code with MINA
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 22/05/2018 à 18:37, Jonathan Valliere a écrit :
> I don’t think there is any benefit of having it public. It’s only useable
> with IoBuffer and it’s built in to IoBuffer. Unless someone added more
> functions to it which aren’t present in IoBuffer such as hex ranges then
> might as well leave it as is.
Actually, I was thinking about adding debug code in
NioConnector/NioProcessor in the read/write methods to get a peek of
what is being read from a socket or written in a socket :
protected int read(NioSession session, IoBuffer buf) throws Exception {
ByteChannel channel = session.getChannel();
int nbRead = channel.read(buf.buf());
if (nbRead != 0) {
IoBuffer copy = buf.duplicate();
copy.flip();
System.out.println("<<< NioSelector read: " +
IoBufferHexDumper.getHexdump(copy, nbRead));
}
return nbRead;
}
(this code is just for some debug session I'm conducting atm, and I used
the old version of the dumper. That explains the sysout that should be
replaced by proper Log calls).
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: How to properly synchronize external code with MINA
Posted by Jonathan Valliere <jo...@apache.org>.
I don’t think there is any benefit of having it public. It’s only useable
with IoBuffer and it’s built in to IoBuffer. Unless someone added more
functions to it which aren’t present in IoBuffer such as hex ranges then
might as well leave it as is.
On Tue, May 22, 2018 at 10:53 AM Emmanuel Lécharny <el...@gmail.com>
wrote:
>
>
> Le 22/05/2018 à 16:04, Jonathan Valliere a écrit :
> > Just made the commit so getHexDump() is now perfectly safe without having
> > to create a duplicate.
>
> Thanks for that !
>
> One more thing : the IoBufferHexDumper class is package protected, which
> is probably a bit limiting. It might be worthful to make it public now
> that it is safe.
>
> wdyt ?
>
> --
> Emmanuel Lecharny
>
> Symas.com
> directory.apache.org
>
>
Re: How to properly synchronize external code with MINA
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 22/05/2018 à 16:04, Jonathan Valliere a écrit :
> Just made the commit so getHexDump() is now perfectly safe without having
> to create a duplicate.
Thanks for that !
One more thing : the IoBufferHexDumper class is package protected, which
is probably a bit limiting. It might be worthful to make it public now
that it is safe.
wdyt ?
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: How to properly synchronize external code with MINA
Posted by Jonathan Valliere <jo...@emoten.com>.
Just made the commit so getHexDump() is now perfectly safe without having
to create a duplicate.
On Tue, May 22, 2018 at 9:59 AM, Emmanuel Lécharny <el...@gmail.com>
wrote:
>
>
> Le 22/05/2018 à 15:09, Jonathan Valliere a écrit :
> > Right, but the caveat is that the duplicate buffer shares the same memory
> > space.
>
> Correct. As soon as you don't change the internal byte [], you are fine
> though. Dumping a duplicate should thenalways be fine.
>
>
> Calling duplicate then changing the contents would be a no-no.
>
> Absolutely.
>
>
> --
> Emmanuel Lecharny
>
> Symas.com
> directory.apache.org
>
>
Re: How to properly synchronize external code with MINA
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 22/05/2018 à 15:09, Jonathan Valliere a écrit :
> Right, but the caveat is that the duplicate buffer shares the same memory
> space.
Correct. As soon as you don't change the internal byte [], you are fine
though. Dumping a duplicate should thenalways be fine.
Calling duplicate then changing the contents would be a no-no.
Absolutely.
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: How to properly synchronize external code with MINA
Posted by Jonathan Valliere <jo...@apache.org>.
Done
Re: How to properly synchronize external code with MINA
Posted by Jonathan Valliere <jo...@emoten.com>.
I'll update getHexDump to be non-modifying right now
On Tue, May 22, 2018 at 9:09 AM, Jonathan Valliere <jo...@emoten.com>
wrote:
> Right, but the caveat is that the duplicate buffer shares the same memory
> space. Calling duplicate then changing the contents would be a no-no.
> Without drilling down and paying attention, it would be easy to think that
> duplicate actually duplicates the buffer instead of creating a slice.
>
> On Tue, May 22, 2018 at 8:42 AM, Emmanuel Lécharny <el...@gmail.com>
> wrote:
>
>>
>>
>> Le 22/05/2018 à 14:02, Jonathan Valliere a écrit :
>> > Duplicating the buffer most likely temporarily moves the position
>> also. If
>> > you submit a buffer to be written, don’t do anything with it until
>> after it
>> > it written.
>>
>> ByteBuffer.duplicate() - which is called by IoBuffer.duplicate() -
>> creates a new Direct/HeapBuffer based o the byte[] and all the
>> associated flags (limit, capacity, position makrk). The original buffer
>> isn't altered.
>>
>> In other words : it's safe.
>>
>> --
>> Emmanuel Lecharny
>>
>> Symas.com
>> directory.apache.org
>>
>>
>
Re: How to properly synchronize external code with MINA
Posted by Jonathan Valliere <jo...@emoten.com>.
Right, but the caveat is that the duplicate buffer shares the same memory
space. Calling duplicate then changing the contents would be a no-no.
Without drilling down and paying attention, it would be easy to think that
duplicate actually duplicates the buffer instead of creating a slice.
On Tue, May 22, 2018 at 8:42 AM, Emmanuel Lécharny <el...@gmail.com>
wrote:
>
>
> Le 22/05/2018 à 14:02, Jonathan Valliere a écrit :
> > Duplicating the buffer most likely temporarily moves the position also.
> If
> > you submit a buffer to be written, don’t do anything with it until after
> it
> > it written.
>
> ByteBuffer.duplicate() - which is called by IoBuffer.duplicate() -
> creates a new Direct/HeapBuffer based o the byte[] and all the
> associated flags (limit, capacity, position makrk). The original buffer
> isn't altered.
>
> In other words : it's safe.
>
> --
> Emmanuel Lecharny
>
> Symas.com
> directory.apache.org
>
>
Re: How to properly synchronize external code with MINA
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 22/05/2018 à 14:02, Jonathan Valliere a écrit :
> Duplicating the buffer most likely temporarily moves the position also. If
> you submit a buffer to be written, don’t do anything with it until after it
> it written.
ByteBuffer.duplicate() - which is called by IoBuffer.duplicate() -
creates a new Direct/HeapBuffer based o the byte[] and all the
associated flags (limit, capacity, position makrk). The original buffer
isn't altered.
In other words : it's safe.
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: How to properly synchronize external code with MINA
Posted by Jonathan Valliere <jo...@apache.org>.
Duplicating the buffer most likely temporarily moves the position also. If
you submit a buffer to be written, don’t do anything with it until after it
it written.
On Tue, May 22, 2018 at 5:55 AM Emmanuel Lécharny <el...@gmail.com>
wrote:
>
>
> Le 22/05/2018 à 11:17, rexxar a écrit :
> > Hi,Emmanuel
> >
> > Thank you very much for your Explanation and Suggest。But I still have
> > doubts about this ：
> >
> > the IoBuffer.getHexDump() method do move the position while dumping the
> > content，but it
> >
> > recover it finally。It seems no other thread will operate the IoBuffer
> during
> > this period，right?
>
> I was wrong, the getHexDump() method does not modify the buffer (if it
> does, its a bug and need to be fixed).
>
> I still strongly suggest you duplicate the buffer before doing anything
> with it, including dumping its content.
>
> --
> Emmanuel Lecharny
>
> Symas.com
> directory.apache.org
>
>
Re: How to properly synchronize external code with MINA
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 22/05/2018 à 11:17, rexxar a écrit :
> Hi,Emmanuel
>
> Thank you very much for your Explanation and Suggest。But I still have
> doubts about this ：
>
> the IoBuffer.getHexDump() method do move the position while dumping the
> content，but it
>
> recover it finally。It seems no other thread will operate the IoBuffer during
> this period，right?
I was wrong, the getHexDump() method does not modify the buffer (if it
does, its a bug and need to be fixed).
I still strongly suggest you duplicate the buffer before doing anything
with it, including dumping its content.
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: How to properly synchronize external code with MINA
Posted by rexxar <x2...@qq.com>.
Hi,Emmanuel
Thank you very much for your Explanation and Suggest。But I still have
doubts about this ：
the IoBuffer.getHexDump() method do move the position while dumping the
content，but it
recover it finally。It seems no other thread will operate the IoBuffer during
this period，right?
Can a logical crossing occur during this period ?
--
Sent from: http://apache-mina.10907.n7.nabble.com/Apache-MINA-Developer-Forum-f6809.html
Re: How to properly synchronize external code with MINA
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi,
the IoBuffer.getHexDump() method does not copy the IoBuffer, and will
move the position while dumping the content.
It's clearly not the smartes utility function, as you can imagine...
Anyway, you have to copy the buffer (duplicate() function) you want to
dump before dumping it, to avoid any intercation with the logic of your
application.
Le 22/05/2018 à 03:58, rexxar a écrit :
> Hi，
>
> here is the Implement of encode"
>
> @Override
> public void encode(IoSession session, Object message, ProtocolEncoderOutput
> out) throws Exception {
> IoBuffer buf = IoBuffer.allocate(100);
> buf.setAutoExpand(true);
>
> NetMessage msg = (NetMessage) message;
> byte[] bs = msg.toByteArray();
>
> int sourceLen = bs.length;
> boolean needCompress = isNeedCompress(msg, sourceLen);
> if (needCompress) {
> bs = ZlibUtils.compressNormal(bs);
> }
>
> boolean needEncrypt = isNeedEncrypt(msg);
>
> if (needEncrypt) {
> bs = cryptor.encrypt(bs, session);
> }
>
> int len = bs.length; //
> sourceLen = len; //
> if (needCompress) {
> len = len + 4;
> len = len | 0x20000000;
> }
> if (needEncrypt) {
> len = len | 0x10000000;
> }
> buf.clear();
> buf.putInt(len);
> if (needCompress) {
> buf.putInt(sourceLen);
> }
>
> buf.put(bs);
> buf.flip();
> out.write(buf);
>
> /*the "InvalidMarkException" happend When the following log is added
>
> try{
> JLogger.fixInfo(String.format("encodeMsg session id[%d], Len[%d],
> Compress[%d], msgId[%d], opcode[%d], buf:%s",
> session.getId(), sourceLen, needCompress ? 1 : 0, msg.id,
> msg.opcode,NetMessage.ioBufferToString(buf) ));
> }catch(Exception e){
> JLogger.fixError("print error...",e);
> }*/
>
> }
> "
>
>
> the NetMessage.ioBufferToString fuction is "
>
> public static String ioBufferToString(IoBuffer buffer) {
> StringBuilder sb = new StringBuilder();
> if (buffer.isDirect()) {
> sb.append("DirectBuffer");
> } else {
> sb.append("HeapBuffer");
> }
> sb.append("[pos=");
> sb.append(buffer.position());
> sb.append(" lim=");
> sb.append(buffer.limit());
> sb.append(" cap=");
> sb.append(buffer.capacity());
> sb.append(": ");
> sb.append(buffer.getHexDump());
> sb.append(']');
> return sb.toString();
> }
> "
> here is the log when the problem happend, it seems happend when multithread
> call IoSession.Write()
> <http://apache-mina.10907.n7.nabble.com/file/t2155/%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_20180522094404.png>
>
>
> <http://apache-mina.10907.n7.nabble.com/file/t2155/%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_2.png>
>
>
> <http://apache-mina.10907.n7.nabble.com/file/t2155/%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_3.png>
>
>
>
> --
> Sent from: http://apache-mina.10907.n7.nabble.com/Apache-MINA-Developer-Forum-f6809.html
>
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: How to properly synchronize external code with MINA
Posted by rexxar <x2...@qq.com>.
Hi，
here is the Implement of encode"
@Override
public void encode(IoSession session, Object message, ProtocolEncoderOutput
out) throws Exception {
IoBuffer buf = IoBuffer.allocate(100);
buf.setAutoExpand(true);
NetMessage msg = (NetMessage) message;
byte[] bs = msg.toByteArray();
int sourceLen = bs.length;
boolean needCompress = isNeedCompress(msg, sourceLen);
if (needCompress) {
bs = ZlibUtils.compressNormal(bs);
}
boolean needEncrypt = isNeedEncrypt(msg);
if (needEncrypt) {
bs = cryptor.encrypt(bs, session);
}
int len = bs.length; //
sourceLen = len; //
if (needCompress) {
len = len + 4;
len = len | 0x20000000;
}
if (needEncrypt) {
len = len | 0x10000000;
}
buf.clear();
buf.putInt(len);
if (needCompress) {
buf.putInt(sourceLen);
}
buf.put(bs);
buf.flip();
out.write(buf);
/*the "InvalidMarkException" happend When the following log is added
try{
JLogger.fixInfo(String.format("encodeMsg session id[%d], Len[%d],
Compress[%d], msgId[%d], opcode[%d], buf:%s",
session.getId(), sourceLen, needCompress ? 1 : 0, msg.id,
msg.opcode,NetMessage.ioBufferToString(buf) ));
}catch(Exception e){
JLogger.fixError("print error...",e);
}*/
}
"
the NetMessage.ioBufferToString fuction is "
public static String ioBufferToString(IoBuffer buffer) {
StringBuilder sb = new StringBuilder();
if (buffer.isDirect()) {
sb.append("DirectBuffer");
} else {
sb.append("HeapBuffer");
}
sb.append("[pos=");
sb.append(buffer.position());
sb.append(" lim=");
sb.append(buffer.limit());
sb.append(" cap=");
sb.append(buffer.capacity());
sb.append(": ");
sb.append(buffer.getHexDump());
sb.append(']');
return sb.toString();
}
"
here is the log when the problem happend, it seems happend when multithread
call IoSession.Write()
<http://apache-mina.10907.n7.nabble.com/file/t2155/%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_20180522094404.png>
<http://apache-mina.10907.n7.nabble.com/file/t2155/%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_2.png>
<http://apache-mina.10907.n7.nabble.com/file/t2155/%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_3.png>
--
Sent from: http://apache-mina.10907.n7.nabble.com/Apache-MINA-Developer-Forum-f6809.html
Re: How to properly synchronize external code with MINA
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi,
Le 21/05/2018 à 15:36, rexxar a écrit :
> hi pete:
> I still come across the same IoBuffer "mark" problem with mina2.0.16 。
Do you have a piece of code demonstrating the problem you have ?
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: How to properly synchronize external code with MINA
Posted by rexxar <x2...@qq.com>.
hi pete:
I still come across the same IoBuffer "mark" problem with mina2.0.16 。It
happened occasionally when I add a log to dump（use getHexDump()） the
iobuffer in the encoder fliter 。 Have you solved this problem since this
problem was put forward in 2008? Have you received other cases with this
same problem recently?
--
Sent from: http://apache-mina.10907.n7.nabble.com/Apache-MINA-Developer-Forum-f6809.html
Re: How to properly synchronize external code with MINA
Posted by peter royal <pr...@apache.org>.
On Jun 23, 2008, at 10:39 AM, Yigal Rachman wrote:
> Sorry for my slow response - I have just returned from a (too short)
> vacation.
>
> Thank you for looking into this.
>
> I think you are correct in believing that the original problem is a
> MINA bug. I ran a soak test that does not involve extra threads to
> see whether the IoBuffer "mark" problem could be made to occur. The
> answer is yes. So it appears that there is a synchronizing problem
> in IoBuffer itself. In fact, I was able to get rid of the error by
> avoiding rewinding the buffer (not sure why this works, except the
> rewind affects the mark).
can you provide a failing unit test for this that we can include in
our test suite?
-pete
>
>
> Yigal
>
>
>
> peter royal wrote:
>> On May 27, 2008, at 3:27 PM, Yigal Rachman wrote:
>>> I asked about this in February and have managed to move ahead
>>> without a proper solution. However, I am seeing glitches that
>>> point to synchronizing problems, so I would greatly appreciate any
>>> help that you could offer me. To recap: how do I synchronize the
>>> operation of a thread external to mina so that it plays nicely
>>> with mina? In particular, I have no idea what to synchronize
>>> *with*. My first thought was that synchronizing with IoSession
>>> would be the thing to do, but an inspection of the mina source
>>> code does not support this notion.
>>
>> do you need to synchronize with mina?
>>
>>> The client must periodically poll the instrument for its data. I
>>> am using a repeating timer to do this.
>>
>> this looks fine. whenever the timer fires, it will write a message
>> on the session. no need to do any additional synchronization.
>>
>> looking back at your original error, it makes me think its an
>> underlying MINA bug if it still occurs..
>>
>> -pete
>>
>> --(peter.royal|osi)@pobox.com - http://fotap.org/~osi
>>
>
>
--
proyal@apache.org - http://fotap.org/~osi
Re: How to properly synchronize external code with MINA
Posted by Yigal Rachman <yi...@uvic.ca>.
Hi, Pete:
Sorry for my slow response - I have just returned from a (too short)
vacation.
Thank you for looking into this.
I think you are correct in believing that the original problem is a MINA
bug. I ran a soak test that does not involve extra threads to see
whether the IoBuffer "mark" problem could be made to occur. The answer
is yes. So it appears that there is a synchronizing problem in IoBuffer
itself. In fact, I was able to get rid of the error by avoiding
rewinding the buffer (not sure why this works, except the rewind affects
the mark).
Yigal
peter royal wrote:
> On May 27, 2008, at 3:27 PM, Yigal Rachman wrote:
>> I asked about this in February and have managed to move ahead without
>> a proper solution. However, I am seeing glitches that point to
>> synchronizing problems, so I would greatly appreciate any help that
>> you could offer me. To recap: how do I synchronize the operation of
>> a thread external to mina so that it plays nicely with mina? In
>> particular, I have no idea what to synchronize *with*. My first
>> thought was that synchronizing with IoSession would be the thing to
>> do, but an inspection of the mina source code does not support this
>> notion.
>
> do you need to synchronize with mina?
>
>> The client must periodically poll the instrument for its data. I am
>> using a repeating timer to do this.
>
> this looks fine. whenever the timer fires, it will write a message on
> the session. no need to do any additional synchronization.
>
> looking back at your original error, it makes me think its an
> underlying MINA bug if it still occurs..
>
> -pete
>
> --(peter.royal|osi)@pobox.com - http://fotap.org/~osi
>