You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@directory.apache.org by tr...@apache.org on 2006/03/11 17:00:32 UTC
svn commit: r385098 - in /directory/trunks/mina/core/src:
main/java/org/apache/mina/common/ByteBuffer.java
test/java/org/apache/mina/common/ByteBufferTest.java
Author: trustin
Date: Sat Mar 11 08:00:31 2006
New Revision: 385098
URL: http://svn.apache.org/viewcvs?rev=385098&view=rev
Log:
Fixed: DIRMINA-186 (ByteBuffer.putString() can loop endlessly with multi-byte UTF-8 characters)
* Fixed an infinite loop when the buffer is sized just-right.
** Applied Peter's patch to ByteBufferTest
** Applied Peter's patch to ByteBuffer after some optimization.
Modified:
directory/trunks/mina/core/src/main/java/org/apache/mina/common/ByteBuffer.java
directory/trunks/mina/core/src/test/java/org/apache/mina/common/ByteBufferTest.java
Modified: directory/trunks/mina/core/src/main/java/org/apache/mina/common/ByteBuffer.java
URL: http://svn.apache.org/viewcvs/directory/trunks/mina/core/src/main/java/org/apache/mina/common/ByteBuffer.java?rev=385098&r1=385097&r2=385098&view=diff
==============================================================================
--- directory/trunks/mina/core/src/main/java/org/apache/mina/common/ByteBuffer.java (original)
+++ directory/trunks/mina/core/src/main/java/org/apache/mina/common/ByteBuffer.java Sat Mar 11 08:00:31 2006
@@ -1087,9 +1087,9 @@
CharBuffer in = CharBuffer.wrap( val );
int expectedLength = (int) (in.remaining() * encoder.averageBytesPerChar()) + 1;
-
+
encoder.reset();
-
+
for (;;) {
CoderResult cr;
if( in.hasRemaining() )
@@ -1107,7 +1107,7 @@
}
if( cr.isOverflow() && isAutoExpand() )
{
- autoExpand( expectedLength );
+ autoExpand( limit() - position() + expectedLength );
continue;
}
cr.throwException();
Modified: directory/trunks/mina/core/src/test/java/org/apache/mina/common/ByteBufferTest.java
URL: http://svn.apache.org/viewcvs/directory/trunks/mina/core/src/test/java/org/apache/mina/common/ByteBufferTest.java?rev=385098&r1=385097&r2=385098&view=diff
==============================================================================
--- directory/trunks/mina/core/src/test/java/org/apache/mina/common/ByteBufferTest.java (original)
+++ directory/trunks/mina/core/src/test/java/org/apache/mina/common/ByteBufferTest.java Sat Mar 11 08:00:31 2006
@@ -19,6 +19,7 @@
package org.apache.mina.common;
import java.nio.BufferOverflowException;
+import java.nio.charset.CharacterCodingException;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CharsetEncoder;
@@ -449,6 +450,48 @@
Assert.assertEquals( ' ', buf.get( 4 ) );
}
+ public void testWideUtf8Characters() throws Exception
+ {
+ Runnable r = new Runnable()
+ {
+ public void run()
+ {
+ ByteBuffer buffer = ByteBuffer.allocate( 1 );
+ buffer.setAutoExpand( true );
+
+ Charset charset = Charset.forName( "UTF-8" );
+
+ CharsetEncoder encoder = charset.newEncoder();
+
+ for( int i = 0; i < 5; i++ )
+ {
+ System.out.println( i );
+ try
+ {
+ buffer.putString( "\u89d2", encoder );
+ }
+ catch( CharacterCodingException e )
+ {
+ fail( e.getMessage() );
+ }
+ }
+ }
+ };
+
+ Thread t = new Thread( r );
+ t.setDaemon( true );
+ t.start();
+
+ Thread.sleep( 5 * 1000 );
+
+ if( t.isAlive() )
+ {
+ t.interrupt();
+
+ fail( "Went into endless loop trying to encode character");
+ }
+ }
+
public void testObjectSerialization() throws Exception
{
ByteBuffer buf = ByteBuffer.allocate( 16 );
Re: svn commit: r385098 - in /directory/trunks/mina/core/src: main/java/org/apache/mina/common/ByteBuffer.java test/java/org/apache/mina/common/ByteBufferTest.java
Posted by peter royal <pr...@apache.org>.
On Mar 11, 2006, at 11:23 AM, Trustin Lee wrote:
> Yep I noticed it by myself and fixed it again:
>
> http://svn.apache.org/viewcvs?rev=385102&view=rev
excellent, ty :)
> The core of my concern is that the expectedLength is only based on
>> the average bytes per character, which for UTF-8 is only 1.1. The max
>> is 4, so its not that hard for the expected length to be way off if
>> you have many wide characters.
>
>
> I thought the max is 6. It seems like there's a wrong documentation:
>
> http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
>
> The official site claims that it takes up to four octets:
>
> http://www.utf-8.com/
Odd, I was just going by what the UTF-8 encoder said was its maximum,
figuring that it wouldn't lie about what it was going to do :)
-pete
--
proyal@apache.org - http://fotap.org/~osi
Re: svn commit: r385098 - in /directory/trunks/mina/core/src: main/java/org/apache/mina/common/ByteBuffer.java test/java/org/apache/mina/common/ByteBufferTest.java
Posted by Trustin Lee <tr...@gmail.com>.
On 3/12/06, peter royal <pe...@pobox.com> wrote:
>
> On Mar 11, 2006, at 11:00 AM, trustin@apache.org wrote:
> > ** Applied Peter's patch to ByteBuffer after some optimization.
>
> I don't like this optimization.. seems premature.
Yep I noticed it by myself and fixed it again:
http://svn.apache.org/viewcvs?rev=385102&view=rev
Trustin
The core of my concern is that the expectedLength is only based on
> the average bytes per character, which for UTF-8 is only 1.1. The max
> is 4, so its not that hard for the expected length to be way off if
> you have many wide characters.
I thought the max is 6. It seems like there's a wrong documentation:
http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
The official site claims that it takes up to four octets:
http://www.utf-8.com/
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
PGP key fingerprints:
* E167 E6AF E73A CBCE EE41 4A29 544D DE48 FE95 4E7E
* B693 628E 6047 4F8F CFA4 455E 1C62 A7DC 0255 ECA6
Re: svn commit: r385098 - in /directory/trunks/mina/core/src: main/java/org/apache/mina/common/ByteBuffer.java test/java/org/apache/mina/common/ByteBufferTest.java
Posted by peter royal <pe...@pobox.com>.
On Mar 11, 2006, at 11:00 AM, trustin@apache.org wrote:
> ** Applied Peter's patch to ByteBuffer after some optimization.
I don't like this optimization.. seems premature.
In my original patch, I first had it trying to auto-expand based on
average bytes per character, then max bytes per character, and then
giving up.. (and then, only calculating expected length if it was
really needed, which could be argued is an optimization..)
In the tweak you committed, it is still possible for it to go into an
infinite loop if we missed a case, and I feel far more comfortable
knowing that the algorithm might be a tad slower and never have the
possibility of going into an infinite loop.
The core of my concern is that the expectedLength is only based on
the average bytes per character, which for UTF-8 is only 1.1. The max
is 4, so its not that hard for the expected length to be way off if
you have many wide characters.
-pete
--
(peter.royal|osi)@pobox.com - http://fotap.org/~osi