You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by "Richard Liang (JIRA)" <ji...@apache.org> on 2006/03/02 02:47:42 UTC

[jira] Created: (HARMONY-148) java.nio.charset.CharsetDecoder: decode(in,out,endOfInput) method doesn't preserve replace string for successive decode invocation while "out" doesn't have engouh space for replace string.

java.nio.charset.CharsetDecoder: decode(in,out,endOfInput) method doesn't preserve replace string for successive decode invocation while "out" doesn't have engouh space for replace string.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

         Key: HARMONY-148
         URL: http://issues.apache.org/jira/browse/HARMONY-148
     Project: Harmony
        Type: Bug
  Components: Classlib  
    Reporter: Richard Liang
    Priority: Minor


Consider following senario:
During the invocation of method java.nio.charset.CharsetDecoder.decode(in,out,false), method "decodeLoop" returns malformed CoderResult, and the action is set as "CodingErrorAction.REPLACE". So replace action should be taken, but "out" doesn't hold enough space for the replace string. Therefore, CoderResult.OVERFLOW should be returned. Harmony does return OVERFLOW currently, but the replace string won't put into new allocated output any more. In other words, the replace string is missing from the view of "stream decoding". Therefore, replace string should be preserved for successive decode(in,out,endOfInput) invacation.
Both RI5.0 and Harmony fails on the following test case. However, according to the spec, we think it is a defect of RI5.0.

================================== Test Case======================
/*
	 * Test the method decode(ByteBuffer) .
	 */
	public void testDecode_ReplaceOverflow() throws Exception {
		String replaceString = "a";
		Charset cs = Charset.forName("UTF-8");
		MockMalformedDecoder decoder = new MockMalformedDecoder(cs);
		decoder.onMalformedInput(CodingErrorAction.REPLACE);
		decoder.replaceWith(replaceString);
		CharBuffer out = CharBuffer.allocate(1);
		// MockMalformedDecoder treats the second byte '0x38' as malformed, 
		// but "out" doesn't have enough space for replace string.
		ByteBuffer in = ByteBuffer.wrap(new byte[]{0x45,0x38,0x45,0x45});
		CoderResult result = decoder.decode(in,out,false);
		assertTrue(result.isOverflow());
		
		// allocate enough space for "out"
		out = CharBuffer.allocate(10);
		// replace string should be put into "out" firstly,
		// and then decode "in". 
		result =decoder.decode(in,out,true);
		out.flip();
		assertTrue(result.isUnderflow());
		assertEquals("abb",out.toString());
	}
	/*
	 * Mock decoder. It treats byte whose value is less than "0x40" as malformed.
	 */
	static class MockMalformedDecoder extends java.nio.charset.CharsetDecoder {

		public MockMalformedDecoder(Charset cs) {
			super(cs, 1, 10);
		}

		/*
		 * It treats byte whose value is less than "0x40" as malformed.
		 * Otherwise, it's decoded as 'b'.
		 */
		protected CoderResult decodeLoop(ByteBuffer in, CharBuffer out) {
			System.out.println("Using MockMalformedDecoder");
			while (in.hasRemaining()) {
				byte b = in.get();
				if(b < 0x40){
					return CoderResult.malformedForLength(1);
				}
				if(!out.hasRemaining()){
					return CoderResult.OVERFLOW;
				}
				out.put((char)'b');
			}
			return CoderResult.UNDERFLOW;
		}
	}


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

[jira] Updated: (HARMONY-148) java.nio.charset.CharsetDecoder: decode(in,out,endOfInput) method doesn't preserve replace string for successive decode invocation while "out" doesn't have engouh space for replace string.

Posted by "Richard Liang (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HARMONY-148?page=all ]

Richard Liang updated HARMONY-148:
----------------------------------

    Attachment: CharsetDecoder_patch_148.txt

This patch fixes bug Harmony 148. Would you please have a try ? 
The patch could be simply applied by clicking "team"->"apply patch...".
Thank you very much!

> java.nio.charset.CharsetDecoder: decode(in,out,endOfInput) method doesn't preserve replace string for successive decode invocation while "out" doesn't have engouh space for replace string.
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: HARMONY-148
>          URL: http://issues.apache.org/jira/browse/HARMONY-148
>      Project: Harmony
>         Type: Bug
>   Components: Classlib
>     Reporter: Richard Liang
>     Priority: Minor
>  Attachments: CharsetDecoder_patch_148.txt
>
> Consider following senario:
> During the invocation of method java.nio.charset.CharsetDecoder.decode(in,out,false), method "decodeLoop" returns malformed CoderResult, and the action is set as "CodingErrorAction.REPLACE". So replace action should be taken, but "out" doesn't hold enough space for the replace string. Therefore, CoderResult.OVERFLOW should be returned. Harmony does return OVERFLOW currently, but the replace string won't put into new allocated output any more. In other words, the replace string is missing from the view of "stream decoding". Therefore, replace string should be preserved for successive decode(in,out,endOfInput) invacation.
> Both RI5.0 and Harmony fails on the following test case. However, according to the spec, we think it is a defect of RI5.0.
> ================================== Test Case======================
> /*
> 	 * Test the method decode(ByteBuffer) .
> 	 */
> 	public void testDecode_ReplaceOverflow() throws Exception {
> 		String replaceString = "a";
> 		Charset cs = Charset.forName("UTF-8");
> 		MockMalformedDecoder decoder = new MockMalformedDecoder(cs);
> 		decoder.onMalformedInput(CodingErrorAction.REPLACE);
> 		decoder.replaceWith(replaceString);
> 		CharBuffer out = CharBuffer.allocate(1);
> 		// MockMalformedDecoder treats the second byte '0x38' as malformed, 
> 		// but "out" doesn't have enough space for replace string.
> 		ByteBuffer in = ByteBuffer.wrap(new byte[]{0x45,0x38,0x45,0x45});
> 		CoderResult result = decoder.decode(in,out,false);
> 		assertTrue(result.isOverflow());
> 		
> 		// allocate enough space for "out"
> 		out = CharBuffer.allocate(10);
> 		// replace string should be put into "out" firstly,
> 		// and then decode "in". 
> 		result =decoder.decode(in,out,true);
> 		out.flip();
> 		assertTrue(result.isUnderflow());
> 		assertEquals("abb",out.toString());
> 	}
> 	/*
> 	 * Mock decoder. It treats byte whose value is less than "0x40" as malformed.
> 	 */
> 	static class MockMalformedDecoder extends java.nio.charset.CharsetDecoder {
> 		public MockMalformedDecoder(Charset cs) {
> 			super(cs, 1, 10);
> 		}
> 		/*
> 		 * It treats byte whose value is less than "0x40" as malformed.
> 		 * Otherwise, it's decoded as 'b'.
> 		 */
> 		protected CoderResult decodeLoop(ByteBuffer in, CharBuffer out) {
> 			System.out.println("Using MockMalformedDecoder");
> 			while (in.hasRemaining()) {
> 				byte b = in.get();
> 				if(b < 0x40){
> 					return CoderResult.malformedForLength(1);
> 				}
> 				if(!out.hasRemaining()){
> 					return CoderResult.OVERFLOW;
> 				}
> 				out.put((char)'b');
> 			}
> 			return CoderResult.UNDERFLOW;
> 		}
> 	}

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

[jira] Commented: (HARMONY-148) java.nio.charset.CharsetDecoder: decode(in,out,endOfInput) method doesn't preserve replace string for successive decode invocation while "out" doesn't have engouh space for replace string.

Posted by "Tim Ellison (JIRA)" <ji...@apache.org>.
    [ http://issues.apache.org/jira/browse/HARMONY-148?page=comments#action_12369217 ] 

Tim Ellison commented on HARMONY-148:
-------------------------------------

Richard,

Fix applied to NIO_CHAR module java.nio.charset.CharsetDecoder at repo revision 383862.

Please can you provide your tests as a patch?  thanks.


> java.nio.charset.CharsetDecoder: decode(in,out,endOfInput) method doesn't preserve replace string for successive decode invocation while "out" doesn't have engouh space for replace string.
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: HARMONY-148
>          URL: http://issues.apache.org/jira/browse/HARMONY-148
>      Project: Harmony
>         Type: Bug
>   Components: Classlib
>     Reporter: Richard Liang
>     Assignee: Tim Ellison
>     Priority: Minor
>  Attachments: CharsetDecoder_patch_148.txt
>
> Consider following senario:
> During the invocation of method java.nio.charset.CharsetDecoder.decode(in,out,false), method "decodeLoop" returns malformed CoderResult, and the action is set as "CodingErrorAction.REPLACE". So replace action should be taken, but "out" doesn't hold enough space for the replace string. Therefore, CoderResult.OVERFLOW should be returned. Harmony does return OVERFLOW currently, but the replace string won't put into new allocated output any more. In other words, the replace string is missing from the view of "stream decoding". Therefore, replace string should be preserved for successive decode(in,out,endOfInput) invacation.
> Both RI5.0 and Harmony fails on the following test case. However, according to the spec, we think it is a defect of RI5.0.
> ================================== Test Case======================
> /*
> 	 * Test the method decode(ByteBuffer) .
> 	 */
> 	public void testDecode_ReplaceOverflow() throws Exception {
> 		String replaceString = "a";
> 		Charset cs = Charset.forName("UTF-8");
> 		MockMalformedDecoder decoder = new MockMalformedDecoder(cs);
> 		decoder.onMalformedInput(CodingErrorAction.REPLACE);
> 		decoder.replaceWith(replaceString);
> 		CharBuffer out = CharBuffer.allocate(1);
> 		// MockMalformedDecoder treats the second byte '0x38' as malformed, 
> 		// but "out" doesn't have enough space for replace string.
> 		ByteBuffer in = ByteBuffer.wrap(new byte[]{0x45,0x38,0x45,0x45});
> 		CoderResult result = decoder.decode(in,out,false);
> 		assertTrue(result.isOverflow());
> 		
> 		// allocate enough space for "out"
> 		out = CharBuffer.allocate(10);
> 		// replace string should be put into "out" firstly,
> 		// and then decode "in". 
> 		result =decoder.decode(in,out,true);
> 		out.flip();
> 		assertTrue(result.isUnderflow());
> 		assertEquals("abb",out.toString());
> 	}
> 	/*
> 	 * Mock decoder. It treats byte whose value is less than "0x40" as malformed.
> 	 */
> 	static class MockMalformedDecoder extends java.nio.charset.CharsetDecoder {
> 		public MockMalformedDecoder(Charset cs) {
> 			super(cs, 1, 10);
> 		}
> 		/*
> 		 * It treats byte whose value is less than "0x40" as malformed.
> 		 * Otherwise, it's decoded as 'b'.
> 		 */
> 		protected CoderResult decodeLoop(ByteBuffer in, CharBuffer out) {
> 			System.out.println("Using MockMalformedDecoder");
> 			while (in.hasRemaining()) {
> 				byte b = in.get();
> 				if(b < 0x40){
> 					return CoderResult.malformedForLength(1);
> 				}
> 				if(!out.hasRemaining()){
> 					return CoderResult.OVERFLOW;
> 				}
> 				out.put((char)'b');
> 			}
> 			return CoderResult.UNDERFLOW;
> 		}
> 	}

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

[jira] Assigned: (HARMONY-148) java.nio.charset.CharsetDecoder: decode(in,out,endOfInput) method doesn't preserve replace string for successive decode invocation while "out" doesn't have engouh space for replace string.

Posted by "Tim Ellison (JIRA)" <ji...@apache.org>.
     [ http://issues.apache.org/jira/browse/HARMONY-148?page=all ]

Tim Ellison reassigned HARMONY-148:
-----------------------------------

    Assign To: Tim Ellison

> java.nio.charset.CharsetDecoder: decode(in,out,endOfInput) method doesn't preserve replace string for successive decode invocation while "out" doesn't have engouh space for replace string.
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>          Key: HARMONY-148
>          URL: http://issues.apache.org/jira/browse/HARMONY-148
>      Project: Harmony
>         Type: Bug
>   Components: Classlib
>     Reporter: Richard Liang
>     Assignee: Tim Ellison
>     Priority: Minor
>  Attachments: CharsetDecoder_patch_148.txt
>
> Consider following senario:
> During the invocation of method java.nio.charset.CharsetDecoder.decode(in,out,false), method "decodeLoop" returns malformed CoderResult, and the action is set as "CodingErrorAction.REPLACE". So replace action should be taken, but "out" doesn't hold enough space for the replace string. Therefore, CoderResult.OVERFLOW should be returned. Harmony does return OVERFLOW currently, but the replace string won't put into new allocated output any more. In other words, the replace string is missing from the view of "stream decoding". Therefore, replace string should be preserved for successive decode(in,out,endOfInput) invacation.
> Both RI5.0 and Harmony fails on the following test case. However, according to the spec, we think it is a defect of RI5.0.
> ================================== Test Case======================
> /*
> 	 * Test the method decode(ByteBuffer) .
> 	 */
> 	public void testDecode_ReplaceOverflow() throws Exception {
> 		String replaceString = "a";
> 		Charset cs = Charset.forName("UTF-8");
> 		MockMalformedDecoder decoder = new MockMalformedDecoder(cs);
> 		decoder.onMalformedInput(CodingErrorAction.REPLACE);
> 		decoder.replaceWith(replaceString);
> 		CharBuffer out = CharBuffer.allocate(1);
> 		// MockMalformedDecoder treats the second byte '0x38' as malformed, 
> 		// but "out" doesn't have enough space for replace string.
> 		ByteBuffer in = ByteBuffer.wrap(new byte[]{0x45,0x38,0x45,0x45});
> 		CoderResult result = decoder.decode(in,out,false);
> 		assertTrue(result.isOverflow());
> 		
> 		// allocate enough space for "out"
> 		out = CharBuffer.allocate(10);
> 		// replace string should be put into "out" firstly,
> 		// and then decode "in". 
> 		result =decoder.decode(in,out,true);
> 		out.flip();
> 		assertTrue(result.isUnderflow());
> 		assertEquals("abb",out.toString());
> 	}
> 	/*
> 	 * Mock decoder. It treats byte whose value is less than "0x40" as malformed.
> 	 */
> 	static class MockMalformedDecoder extends java.nio.charset.CharsetDecoder {
> 		public MockMalformedDecoder(Charset cs) {
> 			super(cs, 1, 10);
> 		}
> 		/*
> 		 * It treats byte whose value is less than "0x40" as malformed.
> 		 * Otherwise, it's decoded as 'b'.
> 		 */
> 		protected CoderResult decodeLoop(ByteBuffer in, CharBuffer out) {
> 			System.out.println("Using MockMalformedDecoder");
> 			while (in.hasRemaining()) {
> 				byte b = in.get();
> 				if(b < 0x40){
> 					return CoderResult.malformedForLength(1);
> 				}
> 				if(!out.hasRemaining()){
> 					return CoderResult.OVERFLOW;
> 				}
> 				out.put((char)'b');
> 			}
> 			return CoderResult.UNDERFLOW;
> 		}
> 	}

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira