You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jakarta.apache.org by bu...@apache.org on 2011/08/10 21:48:30 UTC
DO NOT REPLY [Bug 51645] New: CSVDataSet does not read UTF-8 files
when file.encoding is UTF-8
https://issues.apache.org/bugzilla/show_bug.cgi?id=51645
Bug #: 51645
Summary: CSVDataSet does not read UTF-8 files when
file.encoding is UTF-8
Product: JMeter
Version: 2.4
Platform: All
OS/Version: All
Status: NEW
Severity: major
Priority: P2
Component: Main
AssignedTo: notifications@jakarta.apache.org
ReportedBy: apache_bugzilla@zwiers.ca
Classification: Unclassified
Created attachment 27366
--> https://issues.apache.org/bugzilla/attachment.cgi?id=27366
Patch to fix issue. Variable not renamed to show just a matter of replacing
class.
CSV Data Sets which are encoded in UTF-8 do not work on platforms where the
default file.encoding is UTF-8.
UTF-8 is used to illustrate here, but this would presumably apply to other
non-8bit character sets as well.
Reason: The use of ByteArrayOutputStream in the CSVSaveService.csvReadFile()
method. Specifically, the boas.write(ch) call is implemented (internally in
ByteArrayOutputStream) with a cast to the byte primitive type ( buf[count] =
(byte)b; in my JVM).
Later, the ByteArrayOutputStream is interpreted according to the platform
default (via baos.toString()) and if the content of the array are then
interpreted according to the platform's default char set. If that charset (eg.
ISO-8859-1) is 8-bit, everything is fine. However, unpredictable
results/unmapped chars result for other charsets (like UTF-8).
For example, the character \u0027 (LATIN SMALL LETTER C WITH CEDILLA) with
decimal code point 231. When put into boas, it becomes (7 bit signed) -25.
When converted via toString() with UTF-8 as the default char set, the value is
not recognized as a valid code point and the value \ufffd (decimal code point
65533 == Unicodes "REPLACEMENT CHARACTER" placeholder) is placed in the return
string instead.
Fix: patche attached. Simply replace ByteArrayOutputStream with CharArrayWriter
and the UTF-8 files work regardless of the value for file.encoding.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@jakarta.apache.org
For additional commands, e-mail: notifications-help@jakarta.apache.org
DO NOT REPLY [Bug 51645] CSVDataSet does not read UTF-8 files when
file.encoding is UTF-8
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51645
--- Comment #2 from JZ <ap...@zwiers.ca> 2011-08-10 19:58:17 UTC ---
Created attachment 27368
--> https://issues.apache.org/bugzilla/attachment.cgi?id=27368
.csv file for previous test patch
Required for one of the previously attached tests. Belongs in bin/testfiles
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@jakarta.apache.org
For additional commands, e-mail: notifications-help@jakarta.apache.org
DO NOT REPLY [Bug 51645] CSVDataSet does not read UTF-8 files when
file.encoding is UTF-8
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51645
Sebb <se...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #3 from Sebb <se...@apache.org> 2011-08-11 00:37:08 UTC ---
Thanks very much.
Patch applied:
URL: http://svn.apache.org/viewvc?rev=1156416&view=rev
Log:
Bug 51645 - CSVDataSet does not read UTF-8 files when file.encoding is UTF-8
Added:
jakarta/jmeter/trunk/bin/testfiles/testutf8.csv (with props)
Modified:
jakarta/jmeter/trunk/src/core/org/apache/jmeter/save/CSVSaveService.java
jakarta/jmeter/trunk/test/src/org/apache/jmeter/config/TestCVSDataSet.java
jakarta/jmeter/trunk/test/src/org/apache/jmeter/save/TestCSVSaveService.java
jakarta/jmeter/trunk/xdocs/changes.xml
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@jakarta.apache.org
For additional commands, e-mail: notifications-help@jakarta.apache.org
DO NOT REPLY [Bug 51645] CSVDataSet does not read UTF-8 files when
file.encoding is UTF-8
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=51645
--- Comment #1 from JZ <ap...@zwiers.ca> 2011-08-10 19:56:50 UTC ---
Created attachment 27367
--> https://issues.apache.org/bugzilla/attachment.cgi?id=27367
Test cases to expose bug. Run with file.encoding=UTF-8
Tests will execute successfully if default file.encoding is ISO-8859-1 (or
other 8bit that can handle the chars in the test). However, run with
-Dfile.encoding=UTF-8 VM arg and tests will fail. Requires new
bin/testfiles/testutf8.csv (attached next).
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@jakarta.apache.org
For additional commands, e-mail: notifications-help@jakarta.apache.org