You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@poi.apache.org by Christian Gosch <ch...@inovex.de> on 2008/10/22 18:23:02 UTC

Handling the "s with caron" (u0160 / xA9)

Hi,

we have an application which is not really "international" but contains 
data with personal names from different countries, also from Czech 
Republic. They have a character not present in ISO-8859-1, but in 
Cp1252, which is the encoding used on our clients' machines. The 
character in question is u0161 / xB9 "" and its counterpart, the u0160 
/ xA9.

We have some names with that character in the data, and in the browser 
it shows up OK on German Windows platforms, since they use Cp1252, and 
in the Oracle WEISO88591 char set of the DB server it is just located on 
a "hole" of ISO-8859-1 -- simply a meaningless binary code.

BUT: When building an XLS file with POI (3.1-final and 3.2-final tested, 
3.1-final with setEncoding(UTF8) -> setCellValue(String), 3.2-final with 
setCellValue(HSSFRichString(String)) ), this character shows up as a 
narrow caret.

What happens to this character when putting it in a cell using 
HSSFCell.setCellVaue(new HSSFRichString((String)theValue)) ?

When I look in the binary file using TextPad 4.7.3, it is displayed as 
x9A, also shown as black narrow caret in the "visible characters column" 
of the hex view.

Are there any other characters or binary codes which are handled the 
same way by POI and thus are not really readable in a (POI-)generated 
XLS file?

Thanks for answers,
-- 
Dipl.-Inform. Christian Gosch, PMI PMP
Systems Architecture, Project Management

inovex GmbH
Bro Pforzheim
Karlsruher Strasse 71
D-75179 Pforzheim
Tel: +49 (0)7231 3191-85
Fax: +49 (0)7231 3191-91
c.gosch@inovex.de
www.inovex.de

Sitz der Gesellschaft: Pforzheim
AG Mannheim, HRB 502126
Geschftsfhrer: Stephan Mller 



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org


RE: Handling the "s with caron" (u0160 / xA9)

Posted by Nick Burch <ni...@torchbox.com>.
On Wed, 22 Oct 2008, Christian Gosch wrote:
> Thus maybe the HSSFRichString(String) constructor may have a little flaw 
> on deciding whether to handle a given String as UTF16 or not: It should 
> decide not only on the binary numeric value ( > 255 or not) but rely on 
> Java to check if it is part of a reasonable 8bit charset. This is 
> because there are different 8bit charsets which have "holes" with a 
> "meaning" in other widespread charsets and vice versa.

Any chance you could write a little test case that shows this, and upload 
it to bugzilla? Something that creates a new workbook, and populates some 
high bit characters into it would be great. We can then use that to test 
possible fixes against

Nick

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org


RE: Handling the "s with caron" (u0160 / xA9)

Posted by Christian Gosch <ch...@inovex.de>.
Addition:

I copied the u0161 / x9A from the browser into the sheet when open in 
Excel 2003 and saved a copy of the file and did the same with OOo 3.0. 
Then I reviewed the results in TextPad as binary files. Interesting was: 
MS Excel stores the value behind all other "normal" data in a separate 
block, like a special block for the needed encoding. OOo 3 stored the 
value "inline". But both converted the value to 16bit representation, as 
opposite to all other data which was stored in 8bit fashion.

Thus maybe the HSSFRichString(String) constructor may have a little flaw 
on deciding whether to handle a given String as UTF16 or not: It should 
decide not only on the binary numeric value ( > 255 or not) but rely on 
Java to check if it is part of a reasonable 8bit charset. This is 
because there are different 8bit charsets which have "holes" with a 
"meaning" in other widespread charsets and vice versa.

At the end, the user/client/developer eventually should get back a 
chance to decide which encoding to use, at least optionally.

Regards,
Christian Gosch
-- 
(footer as below)

> -----Original Message-----
> From: Christian Gosch
> Sent: Wednesday, October 22, 2008 6:23 PM
> To: user
> Subject: Handling the "s with caron" (u0160 / xA9)
> 
> Hi,
> 
> we have an application which is not really "international" but 
contains
> data with personal names from different countries, also from Czech
> Republic. They have a character not present in ISO-8859-1, but in
> Cp1252, which is the encoding used on our clients' machines. The
> character in question is u0161 / x9A "" and its counterpart, the u0160
> / x9B.
> 
> We have some names with that character in the data, and in the browser
> it shows up OK on German Windows platforms, since they use Cp1252, and
> in the Oracle WEISO88591 char set of the DB server it is just located 
on
> a "hole" of ISO-8859-1 -- simply a meaningless binary code.
> 
> BUT: When building an XLS file with POI (3.1-final and 3.2-final 
tested,
> 3.1-final with setEncoding(UTF8) -> setCellValue(String), 3.2-final 
with
> setCellValue(HSSFRichString(String)) ), this character shows up as a
> narrow caret.
> 
> What happens to this character when putting it in a cell using
> HSSFCell.setCellVaue(new HSSFRichString((String)theValue)) ?
> 
> When I look in the binary file using TextPad 4.7.3, it is displayed as
> x9A, also shown as black narrow caret in the "visible characters 
column"
> of the hex view.
> 
> Are there any other characters or binary codes which are handled the
> same way by POI and thus are not really readable in a (POI-)generated
> XLS file?
> 
> Thanks for answers,
> --
> Dipl.-Inform. Christian Gosch, PMI PMP
> Systems Architecture, Project Management
> 
> inovex GmbH
> Bro Pforzheim
> Karlsruher Strasse 71
> D-75179 Pforzheim
> Tel: +49 (0)7231 3191-85
> Fax: +49 (0)7231 3191-91
> c.gosch@inovex.de
> www.inovex.de
> 
> Sitz der Gesellschaft: Pforzheim
> AG Mannheim, HRB 502126
> Geschftsfhrer: Stephan Mller
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
> For additional commands, e-mail: user-help@poi.apache.org
> 
> 
> !DSPAM:48ff540222933942915773!
> 
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org


RE: Handling the "s with caron" (u0160 / xA9)

Posted by Christian Gosch <ch...@inovex.de>.
Hi,

if you managed to find *this*, you may be interested in the discussion 
on topic 

"Old versions of POI available for download?"

There are some useful hints and ideas presented. I will not present a 
test case any more on this because (1) my problem is solved and (2) it 
is not really a problem with POI but with a somewhat dangerous mix of 
Cp1252 and ISO-8859-1. See the mentioned discussion for details.

Regards,
--cg

> -----Original Message-----
> From: Christian Gosch
> Sent: Wednesday, October 22, 2008 6:24 PM
> To: 'POI Users List'
> Subject: Handling the "s with caron" (u0160 / xA9)
> 
> Hi,
> 
> we have an application which is not really "international" but 
contains
> data with personal names from different countries, also from Czech
> Republic. They have a character not present in ISO-8859-1, but in 
Cp1252,
> which is the encoding used on our clients' machines. The character in
> question is u0161 / xB9 "" and its counterpart, the u0160 / xA9.
> 
> We have some names with that character in the data, and in the browser 
it
> shows up OK on German Windows platforms, since they use Cp1252, and in 
the
> Oracle WEISO88591 char set of the DB server it is just located on a 
"hole"
> of ISO-8859-1 -- simply a meaningless binary code.
> 
> BUT: When building an XLS file with POI (3.1-final and 3.2-final 
tested,
> 3.1-final with setEncoding(UTF8) -> setCellValue(String), 3.2-final 
with
> setCellValue(HSSFRichString(String)) ), this character shows up as a
> narrow caret.
> 
> What happens to this character when putting it in a cell using
> HSSFCell.setCellVaue(new HSSFRichString((String)theValue)) ?
> 
> When I look in the binary file using TextPad 4.7.3, it is displayed as
> x9A, also shown as black narrow caret in the "visible characters 
column"
> of the hex view.
> 
> Are there any other characters or binary codes which are handled the 
same
> way by POI and thus are not really readable in a (POI-)generated XLS 
file?
> 
> Thanks for answers,
> --
> Dipl.-Inform. Christian Gosch, PMI PMP
> Systems Architecture, Project Management
> 
> inovex GmbH
> Bro Pforzheim
> Karlsruher Strasse 71
> D-75179 Pforzheim
> Tel: +49 (0)7231 3191-85
> Fax: +49 (0)7231 3191-91
> c.gosch@inovex.de
> www.inovex.de
> 
> Sitz der Gesellschaft: Pforzheim
> AG Mannheim, HRB 502126
> Geschftsfhrer: Stephan Mller
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@poi.apache.org
For additional commands, e-mail: user-help@poi.apache.org