[jira] [Commented] (TEXT-192) HTML unescape does not parse Windows-1252 correctly

["Bruno P. Kinoshita (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEXT-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248247#comment-17248247 ] 

Bruno P. Kinoshita commented on TEXT-192:
-----------------------------------------

Hi [~cpars509]

Thanks for the well detailed issue report.

>Looking at the source for how StringEscapeUtils.UNESCAPE_HTML4 works, I think (big question marks here) that an escape set needs to be added for Windows-1252?

I believe you are correct. That field is created for ISO-8859-1. You could probably create a similar object, but creating a new entry in EntityArrays for Windows-1252. Maybe we could have that added via a PR, to support windows encoding by default.

 

Cheers

Bruno

> HTML unescape does not parse Windows-1252 correctly
> ---------------------------------------------------
>
>                 Key: TEXT-192
>                 URL: https://issues.apache.org/jira/browse/TEXT-192
>             Project: Commons Text
>          Issue Type: Bug
>    Affects Versions: 1.8
>         Environment: Java, macOS; should not be platform specific
>            Reporter: Cyril Parsons
>            Priority: Minor
>
> Looking at [https://en.wikipedia.org/wiki/Windows-1252#Character_set] there are differences in parsing Windows-1252 and ISO 8859-1. Code points between 128 and 159 (on Windows-1252) are improperly decoded.
> In a MMVE:
> {code:java}
> import org.apache.commons.text.StringEscapeUtils;
> ...
> String w1252 = "~—¡";
> String output = StringEscapeUtils.unescapeHtml4(w1252);
> System.out.println(output);
> System.out.println(output.chars().mapToLong(Long::valueOf)
>         .boxed().collect(Collectors.toList()));
> {code}
> The output is:
> {code:java}
> ~— ¡
> [126, 151, 161]
> {code}
> (Space substituted for the Unicode character "End Of Guarded Area".) Expected output would be that em-dash would appear. Code points right outside of the Windows-1252/ISO 8859-1 inconsistency zone are all okay. Looking at the source for how StringEscapeUtils.UNESCAPE_HTML4 works, I think (big question marks here) that an escape set needs to be added for Windows-1252?
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)