You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by GitBox <gi...@apache.org> on 2020/08/13 07:35:01 UTC
[GitHub] [tika] PeterAlfredLee opened a new pull request #338: Tika-2421 : About the encoding of HTML
PeterAlfredLee opened a new pull request #338:
URL: https://github.com/apache/tika/pull/338
Seems we can use `charsetdetector.StandardHtmlEncodingDetector` for charset detecting of HTML. I'm wondering why we are not using it?
And I stopped treating ISO-8859-1 as Windows-1252.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tballison commented on pull request #338: Tika-2421 : About the encoding of HTML
Posted by GitBox <gi...@apache.org>.
tballison commented on pull request #338:
URL: https://github.com/apache/tika/pull/338#issuecomment-686610377
Wait, it turns out I did get around to doing this study...
https://github.com/tballison/share/blob/main/slides/Tika_charset_detector_study_201909.docx
Let me read it and remember what I found... :rofl:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] PeterAlfredLee commented on pull request #338: Tika-2421 : About the encoding of HTML
Posted by GitBox <gi...@apache.org>.
PeterAlfredLee commented on pull request #338:
URL: https://github.com/apache/tika/pull/338#issuecomment-673823280
Like [TIKA-2421](https://issues.apache.org/jira/browse/TIKA-2421) says , according to [w3 description](https://www.w3.org/International/questions/qa-html-encoding-declarations#utf16) , we should read html byte mark order first.
If there is no BOM , that means it is ASCII-compatible , then we can read this html's meta tag with ACSII and get charset.
HtmlEncodingDetector will not read html's BOM first , it assume html's meta tag is ASCII-compatible.
StandardHtmlEncodingDetector will read BOM first , then read metadata if there is no BOM , then read meta tag if no charset in metadata.
So I think use StandardHtmlEncodingDetector is more compliant to the w3 standard.
Only problem I can see is StandardHtmlEncodingDetector treating ISO-8859-1 as Windows-1252 , I have modify that in this PR.
So I think we can change StandardHtmlEncodingDetector as default detector.
Or we can modify HtmlEncodingDetector to compliant to w3 standard. WDYT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tika] tballison commented on pull request #338: Tika-2421 : About the encoding of HTML
Posted by GitBox <gi...@apache.org>.
tballison commented on pull request #338:
URL: https://github.com/apache/tika/pull/338#issuecomment-673508020
Inertia... I never got around to doing a bakeoff between the two, and, unless there's evidence of improvement, I'm hesitant to make the change as the default detector.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org