You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@any23.apache.org by HansBrende <gi...@git.apache.org> on 2018/11/06 21:48:18 UTC
[GitHub] any23 pull request #131: ANY23-418 improve TikaEncodingDetector
GitHub user HansBrende opened a pull request:
https://github.com/apache/any23/pull/131
ANY23-418 improve TikaEncodingDetector
Improves TikaEncodingDetector by:
1. Not second-guessing UTF-8 if there is *any* indication that a stream is UTF-8-encoded. We can't afford false positives from obscure, obsolete charsets such as IBM500 (See [TIKA-2771](https://issues.apache.org/jira/browse/TIKA-2771)).
2. Taking entire stream into account rather than a prefix (this shouldn't be a huge memory issue, as we are already holding the entire stream in memory to pass to each extractor, and extractors such as RDFa already parse the entire content into a DOM before generating the triples. If we want to make Any23 "streaming"-capable in the future to reduce memory requirements, we can look into that, but for now, since we're not, we may as well use that to our advantage to be more accurate in charset detection.)
3. Taking [TIKA-2771](https://issues.apache.org/jira/browse/TIKA-2771), [TIKA-2038](https://issues.apache.org/jira/browse/TIKA-2038), and [TIKA-539](https://issues.apache.org/jira/browse/TIKA-539) into account.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HansBrende/any23 ANY23-418
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/any23/pull/131.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #131
----
commit d64dac9dfe0752c45d3ff9fbca37bbe447e5c55b
Author: Hans <fi...@...>
Date: 2018-11-06T21:27:00Z
ANY23-418 improve TikaEncodingDetector
----
---
[GitHub] any23 issue #131: ANY23-418 improve TikaEncodingDetector
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/any23/pull/131
You've brought up an excellent topic for conversation. Tika currently has a batch, regression job which essentially enables them to run over loads of documents and analyze the output. The result being that they know how changes to the source are affecting Tika's ability to do what it claims it is doing over time. We do not have that in Any23 but I think we should make an effort to build bridges withe the Tika community in this regard with the aim of us sharing resources (both available computing to run large batch parse jobs, as well as dataset(s) we can use to run Any23 over.)
I have been thinking for the longest time now about implementing a ```tika.triplify``` API which would encapsulate Any23 run it on the Tika data streams but I just never got around to it. Maybe now is a better time to bring that idea back to life.
I was thinking we could possibly use common crawl but they do not publish the raw data AFAIK it is the Nutch segments or some alternative e.g. the WebArchive files.
---
[GitHub] any23 issue #131: ANY23-418 improve TikaEncodingDetector
Posted by HansBrende <gi...@git.apache.org>.
Github user HansBrende commented on the issue:
https://github.com/apache/any23/pull/131
@lewismc I've simplified the code a lot so it should be a whole lot easier to see what's going on now.
Also, I improved the UTF-8 detector by reverse engineering jchardet's methodology for UTF-8 detection, and created a UTF-8 state machine which does the same thing as jchardet (in a much more human-readable manner), plus fixed two bugs in jchardet's UTF-8 detector along the way (possibly due to the lack of human-readability in the original source code).
I started looking into jchardet because, according to [TIKA-2038](https://issues.apache.org/jira/browse/TIKA-2038), using it to detect UTF-8 before anything else increased the accuracy of charset detection from ~72% to ~96%.
Our encoding detector should now be at least as accurate.
Any thoughts on the methodology, as compared to what we had before?
---
[GitHub] any23 issue #131: ANY23-418 improve TikaEncodingDetector
Posted by lewismc <gi...@git.apache.org>.
Github user lewismc commented on the issue:
https://github.com/apache/any23/pull/131
There is a fair bit of code here but I am not really sure how to test it. Unfortunately I am going to say, please provide unit test. I've been aware of some encoding detection issues previously with Any23 but never got around to logging them here.
---
[GitHub] any23 issue #131: ANY23-418 improve TikaEncodingDetector
Posted by HansBrende <gi...@git.apache.org>.
Github user HansBrende commented on the issue:
https://github.com/apache/any23/pull/131
@lewismc I've added some additional unit tests which test against the main issues we've been having with encoding detection.
Unfortunately, the only real way to comprehensively test this is to compare against millions of webpages "in the wild", but I am confident that it represents a huge improvement over what we have *now*, based on our past problems with encoding detection, plus discussions over in Tika regarding the various issues *they've* been having with encoding detection.
Compare to the original version of this file [here](https://github.com/apache/any23/blob/bd607c1cc8c63225f9678ec967c73daa474b45aa/encoding/src/main/java/org/apache/any23/encoding/TikaEncodingDetector.java).
Since that time, I've made a couple changes to the algorithm to fix up problems we've encountered along the way, but those tweaks weren't as comprehensive as this one is.
Ideally, I'd like to compare this more comprehensive solution against our original solution across millions of webpages, but I'm not yet sure how to proceed in that regard.
---
[GitHub] any23 issue #131: ANY23-418 improve TikaEncodingDetector
Posted by HansBrende <gi...@git.apache.org>.
Github user HansBrende commented on the issue:
https://github.com/apache/any23/pull/131
@lewismc any thoughts about this?
---