You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Tim Allison (JIRA)" <ji...@apache.org> on 2015/12/09 18:07:10 UTC
[jira] [Comment Edited] (TIKA-1599) Switch from TagSoup to JSoup
[ https://issues.apache.org/jira/browse/TIKA-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048978#comment-15048978 ]
Tim Allison edited comment on TIKA-1599 at 12/9/15 5:06 PM:
------------------------------------------------------------
bq. So I'm not seeing how the parser that CommonCrawl uses would factor into that.
Right, agree. I was just seeing if they had also picked up JSoup or were using something else.
bq. But I also don't have a good suggestion for how to automatically evaluate the TagSoup vs. JSoup parse-off, other than simple measures like how many failed, or maybe the amount of text extracted?
Y, agreed. For PDFBOX-3058, and as part of TIKA-1332, I have some alpha-level eval code that will do these comparisons. Following [~tilman]'s recommendation, I added counts for "common English" words.
was (Author: tallison@mitre.org):
bq. So I'm not seeing how the parser that CommonCrawl uses would factor into that.
Right, agree. I was just seeing if they had also picked up JSoup or were using something else.
bq. But I also don't have a good suggestion for how to automatically evaluate the TagSoup vs. JSoup parse-off, other than simple measures like how many failed, or maybe the amount of text extracted?
Y, agreed. For PDFBOX-3058, and as part of TIKA-1330, I have some alpha-level eval code that will do these comparisons. Following [~tilman]'s recommendation, I added counts for "common English" words.
> Switch from TagSoup to JSoup
> ----------------------------
>
> Key: TIKA-1599
> URL: https://issues.apache.org/jira/browse/TIKA-1599
> Project: Tika
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.7, 1.8
> Reporter: Ken Krugler
> Assignee: Ken Krugler
> Priority: Minor
>
> There are several Tika issues related to how TagSoup cleans up HTML ([TIKA-381], [TIKA-985], maybe [TIKA-715]), but TagSoup doesn't seem to be under active development.
> On the other hand I know of several projects that are now using [JSoup|https://github.com/jhy/jsoup], which is an active project (albeit only one main contributor) under the MIT license.
> I haven't looked into how hard it would be to switch this dependency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)