[jira] [Commented] (NUTCH-2603) Bring back legacy pre-Tika parsers and use them as back up parsers

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518239#comment-16518239 ] 

Sebastian Nagel commented on NUTCH-2603:
----------------------------------------

Hi [~ArkadiKosmynin], this is in contradiction with my experience: I've used Nutch since version 0.9 (without parse-tika) to process also PDFs and office documents. Most issues with these documents disappeared over time while Tika has become mature. 

I've tried to reproduce any parsing issues using the recent Nutch master branch (with Tika 1.18) by picking randomly documents parsed with one of the "legacy" parsers. 
- at least, some documents require authentication, e.g. [1|https://www.atnf.csiro.au/lists/at_meetings/2006/att-0064/CJ_SKA_Industry_Astro_update_Apr_06.ppt], [2|https://www.atnf.csiro.au/lists/at_meetings/2008/att-0001/ASKAP_Antennas_summary.doc], [3|https://svn.atnf.csiro.au/askap/ASKAPDesignEnhancements/PCB/790-0015-Bullant/Datasheets/Control.xls]. [~ArkadiKosmynin], could you provide some of these documents or remove the access restrictions?
- among the remaining URLs there is one systematic error: the server behind www.atnf.csiro.au regularly sends {{application/msword}} for plain-text documents ending in {{.doc}}, eg. [update.doc|https://www.atnf.csiro.au/computing/software/gipsy/doc/update.doc]. Tika can parse this document, opened NUTCH-2606 to address this.
- Tika fails to parse MS Word 2.0 documents: [zenpap4.doc|http://www.atnf.csiro.au/pub/data/pmn/zenpap4.doc]. It's a known issue that Tika (resp. POI) cannot parse old MS Office document, see TIKA-2107. However, I doubt that they have been successfully parsed using the "legacy" parsers:
-* testing with Nutch 1.0 and parse-msword I get a failed parse with the error:
{noformat}
Can't be handled as Microsoft document. java.io.IOException: Invalid header signature; read 867295287388775899, expected -2226271756974174256
{noformat}
-* similar as using  the recent Nutch master and Tika 1.18:
{noformat}
% bin/nutch parsechecker -Dplugin.includes='protocol-http|parse-tika' -forceAs application/msword http://www.atnf.csiro.au/pub/data/pmn/zenpap4.doc
...
Status: failed(2,0): Invalid header signature; read 0x0C094078002DA5DB, expected 0xE11AB1A1E011CFD0 - Your file appears not to be a valid OLE2 document
{noformat}
-* when forcing the OpenDocumentParser, the result is a successful but empty parse, both for recent Nutch/Tika as well as Nutch 1.0 and parse-oo:
{noformat}
% bin/ nutch parsechecker -Dplugin.includes='protocol-http|parse-tika'  -forceAs application/vnd.oasis.opendocument.text http://www.atnf.csiro.au/pub/data/pmn/zenpap4.doc
...
Status: success(1,0)
Title: 
Outlinks: 0
{noformat}
resp.
{noformat}
% nutch-1.0/bin/nutch org.apache.nutch.parse.ParserChecker -forceAs application/vnd.oasis.opendocument.text http://www.atnf.csiro.au/pub/data/pmn/zenpap4.doc
...
Status: success(1,0)
Title: 
Outlinks: 0
...
{noformat}
same when calling the main routine of the plugin parse-oo directly (with a local file):
{noformat}
% nutch-1.0/bin/nutch plugin parse-oo org.apache.nutch.parse.oo.OOParser .../zenpap4.doc
Version: 5
Status: success(1,0)
Title: 
Outlinks: 0
Content Metadata: 
Parse Metadata: 

Text: ''
{noformat}
I've opened TIKA-2675 to address this problem. [~ArkadiKosmynin], is it possible that the message in the attached public_docs.txt is also misleading?
{noformat}
arch.log.2018-06-15:2018-06-15 16:05:49,686 INFO  parse.ParseUtil - Successfully parsed [http://www.atnf.csiro.au/pub/data/pmn/zenpap4.doc] with [org.apache.nutch.parse.oo.OOParser@380d5de2]
{noformat}

So far, these are the only problems I've seen so far. Please open separate issues for other problems. I strongly opt for fixing issues, instead of maintaining potentially buggy legacy parsers.

> Bring back legacy pre-Tika parsers and use them as back up parsers
> ------------------------------------------------------------------
>
>                 Key: NUTCH-2603
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2603
>             Project: Nutch
>          Issue Type: Improvement
>          Components: parser
>    Affects Versions: 1.15
>            Reporter: Arkadi Kosmynin
>            Priority: Major
>         Attachments: public_docs.txt
>
>
> There are cases when legacy parsers successfully parse documents on which Tika fails. I am attaching a list of examples of such documents. Nutch allows use of more than one parser on a document, in a sequence, until the document has been parsed successfully. Thus, old parsers can be combined with Tika to achieve better parsing success rate, at least until Tika is perfect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)