[jira] [Commented] (NUTCH-2891) Upgrade to Tika 2.1

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17449249#comment-17449249 ] 

ASF GitHub Bot commented on NUTCH-2891:
---------------------------------------

sebastian-nagel commented on pull request #700:
URL: https://github.com/apache/nutch/pull/700#issuecomment-979330868


   Hi @lewismc, in addition to running local tests on just a few documents
   - a comparison of Tika parse results for an equivalent upgrade (/Digitalpebble/storm-crawler#920) showed that any custom extractors need to review metadata names. But the consolidation of Tika metadata names/keys is for sure an improvement for the long term.
   - just upgraded the [Common Crawl Nutch fork](/commoncrawl/nutch) and now running a crawl (several 100 million pages) with parse-tika enabled. I'll look for suspicious exceptions in the log files and will report tomorrow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Upgrade to Tika 2.1
> -------------------
>
>                 Key: NUTCH-2891
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2891
>             Project: Nutch
>          Issue Type: Improvement
>          Components: parser, plugin
>    Affects Versions: 1.18
>            Reporter: Sebastian Nagel
>            Assignee: Sebastian Nagel
>            Priority: Major
>             Fix For: 1.19
>
>
> There's already the second release of Tika 2 ([2.1.0|https://tika.apache.org/2.1.0/index.html]). Following the [2.0 release notes|https://archive.apache.org/dist/tika/2.0.0/CHANGES-2.0.0.txt] and the [migration guide|https://cwiki.apache.org/confluence/display/TIKA/Migrating+to+Tika+2.0.0]:
> * Tika 2 is more modular which should allow us to build a smaller parse-tika (66 MiB in the 1.18 binary package) by dropping rarely used parsers - but users should be able to include them if they build Nutch from the sources.
> * the language-identifier plugin needs to be upgraded as well (in addition to Nutch core and the parse-tika plugin). This would include or overlap with NUTCH-2449.
> * to avoid that the PDF parser times out we probably want to disable the OCR by default, or at least, provide the configuration snippet for this purpose



--
This message was sent by Atlassian Jira
(v8.20.1#820001)