You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by GitBox <gi...@apache.org> on 2021/10/18 13:57:14 UTC
[GitHub] [nutch] sebastian-nagel opened a new pull request #700: NUTCH-2891 Upgrade to Tika 2.1.0
sebastian-nagel opened a new pull request #700:
URL: https://github.com/apache/nutch/pull/700
- upgrade Nutch core and the plugins parse-tika and language-identifier
- parse-tika uses on "tika-parsers-standard-package" (no extended and scientific parsers)
- the language-identifier is now based on [Tika's wrapper around Optimaize](https://tika.apache.org/2.1.0/api/org/apache/tika/langdetect/optimaize/OptimaizeLangDetector.html) (overlaps with / obsoletes NUTCH-2449)
- tika-config.xml
- disable Tesseract OCR
- add work-around for TIKA-3551
I'll continue testing during the next days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel merged pull request #700: NUTCH-2891 Upgrade to Tika 2.1.0
Posted by GitBox <gi...@apache.org>.
sebastian-nagel merged pull request #700:
URL: https://github.com/apache/nutch/pull/700
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc edited a comment on pull request #700: NUTCH-2891 Upgrade to Tika 2.1.0
Posted by GitBox <gi...@apache.org>.
lewismc edited a comment on pull request #700:
URL: https://github.com/apache/nutch/pull/700#issuecomment-982967509
Hi @sebastian-nagel I didn't encounter `NoClassDefFoundError` because I am not using that parser.
My ongoing testing has not uncovered anything else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on pull request #700: NUTCH-2891 Upgrade to Tika 2.1.0
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on pull request #700:
URL: https://github.com/apache/nutch/pull/700#issuecomment-979330868
Hi @lewismc, in addition to running local tests on just a few documents
- a comparison of Tika parse results for an equivalent upgrade (/Digitalpebble/storm-crawler#920) showed that any custom extractors need to review metadata names. But the consolidation of Tika metadata names/keys is for sure an improvement for the long term.
- just upgraded the [Common Crawl Nutch fork](/commoncrawl/nutch) and now running a crawl (several 100 million pages) with parse-tika enabled. I'll look for suspicious exceptions in the log files and will report tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc commented on pull request #700: NUTCH-2891 Upgrade to Tika 2.1.0
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #700:
URL: https://github.com/apache/nutch/pull/700#issuecomment-946851661
Will test shortly. Thanks @sebastian-nagel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] lewismc commented on pull request #700: NUTCH-2891 Upgrade to Tika 2.1.0
Posted by GitBox <gi...@apache.org>.
lewismc commented on pull request #700:
URL: https://github.com/apache/nutch/pull/700#issuecomment-982967509
Hi @sebastian-nagel I didn't encounter `NoClassDefFoundError` because I am not using that parser.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on pull request #700: NUTCH-2891 Upgrade to Tika 2.1.0
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on pull request #700:
URL: https://github.com/apache/nutch/pull/700#issuecomment-982797518
Ok, the test crawl was successful. The only dubious errors which need a look at are NoClassDefFoundError parsing files of MIME type `application/x-7z-compressed`. E.g. reproducible by parsing one of the Tika test documents:
```
bin/nutch parsechecker -Dplugin.includes='protocol-file|parse-tika' file:/path/to/tika/repo/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pkg-module/src/test/resources/test-documents/test-documents.7z
```
Fixed by removing common-codec and commons-compress from the exclusions. Although we have the same versions in core and parse-tika dependencies. Obviously an issue with the plugin and core/parent class loaders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org