You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tika.apache.org by "Tim Allison (JIRA)" <ji...@apache.org> on 2014/05/21 13:45:39 UTC
[jira] [Comment Edited] (TIKA-1302) Let's run Tika against a large
batch of docs nightly
[ https://issues.apache.org/jira/browse/TIKA-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004592#comment-14004592 ]
Tim Allison edited comment on TIKA-1302 at 5/21/14 11:44 AM:
-------------------------------------------------------------
[~jnioche], very cool corpus. My dream would be to run Tika via hadoop against a corpus that big, diverse and noisy whenever there's a commit (or better yet, let developers upload their mods as a tika-server, go for a cup of coffee and come back to see the results). For initial steps, govdocs1 seems like a great start...perhaps we could include a random sample from CC downloaded to Apache servers if that is consistent with both Apache and CC licenses?
[~chrismattmann], thank you for pointing out [~gostep]'s work!
[~gostep], please post a link to your work to this issue when it is published. Are there any evaluation components that you'd like to contribute? Do you think there would be a way to share your datasets? And, finally, I'm not sure what you mean by all of the metadata is the same. I am just getting started with the govdocs corpus...when I open up two different pdf files, the have different pdf versions, different authors, different producers.
was (Author: tallison@mitre.org):
[~jnioche], very cool corpus. My dream would be to run Tika via hadoop against a corpus that big, diverse and noisy whenever there's a commit (or better yet, let developers upload their mods as a tika-server, go for a cup of coffee and come back to see the results). For initial steps, govdocs1 seems like a decent start...perhaps we could include a random sample from CC downloaded to Apache servers if that is consistent with both Apache and CC licenses?
[~chrismattmann], thank you for pointing out [~gostep]'s work!
[~gostep], please post a link to your work to this issue when it is published. Are there any evaluation components that you'd like to contribute? Do you think there would be a way to share your datasets?
> Let's run Tika against a large batch of docs nightly
> ----------------------------------------------------
>
> Key: TIKA-1302
> URL: https://issues.apache.org/jira/browse/TIKA-1302
> Project: Tika
> Issue Type: Improvement
> Reporter: Tim Allison
>
> Many thanks to [~lewismc] for TIKA-1301! Once we get nightly builds up and running again, it might be fun to run Tika regularly against a large set of docs and report metrics.
> One excellent candidate corpus is govdocs1: http://digitalcorpora.org/corpora/files.
> Any other candidate corpora?
> [~willp-bl], have anything handy you'd like to contribute?
> [http://www.openplanetsfoundation.org/blogs/2014-03-21-tika-ride-characterising-web-content-nanite]
> ;)
--
This message was sent by Atlassian JIRA
(v6.2#6252)