[jira] Updated: (NUTCH-562) Port mime type framework to use Tika mime detection framework

["Chris A. Mattmann (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris A. Mattmann updated NUTCH-562:
------------------------------------

    Attachment: NUTCH-562.Mattmann.patch.txt

Initial patch for comments:

1. This patch removes the MimeType system, and its associated java src files, config files and unit tests from Nutch. This information is in Tika now and is replaced by its TIka counterparts.
2. This patch uses the unreleased 0.1-dev version of Tika. When 0.1 is officially released, we can convert to that, though I don't anticipate any MimeType API changes between now and then.
3. All unit tests for core and plugins pass, however, it's probably a good idea to run at least a small crawl with this patch and see if everything works fine. I don't really have the time for this now, so anyone want to try? (cough cough Dougacan cough cough ;) )
4. It's worth noting that this MimeType system from Tika changes the traditional Nutch mime type system (IMO for the better) in a couple of ways. First, whereas the old MimeType system was very happy to return null in places where it couldn't figure out the MimeType, this system tries to return a "default" MimeType (which in this case is "application/octet-stream") if it can't guess the mime type from those that it knows about. Second, this mime type system uses a different type of XML repo file -- based on the one available from freedesktop.org's shared MIME package. 

Okay, so if someone gets a chance please run a small crawl with this in the next few days and let us know how it works. Otherwise, I'll do the same myself in a couple days and if there are no objections, I'd like to commit this then.

> Port mime type framework to use Tika mime detection framework
> -------------------------------------------------------------
>
>                 Key: NUTCH-562
>                 URL: https://issues.apache.org/jira/browse/NUTCH-562
>             Project: Nutch
>          Issue Type: Improvement
>          Components: mime_type_detector
>    Affects Versions: 1.0.0
>         Environment: Mac Book Pro, Intel Core Duo 2.0 Ghz, 2.0 GB RAM, Mac OS X 10.4 although improvement is indep of env
>            Reporter: Chris A. Mattmann
>            Assignee: Chris A. Mattmann
>            Priority: Minor
>         Attachments: NUTCH-562.Mattmann.patch.txt
>
>
> With Tika (http://incubator.apache.org/tika/) nearing  a stable 0.1 release candidate, I think it would be a good time to patch Nutch to use Tika's mime detection system (an improvement over the existing Nutch one written primarily by Jerome). Tika's mime system is based on the mime system from Freedesktop.org and includes several improvements over the existing Nutch mime system such as:
> 1. reliable XML-based content detection (a clear issue plaguing Nutch for some time now), ability to delineate between RSS, XML, ATOM, etc.
> 2. mime magic pattern matching, including support for multiple patterns
> 3. glob pattern matches (ability to support > 1)
> I'll get together a patch and then attach it to the list once it's relatively stable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.