You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tika.apache.org by Jonathan Koren <jo...@soe.ucsc.edu> on 2009/01/24 11:06:00 UTC
detecting mime types
How does tika detect mime types when asked to parse a file? It
routinely fails to detect text/plain files when the file has no
extension; whereas the `file` command (i.e. libmagic) can correctly
determine the same file is "ASCII English Text".
From what I understand from the mailing list archives, Tika was going
to use some XML file from freedesktop.org, but now is using something
from Nutch. Is this true? Where is this XML file? How compatible
is it with libmagic's file?
--
Jonathan Koren
jonathan@soe.ucsc.edu
http://www.soe.ucsc.edu/~jonathan/
Re: failing to detecting mime types from custom mimetype.xml
Posted by Jonathan Koren <jo...@soe.ucsc.edu>.
On Jan 26, 2009, at 2:15 PM, Jukka Zitting wrote:
> On Mon, Jan 26, 2009 at 11:03 PM, Jonathan Koren <jonathan@soe.ucsc.edu
> > wrote:
>> On Jan 26, 2009, at 12:05 PM, Jukka Zitting wrote:
>> I was just about to go through and a bunch of the missing magic
>> from the
>> libmagic files for my own application. So I can just go ahead and
>> mail out
>> mine when it's done.
>
> That would be great! Please attach any related changes to TIKA-192
> where it's easiest to keep track of them.
>>
Diff attached to 192.
--
Jonathan Koren
jonathan@soe.ucsc.edu
http://www.soe.ucsc.edu/~jonathan/
Re: failing to detecting mime types from custom mimetype.xml
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Jan 27, 2009 at 12:39 AM, Jonathan Koren <jo...@soe.ucsc.edu> wrote:
> On Jan 26, 2009, at 2:15 PM, Jukka Zitting wrote:
>> Cool! However, see http://markmail.org/message/rgesbchrufeauxnw for a
>> discussion on how complex a parser implementation within Tika can
>> become until it would be better to look for (or create) an external
>> parser library for that format.
>
> I particularly liked the part where the example given as a good enough
> parser was the very parser I singled out. :)
>
> So the takeaway is "Don't be PDFBox," and "Don't be afraid to add yet
> another dependency, if reimplementing is easy?"
Yeah. If you can do something reasonable with at most a few hundred
lines of code, then it's OK to have it in Tika. But as soon as you go
beyond that, the effort is better spent by contributing to some more
external parser library and using the result in TIka.
BR,
Jukka Zitting
Re: failing to detecting mime types from custom mimetype.xml
Posted by Jonathan Koren <jo...@soe.ucsc.edu>.
On Jan 26, 2009, at 2:15 PM, Jukka Zitting wrote:
> On Mon, Jan 26, 2009 at 11:03 PM, Jonathan Koren <jonathan@soe.ucsc.edu
> > wrote:
>>
>> I also was going to hack some of the parsers to get some better
>> quality
>> metadata from them. For instance, the MP3 parser doesn't handles
>> ID3v2.
>> So if/when I do that, I'll submit a patch.
>
> Cool! However, see http://markmail.org/message/rgesbchrufeauxnw for a
> discussion on how complex a parser implementation within Tika can
> become until it would be better to look for (or create) an external
> parser library for that format.
I particularly liked the part where the example given as a good enough
parser was the very parser I singled out. :)
So the takeaway is "Don't be PDFBox," and "Don't be afraid to add yet
another dependency, if reimplementing is easy?"
I can't imagine that ID3v2 would be the hard to implement, and is v1
even used anymore?
--
Jonathan Koren
jonathan@soe.ucsc.edu
http://www.soe.ucsc.edu/~jonathan/
Re: failing to detecting mime types from custom mimetype.xml
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Mon, Jan 26, 2009 at 11:03 PM, Jonathan Koren <jo...@soe.ucsc.edu> wrote:
> On Jan 26, 2009, at 12:05 PM, Jukka Zitting wrote:
>> Hmm, you're right. As no noted, the proper configuration for GIF is
>> missing. I'll fix that in TIKA-192.
>
> I was just about to go through and a bunch of the missing magic from the
> libmagic files for my own application. So I can just go ahead and mail out
> mine when it's done.
That would be great! Please attach any related changes to TIKA-192
where it's easiest to keep track of them.
> I also was going to hack some of the parsers to get some better quality
> metadata from them. For instance, the MP3 parser doesn't handles ID3v2.
> So if/when I do that, I'll submit a patch.
Cool! However, see http://markmail.org/message/rgesbchrufeauxnw for a
discussion on how complex a parser implementation within Tika can
become until it would be better to look for (or create) an external
parser library for that format.
BR,
Jukka Zitting
Re: failing to detecting mime types from custom mimetype.xml
Posted by Jonathan Koren <jo...@soe.ucsc.edu>.
On Jan 26, 2009, at 12:05 PM, Jukka Zitting wrote:
>> Tika's mime type detection routinely fails on fairly common files.
>> For
>> instance, every gif I've tried Tika returns application/octet-
>> stream rather
>> than image/gif.
>
> Hmm, you're right. As no noted, the proper configuration for GIF is
> missing. I'll fix that in TIKA-192.
I was just about to go through and a bunch of the missing magic from
the libmagic files for my own application. So I can just go ahead and
mail out mine when it's done. (Don't worry, I'm not going to load it
up with things like Xenix core files and Apple ][ NuFiles.)
I also was going to hack some of the parsers to get some better
quality metadata from them. For instance, the MP3 parser doesn't
handles ID3v2. So if/when I do that, I'll submit a patch.
>> but apparently the config file is silently failing to be loaded, or
>> being
>> ignored, or AutoDetectParser's mime detector isn't correctly
>> checking the
>> globs or something, none of which makes any sense. Something
>> should either
>> write to stderr or throw an exception if this was the case.
>
> Hmm, I'll look into that.
I stuck a x at the very beginning of the file when I wrote it out with
xemacs.
Four days for a single one byte error. I suck. :(
>> I've even tried creating a new tika-config.xml with the fullpath to
>> my
>> tika-mimetypes.xml in it
>>
[...]
>> but that just causes NullPointerException s to be thrown.
>
> Where's the NPE coming from? Can you file a bug for that?
Sorry. I tried replicating it right now, and now I can't get it to do
it. Might had been related to having a bogus xml file.
--
Jonathan Koren
jonathan@soe.ucsc.edu
http://www.soe.ucsc.edu/~jonathan/
Re: failing to detecting mime types from custom mimetype.xml
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Mon, Jan 26, 2009 at 6:49 AM, Jonathan Koren <jo...@soe.ucsc.edu> wrote:
> I tried asking this on the users' list, but no one responded, so I guess
> I'll try the dev list.
Sorry for the silence.
> Tika's mime type detection routinely fails on fairly common files. For
> instance, every gif I've tried Tika returns application/octet-stream rather
> than image/gif.
Hmm, you're right. As no noted, the proper configuration for GIF is
missing. I'll fix that in TIKA-192.
> Plain text files without any extension also get marked
> application/octet-stream instead of text/plain.
See TIKA-154, where we just added support for automatically detecting
plain text from nothing but the document input stream.
> I loaded the xml file via:
>
> mimeTypes =
> MimeTypesFactory.create("/fullpath/tika-mimetypes.xml");
> parser = new AutoDetectParser();
> parser.setMimeTypes(mimeTypes);
>
> but apparently the config file is silently failing to be loaded, or being
> ignored, or AutoDetectParser's mime detector isn't correctly checking the
> globs or something, none of which makes any sense. Something should either
> write to stderr or throw an exception if this was the case.
Hmm, I'll look into that.
> MimeTypes doesn't have a way to listing what mime-types are registered, and
> I can't find a publicly accessible way to programatically add a new MimeType
> to the MimeTypes class. (You can add glob patterns, but not actual types.
> `new MimeType()` fails because it's called outside its package.)
Agreed, the MimeTypes class is not as user friendly as it could be.
> I've even tried creating a new tika-config.xml with the fullpath to my
> tika-mimetypes.xml in it
>
> try {
> configFile= new File("/fullpath/tika-config.xml");
> config = new TikaConfig(configFile);
> parser = new AutoDetectParser(config);
> contentHandler = getContentHandler();
> } catch (org.xml.sax.SAXException e) {
> System.err.println("cant parse " + e.getMessage());
> } catch (TikaException e) {
> System.err.println("tika exception " + e.getMessage());
> } catch (IOException e) {
> System.err.println("can't read configfile " + e.getMessage());
> }
> }
>
> but that just causes NullPointerException s to be thrown.
Where's the NPE coming from? Can you file a bug for that?
BR,
Jukka Zitting
failing to detecting mime types from custom mimetype.xml
Posted by Jonathan Koren <jo...@soe.ucsc.edu>.
I tried asking this on the users' list, but no one responded, so I
guess I'll try the dev list.
Tika's mime type detection routinely fails on fairly common files.
For instance, every gif I've tried Tika returns application/octet-
stream rather than image/gif. Plain text files without any extension
also get marked application/octet-stream instead of text/plain.
I copied the tika-mimetypes.xml that's included in tika-0.3-
SNAPSHOT.tar and added a glob for image/gif via:
<mime-type type="image/gif">
<glob pattern="*.gif" />
</mime-type>
(which is odd that this has to be added since the default tika-
config.xml configures a parser for this mime-type.)
I loaded the xml file via:
mimeTypes = MimeTypesFactory.create("/fullpath/tika-
mimetypes.xml");
parser = new AutoDetectParser();
parser.setMimeTypes(mimeTypes);
but apparently the config file is silently failing to be loaded, or
being ignored, or AutoDetectParser's mime detector isn't correctly
checking the globs or something, none of which makes any sense.
Something should either write to stderr or throw an exception if this
was the case.
MimeTypes doesn't have a way to listing what mime-types are
registered, and I can't find a publicly accessible way to
programatically add a new MimeType to the MimeTypes class. (You can
add glob patterns, but not actual types. `new MimeType()` fails
because it's called outside its package.)
There is either a bug here, or there's some trick that is completely
undocumented, or somehow tika-0.3-SNAPSHOT-standalone.jar is
overriding everything.
I've even tried creating a new tika-config.xml with the fullpath to my
tika-mimetypes.xml in it
try {
configFile= new File("/fullpath/tika-config.xml");
config = new TikaConfig(configFile);
parser = new AutoDetectParser(config);
contentHandler = getContentHandler();
} catch (org.xml.sax.SAXException e) {
System.err.println("cant parse " + e.getMessage());
} catch (TikaException e) {
System.err.println("tika exception " + e.getMessage());
} catch (IOException e) {
System.err.println("can't read configfile " +
e.getMessage());
}
}
but that just causes NullPointerException s to be thrown.
This is beyond frustrating.
--
Jonathan Koren
jonathan@soe.ucsc.edu
http://www.soe.ucsc.edu/~jonathan/