You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Andreas Kemkes <a5...@yahoo.com> on 2011/02/25 21:10:23 UTC
Tika metadata extracted per supported document format?
Hello,
I've asked this on the Tika mailing list w/o an answer, so apologies for
cross-posting.
I'm trying to find information that tells me specifically what metadata is
provided for the different supported document formats. Unfortunately all I was
able to find so far is "The Metadata produced depends on the type of document
submitted."
Currently, I'm using ExtractingRequestHandler from Solr 1.4 (with Tika 0.4), so
I'm particularly interested in that version, but also in changes that are
provided in newer versions of Tika.
Where are the best places to look for such information?
Thanks in advance,
Andreas
Re: Tika metadata extracted per supported document format?
Posted by Andreas Kemkes <a5...@yahoo.com>.
Chris:
Yes, I only see the output below.
I'm familiar with the information in
http://wiki.apache.org/solr/ExtractingRequestHandler, except for
the tika.config part, which I haven't touched.
Even when running documents through Tika directly, the output of metadata is
highly dependent on what metadata the document contains (obviously). I haven't
found the right place in the Tika source code yet either. Would digging into
POI, PDFBox, ... help me any further on my pursuit? A Matrix that lists the
complete set of metadata for the most popular formats would sure be helpful to
me. I would help providing it, if properly directed.
Thanks,
Andreas
PS: I've also noticed some differences in the date formats being used (using
version 0.9). Is that something I should be concerned about when using it
through SolrCell?
<meta name="Creation-Date" content="Mon May 17 10:10:15 PDT 2010"/> (from a Word
document)
<meta name="Creation-Date" content="2011-01-03T18:45:50Z"/> (from a PDF)
________________________________
From: "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Sent: Fri, February 25, 2011 4:11:00 PM
Subject: Re: Tika metadata extracted per supported document format?
Hi Andreas,
> java -jar tika-app-0.9.jar --list-met-models
> TikaMetadataKeys
> PROTECTED
> RESOURCE_NAME_KEY
> TikaMimeKeys
> MIME_TYPE_MAGIC
> TIKA_MIME_FILE
>
> Both 0.8 and 0.9 give me the same list. Is that a configuration issue?
Strange -- those are the only met models you're seeing listed?
>
> I'm a bit unclear if that gets me to what I was looking for - metadata
> like "content_type" or "last_modified". Or am I confusing Tika metadata
> with SolrCell metadata?
>
> I thought SolrCell metadata comes from Tika, or does it not?
It does come from Tika that's for sure, but in SolrCell, there is a
configuration for the ExtractingRequestHandler that remaps
the field names from Tika to Solr. So that's probably where it's coming from.
Check this out:
http://wiki.apache.org/solr/ExtractingRequestHandler
HTH!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Tika metadata extracted per supported document format?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Andreas,
> java -jar tika-app-0.9.jar --list-met-models
> TikaMetadataKeys
> PROTECTED
> RESOURCE_NAME_KEY
> TikaMimeKeys
> MIME_TYPE_MAGIC
> TIKA_MIME_FILE
>
> Both 0.8 and 0.9 give me the same list. Is that a configuration issue?
Strange -- those are the only met models you're seeing listed?
>
> I'm a bit unclear if that gets me to what I was looking for - metadata
> like "content_type" or "last_modified". Or am I confusing Tika metadata
> with SolrCell metadata?
>
> I thought SolrCell metadata comes from Tika, or does it not?
It does come from Tika that's for sure, but in SolrCell, there is a configuration for the ExtractingRequestHandler that remaps
the field names from Tika to Solr. So that's probably where it's coming from. Check this out:
http://wiki.apache.org/solr/ExtractingRequestHandler
HTH!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Tika metadata extracted per supported document format?
Posted by Andreas Kemkes <a5...@yahoo.com>.
Hi Chris,
java -jar tika-app-0.9.jar --list-met-models
TikaMetadataKeys
PROTECTED
RESOURCE_NAME_KEY
TikaMimeKeys
MIME_TYPE_MAGIC
TIKA_MIME_FILE
Both 0.8 and 0.9 give me the same list. Is that a configuration issue?
I'm a bit unclear if that gets me to what I was looking for - metadata
like "content_type" or "last_modified". Or am I confusing Tika metadata
with SolrCell metadata?
I thought SolrCell metadata comes from Tika, or does it not?
Regards,
Andreas
________________________________
From: "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Cc: "user@tika.apache.org" <us...@tika.apache.org>
Sent: Fri, February 25, 2011 1:21:33 PM
Subject: Re: Tika metadata extracted per supported document format?
Hi Andreas,
In Tika 0.8+, you can run the --list-met-models command from tika-app:
java -jar tika-app-<version>.jar --list-met-models
And get a print out of the met keys that Tika supports. Some parsers add their
own that aren't part of this met listing, but this is a relatively comprehensive
list.
Cheers,
Chris
On Feb 25, 2011, at 12:10 PM, Andreas Kemkes wrote:
> Hello,
>
> I've asked this on the Tika mailing list w/o an answer, so apologies for
> cross-posting.
>
> I'm trying to find information that tells me specifically what metadata is
> provided for the different supported document formats. Unfortunately all I was
>
> able to find so far is "The Metadata produced depends on the type of document
> submitted."
>
> Currently, I'm using ExtractingRequestHandler from Solr 1.4 (with Tika 0.4), so
>
> I'm particularly interested in that version, but also in changes that are
> provided in newer versions of Tika.
>
> Where are the best places to look for such information?
>
> Thanks in advance,
>
> Andreas
>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Tika metadata extracted per supported document format?
Posted by Andreas Kemkes <a5...@yahoo.com>.
Hi Chris,
Thank you so much - that's a great start.
Andreas
________________________________
From: "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Cc: "user@tika.apache.org" <us...@tika.apache.org>
Sent: Fri, February 25, 2011 1:21:33 PM
Subject: Re: Tika metadata extracted per supported document format?
Hi Andreas,
In Tika 0.8+, you can run the --list-met-models command from tika-app:
java -jar tika-app-<version>.jar --list-met-models
And get a print out of the met keys that Tika supports. Some parsers add their
own that aren't part of this met listing, but this is a relatively comprehensive
list.
Cheers,
Chris
On Feb 25, 2011, at 12:10 PM, Andreas Kemkes wrote:
> Hello,
>
> I've asked this on the Tika mailing list w/o an answer, so apologies for
> cross-posting.
>
> I'm trying to find information that tells me specifically what metadata is
> provided for the different supported document formats. Unfortunately all I was
>
> able to find so far is "The Metadata produced depends on the type of document
> submitted."
>
> Currently, I'm using ExtractingRequestHandler from Solr 1.4 (with Tika 0.4), so
>
> I'm particularly interested in that version, but also in changes that are
> provided in newer versions of Tika.
>
> Where are the best places to look for such information?
>
> Thanks in advance,
>
> Andreas
>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Tika metadata extracted per supported document format?
Posted by Andreas Kemkes <a5...@yahoo.com>.
Hi Chris,
Thank you so much - that's a great start.
Andreas
________________________________
From: "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Cc: "user@tika.apache.org" <us...@tika.apache.org>
Sent: Fri, February 25, 2011 1:21:33 PM
Subject: Re: Tika metadata extracted per supported document format?
Hi Andreas,
In Tika 0.8+, you can run the --list-met-models command from tika-app:
java -jar tika-app-<version>.jar --list-met-models
And get a print out of the met keys that Tika supports. Some parsers add their
own that aren't part of this met listing, but this is a relatively comprehensive
list.
Cheers,
Chris
On Feb 25, 2011, at 12:10 PM, Andreas Kemkes wrote:
> Hello,
>
> I've asked this on the Tika mailing list w/o an answer, so apologies for
> cross-posting.
>
> I'm trying to find information that tells me specifically what metadata is
> provided for the different supported document formats. Unfortunately all I was
>
> able to find so far is "The Metadata produced depends on the type of document
> submitted."
>
> Currently, I'm using ExtractingRequestHandler from Solr 1.4 (with Tika 0.4), so
>
> I'm particularly interested in that version, but also in changes that are
> provided in newer versions of Tika.
>
> Where are the best places to look for such information?
>
> Thanks in advance,
>
> Andreas
>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Tika metadata extracted per supported document format?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Andreas,
In Tika 0.8+, you can run the --list-met-models command from tika-app:
java -jar tika-app-<version>.jar --list-met-models
And get a print out of the met keys that Tika supports. Some parsers add their own that aren't part of this met listing, but this is a relatively comprehensive list.
Cheers,
Chris
On Feb 25, 2011, at 12:10 PM, Andreas Kemkes wrote:
> Hello,
>
> I've asked this on the Tika mailing list w/o an answer, so apologies for
> cross-posting.
>
> I'm trying to find information that tells me specifically what metadata is
> provided for the different supported document formats. Unfortunately all I was
> able to find so far is "The Metadata produced depends on the type of document
> submitted."
>
> Currently, I'm using ExtractingRequestHandler from Solr 1.4 (with Tika 0.4), so
> I'm particularly interested in that version, but also in changes that are
> provided in newer versions of Tika.
>
> Where are the best places to look for such information?
>
> Thanks in advance,
>
> Andreas
>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Tika metadata extracted per supported document format?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Andreas,
In Tika 0.8+, you can run the --list-met-models command from tika-app:
java -jar tika-app-<version>.jar --list-met-models
And get a print out of the met keys that Tika supports. Some parsers add their own that aren't part of this met listing, but this is a relatively comprehensive list.
Cheers,
Chris
On Feb 25, 2011, at 12:10 PM, Andreas Kemkes wrote:
> Hello,
>
> I've asked this on the Tika mailing list w/o an answer, so apologies for
> cross-posting.
>
> I'm trying to find information that tells me specifically what metadata is
> provided for the different supported document formats. Unfortunately all I was
> able to find so far is "The Metadata produced depends on the type of document
> submitted."
>
> Currently, I'm using ExtractingRequestHandler from Solr 1.4 (with Tika 0.4), so
> I'm particularly interested in that version, but also in changes that are
> provided in newer versions of Tika.
>
> Where are the best places to look for such information?
>
> Thanks in advance,
>
> Andreas
>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++