You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Erlend Garåsen <e....@usit.uio.no> on 2010/10/08 11:28:44 UTC
Parse MS Office etc. in Nutch 1.2
Hello list,
I'm trying to parse different document formats with Nutch 1.2, but only
an XLSX (Excel) document is parsed in addition to HTML files. This is
how my plugin.includes settings are configured at the moment:
<value>protocol-httpclient|parse-(text|html|pdf|msword|tika)|index-(basic|more)|query-(basic|site|url|lang)</value>
And I have several questions as well:
1. Should I define formats such as pdf and msword when tika is set? Or
is the tika setting sufficient?
2. Do I have to enable all document formats in parse-plugins.xml? A lot
of document formats are disabled. Even though I tried to enable them,
e.g. <mimeType name="application/msword">, it still does not parse MS Word.
3. Why is the XLSX document parsed when I (still) haven't defined
msexcel (or enabled the type in parse-plugins.xml?
Here are some lines from the hadoop.log file regarding MS Word:
2010-10-08 10:29:03,925 INFO fetcher.Fetcher - fetching
http://ridder.uio.no/wtest2.doc
2010-10-08 10:29:04,012 WARN parse.ParseUtil - Unable to successfully
parse content http://ridder.uio.no/wtest2.doc of type
application/x-tika-msoffice
2010-10-08 10:29:04,013 WARN fetcher.Fetcher - Error parsing:
http://ridder.uio.no/wtest2.doc: failed(2,200):
org.apache.nutch.parse.ParseException: Unable to successfully parse content
I need to parse the following document types:
- html
- xml
- MS Word (doc and docx)
- MS Excel (xls and xlsx)
- RTF
- PDF
- ODT
Erlend
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: Parse MS Office etc. in Nutch 1.2
Posted by Markus Jelsma <ma...@openindex.io>.
parseDate needs yyyy-MM-ddTHH:mm:ssZ as well?
On Friday, October 08, 2010 02:40:19 pm Julien Nioche wrote:
> > Sorry, I'm able to parse doc, docx, sxw, odt and rtf as well. After I
> > removed the plugins.folder I changed in order to run Nutch inside
> > Eclipse, everything works.
>
> Good
>
> > BTW, I see the following in my log file:
> > 2010-10-08 13:56:32,555 WARN more.MoreIndexingFilter -
> > http://ridder.uio.no/test1.xlsx: can't parse erroneous date:
> > 2010-10-08T13:55:54Z
> > 2010-10-08 13:56:32,558 WARN more.MoreIndexingFilter -
> > http://ridder.uio.no/wtest1.docx: can't parse erroneous date:
> > 2010-10-08T13:55:49Z
> >
> > Should I report this as an IndexingFilter bug? It seems that I need to
> > rewrite it in order to parse the date correctly, but not a big issue
> > right now.
>
> yes please
>
> Thanks
>
> Julien
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536600 / 06-50258350
Re: Parse MS Office etc. in Nutch 1.2
Posted by Julien Nioche <li...@gmail.com>.
>
> Sorry, I'm able to parse doc, docx, sxw, odt and rtf as well. After I
> removed the plugins.folder I changed in order to run Nutch inside Eclipse,
> everything works.
>
Good
>
> BTW, I see the following in my log file:
> 2010-10-08 13:56:32,555 WARN more.MoreIndexingFilter -
> http://ridder.uio.no/test1.xlsx: can't parse erroneous date:
> 2010-10-08T13:55:54Z
> 2010-10-08 13:56:32,558 WARN more.MoreIndexingFilter -
> http://ridder.uio.no/wtest1.docx: can't parse erroneous date:
> 2010-10-08T13:55:49Z
>
> Should I report this as an IndexingFilter bug? It seems that I need to
> rewrite it in order to parse the date correctly, but not a big issue right
> now.
>
>
yes please
Thanks
Julien
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
Re: Parse MS Office etc. in Nutch 1.2
Posted by Erlend Garåsen <e....@usit.uio.no>.
Erlend Garåsen wrote:
> After I changed the nutch-default.xml file back to its original form,
> I'm able to parse html, pdf, xls and xlsz (by using the command line
> instead of running Nutch inside Eclipse). But do you know how to parse
> MS Office, RTF and ODT?
Sorry, I'm able to parse doc, docx, sxw, odt and rtf as well. After I
removed the plugins.folder I changed in order to run Nutch inside
Eclipse, everything works.
BTW, I see the following in my log file:
2010-10-08 13:56:32,555 WARN more.MoreIndexingFilter -
http://ridder.uio.no/test1.xlsx: can't parse erroneous date:
2010-10-08T13:55:54Z
2010-10-08 13:56:32,558 WARN more.MoreIndexingFilter -
http://ridder.uio.no/wtest1.docx: can't parse erroneous date:
2010-10-08T13:55:49Z
Should I report this as an IndexingFilter bug? It seems that I need to
rewrite it in order to parse the date correctly, but not a big issue
right now.
Erlend
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: Parse MS Office etc. in Nutch 1.2
Posted by Erlend Garåsen <e....@usit.uio.no>.
Julien Nioche wrote:
>> And I have several questions as well:
>> 1. Should I define formats such as pdf and msword when tika is set? Or is
>> the tika setting sufficient?
>>
>
> Tika handles the pdf and msword formats so you should not need to specify
> them in plugin.includes
OK, thanks for clarifying.
>> 2. Do I have to enable all document formats in parse-plugins.xml? A lot of
>> document formats are disabled. Even though I tried to enable them, e.g.
>> <mimeType name="application/msword">, it still does not parse MS Word.
>
>
> Since you've added parse-tika to plugin.includes, it will be used by default
> on all mime-types which is why we don't have an explicit association for all
> mimeTypes in parse-plugins.xml
OK, then I will let the parse-plugin.xml file remain untouched.
>> Here are some lines from the hadoop.log file regarding MS Word:
>>
>> 2010-10-08 10:29:03,925 INFO fetcher.Fetcher - fetching
>> http://ridder.uio.no/wtest2.doc
>> 2010-10-08 10:29:04,012 WARN parse.ParseUtil - Unable to successfully
>> parse content http://ridder.uio.no/wtest2.doc of type
>> application/x-tika-msoffice
>> 2010-10-08 10:29:04,013 WARN fetcher.Fetcher - Error parsing:
>> http://ridder.uio.no/wtest2.doc: failed(2,200):
>> org.apache.nutch.parse.ParseException: Unable to successfully parse content
>>
>
> Strange, I can't reproduce the issue with nutch-1.2. Can you try running the
> command below?
>
> *bin/nutch org.apache.nutch.parse.ParserChecker
> http://ridder.uio.no/wtest2.doc
I guess I have found the source of some of my problems. I followed the
Eclipse tutorial which is probably outdated:
http://wiki.apache.org/nutch/RunNutchInEclipse1.0
Probably because I did the following:
"Change the property "plugin.folders" to "./src/plugin" on
$NUTCH_HOME/conf/nutch-default.xml"
After I changed the nutch-default.xml file back to its original form,
I'm able to parse html, pdf, xls and xlsz (by using the command line
instead of running Nutch inside Eclipse). But do you know how to parse
MS Office, RTF and ODT?
BTW, here's the output:
hoppalong:apache-nutch-1.2 erlendfg$ bin/nutch
org.apache.nutch.parse.ParserChecker http://ridder.uio.no/wtest2.doc
---------
Url
---------------
http://ridder.uio.no/wtest2.doc---------
ParseData
---------
Version: 5
Status: success(1,0)
Title:
Outlinks: 0
Content Metadata: ETag="20064f-5600-47d0d1f0f8400" Date=Fri, 08 Oct 2010
11:26:38 GMT Content-Length=22016 Last-Modified=Wed, 13 Jan 2010
15:06:56 GMT Content-Type=application/msword Connection=close
Accept-Ranges=bytes Server=Apache/2.2.16 (Unix) mod_ssl/2.2.16
OpenSSL/0.9.8j DAV/2 PHP/5.2.14 mod_perl/2.0.4 Perl/v5.8.8
Parse Metadata: Revision-Number=2 Last-Author=Erlend Gar?sen
Template=Normal.dotm subject= Page-Count=1 Application-Name=Microsoft
Macintosh Word Author=Erlend Gar?sen Edit-Time=600000000
Creation-Date=Wed Jan 13 15:58:00 CET 2010 Company=Universitetet i Oslo
Content-Type=application/msword Keywords= Last-Save-Date=Wed Jan 13
15:58:00 CET 2010
Erlend
--
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
Re: Parse MS Office etc. in Nutch 1.2
Posted by Julien Nioche <li...@gmail.com>.
Hi
> I'm trying to parse different document formats with Nutch 1.2, but only an
> XLSX (Excel) document is parsed in addition to HTML files. This is how my
> plugin.includes settings are configured at the moment:
>
>
> <value>protocol-httpclient|parse-(text|html|pdf|msword|tika)|index-(basic|more)|query-(basic|site|url|lang)</value>
>
> And I have several questions as well:
> 1. Should I define formats such as pdf and msword when tika is set? Or is
> the tika setting sufficient?
>
Tika handles the pdf and msword formats so you should not need to specify
them in plugin.includes
> 2. Do I have to enable all document formats in parse-plugins.xml? A lot of
> document formats are disabled. Even though I tried to enable them, e.g.
> <mimeType name="application/msword">, it still does not parse MS Word.
Since you've added parse-tika to plugin.includes, it will be used by default
on all mime-types which is why we don't have an explicit association for all
mimeTypes in parse-plugins.xml
> 3. Why is the XLSX document parsed when I (still) haven't defined msexcel
> (or enabled the type in parse-plugins.xml?
>
See explanation for point 2 above
>
> Here are some lines from the hadoop.log file regarding MS Word:
>
> 2010-10-08 10:29:03,925 INFO fetcher.Fetcher - fetching
> http://ridder.uio.no/wtest2.doc
> 2010-10-08 10:29:04,012 WARN parse.ParseUtil - Unable to successfully
> parse content http://ridder.uio.no/wtest2.doc of type
> application/x-tika-msoffice
> 2010-10-08 10:29:04,013 WARN fetcher.Fetcher - Error parsing:
> http://ridder.uio.no/wtest2.doc: failed(2,200):
> org.apache.nutch.parse.ParseException: Unable to successfully parse content
>
Strange, I can't reproduce the issue with nutch-1.2. Can you try running the
command below?
*bin/nutch org.apache.nutch.parse.ParserChecker
http://ridder.uio.no/wtest2.doc
---------
Url
---------------
http://ridder.uio.no/wtest2.doc---------
ParseData
---------
Version: 5
Status: success(1,0)
Title:
Outlinks: 0
Content Metadata: ETag="20064f-5600-47d0d1f0f8400" Date=Fri, 08 Oct 2010
10:19:19 GMT Content-Length=22016 Last-Modified=Wed, 13 Jan 2010 15:06:56
GMT Content-Type=application/msword Connection=close Accept-Ranges=bytes
Server=Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8j DAV/2 PHP/5.2.14
mod_perl/2.0.4 Perl/v5.8.8
Parse Metadata: Revision-Number=2 Last-Author=Erlend Garåsen
Template=Normal.dotm subject= Page-Count=1 Application-Name=Microsoft
Macintosh Word Author=Erlend Garåsen Edit-Time=600000000 Creation-Date=Wed
Jan 13 14:58:00 GMT 2010 Company=Universitetet i Oslo
Content-Type=application/msword Keywords= Last-Save-Date=Wed Jan 13 14:58:00
GMT 2010 *
HTH
Julien
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
Re: Parse MS Office etc. in Nutch 1.2
Posted by Markus Jelsma <ma...@openindex.io>.
You can patch your version now.
https://issues.apache.org/jira/browse/NUTCH-912
On Friday 08 October 2010 11:28:44 Erlend Garåsen wrote:
> Hello list,
>
> I'm trying to parse different document formats with Nutch 1.2, but only
> an XLSX (Excel) document is parsed in addition to HTML files. This is
> how my plugin.includes settings are configured at the moment:
>
> <value>protocol-httpclient|parse-(text|html|pdf|msword|tika)|index-(basic|m
> ore)|query-(basic|site|url|lang)</value>
>
> And I have several questions as well:
> 1. Should I define formats such as pdf and msword when tika is set? Or
> is the tika setting sufficient?
>
> 2. Do I have to enable all document formats in parse-plugins.xml? A lot
> of document formats are disabled. Even though I tried to enable them,
> e.g. <mimeType name="application/msword">, it still does not parse MS Word.
>
> 3. Why is the XLSX document parsed when I (still) haven't defined
> msexcel (or enabled the type in parse-plugins.xml?
>
> Here are some lines from the hadoop.log file regarding MS Word:
>
> 2010-10-08 10:29:03,925 INFO fetcher.Fetcher - fetching
> http://ridder.uio.no/wtest2.doc
> 2010-10-08 10:29:04,012 WARN parse.ParseUtil - Unable to successfully
> parse content http://ridder.uio.no/wtest2.doc of type
> application/x-tika-msoffice
> 2010-10-08 10:29:04,013 WARN fetcher.Fetcher - Error parsing:
> http://ridder.uio.no/wtest2.doc: failed(2,200):
> org.apache.nutch.parse.ParseException: Unable to successfully parse content
>
> I need to parse the following document types:
> - html
> - xml
> - MS Word (doc and docx)
> - MS Excel (xls and xlsx)
> - RTF
> - PDF
> - ODT
>
> Erlend
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536600 / 06-50258350