You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Richard Braman <rb...@bramantax.com> on 2006/02/28 13:43:00 UTC
Nutch Parsing PDFs, and general PDF extraction
I see that there is a class for parsing pdfs in nutch using pdfbox
<http://lucene.apache.org/nutch/apidocs/org/apache/nutch/parse/pdf/packa
ge-summary.html> org.apache.nutch.parse.pdf (Nutch 0.7.1 API)
but I dont see it in the source of 0.7.1 downloaded
I see it on cvs here:
http://cvs.sourceforge.net/viewcvs.py/nutch/nutch/src/plugin/parse-pdf/s
rc/java/net/nutch/parse/pdf/
but my nutch doesn't seem to run the pdf parse class as my log file
shows it fecthing pdfs, but saying nutch is unable to parse content type
application/pdf
Why is this? Was it left out because of performace?
IMO, The class used by nutch (shown below) wont cut it for most pdfs
though, as the pdf structure is usually too complicated. Please see
some of resources I cited in my last posts including
http://www.tamirhassan.dsl.pipex.com/final.pdf
and
http://www.chilisoftware.net/Private/Christian/ideas_for_extracting_data
_from_unstructured_documents.pdf.
as they bring up some good algorithms for parsing pdf. 90% of PDFs are
unstrcutured, they dont contain any XML content that describes how the
pages flow. The content could be in any order, and that might make
searching for literals throw innacurate results.
the other 10% of PDFs use tagging, and nutch could use this to parse
through the tagged ones quite easily using PDFBox.
We need to have nutch/lucense parsing pdfs, it is one of the features of
google that users value, and there is simplying to much pdf content to
ignore. IN addition it would be nice to have nutch be able to show the
pdf as html like google does. I think Tamirs paper is a good read on
this because he does a good analysis of the googles functionality here
and his original objective was to format the PDF as an html, whihc
requires correctly parsing the pdf.
Some more references
http://snowtide.com/home/PDFTextStream/techtips/easy_lucene_integration
http://www.jguru.com/faq/view.jsp?EID=862443
/* Copyright (c) 2004 The Nutch Organization. All rights reserved. */
/* Use subject to the conditions in http://www.nutch.org/LICENSE.txt. */
package net.nutch.parse.pdf;
import org.pdfbox.encryption.DocumentEncryption;
import org.pdfbox.pdfparser.PDFParser;
import org.pdfbox.pdmodel.PDDocument;
import org.pdfbox.pdmodel.PDDocumentInformation;
import org.pdfbox.util.PDFTextStripper;
import org.pdfbox.exceptions.CryptographyException;
import org.pdfbox.exceptions.InvalidPasswordException;
import net.nutch.protocol.Content;
import net.nutch.util.LogFormatter;
import net.nutch.parse.Parser;
import net.nutch.parse.Parse;
import net.nutch.parse.ParseData;
import net.nutch.parse.ParseImpl;
import net.nutch.parse.Outlink;
import net.nutch.parse.ParseException;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Properties;
import java.util.logging.Logger;
import java.io.ByteArrayInputStream;
import java.io.IOException;
/*********************************************
* parser for mime type application/pdf.
* It is based on org.pdfbox.*. We have to see how well it does the job.
*
* @author John Xing
*
* Note on 20040614 by Xing:
* Some codes are stacked here for convenience (see inline comments).
* They may be moved to more appropriate places when new codebase
* stabilizes, especially after code for indexing is written.
*
*********************************************/
public class PdfParser implements Parser {
public static final Logger LOG =
LogFormatter.getLogger("net.nutch.parse.pdf");
public PdfParser () {
// redirect org.apache.log4j.Logger to java's native logger, in order
// to, at least, suppress annoying log4j warnings.
// Note on 20040614 by Xing:
// log4j is used by pdfbox. This snippet'd better be moved
// to a common place shared by all parsers that use log4j.
org.apache.log4j.Logger rootLogger =
org.apache.log4j.Logger.getRootLogger();
rootLogger.setLevel(org.apache.log4j.Level.INFO);
org.apache.log4j.Appender appender = new
org.apache.log4j.WriterAppender(
new org.apache.log4j.SimpleLayout(),
net.nutch.util.LogFormatter.getLogStream(
this.LOG, java.util.logging.Level.INFO));
rootLogger.addAppender(appender);
}
public Parse getParse(Content content) throws ParseException {
// check that contentType is one we can handle
String contentType = content.getContentType();
if (contentType != null && !contentType.startsWith("application/pdf"))
throw new ParseException(
"Content-Type not application/pdf: "+contentType);
// in memory representation of pdf file
PDDocument pdf = null;
String text = null;
String title = null;
try {
byte[] raw = content.getContent();
String contentLength = content.get("Content-Length");
if (contentLength != null
&& raw.length != Integer.parseInt(contentLength)) {
throw new ParseException("Content truncated at "+raw.length
+" bytes. Parser can't handle incomplete pdf file.");
}
PDFParser parser = new PDFParser(
new ByteArrayInputStream(raw));
parser.parse();
pdf = parser.getPDDocument();
if (pdf.isEncrypted()) {
DocumentEncryption decryptor = new DocumentEncryption(pdf);
//Just try using the default password and move on
decryptor.decryptDocument("");
}
// collect text
PDFTextStripper stripper = new PDFTextStripper();
text = stripper.getText(pdf);
// collect title
PDDocumentInformation info = pdf.getDocumentInformation();
title = info.getTitle();
// more useful info, currently not used. please keep them for future
use.
// pdf.getPageCount();
// info.getAuthor()
// info.getSubject()
// info.getKeywords()
// info.getCreator()
// info.getProducer()
// info.getTrapped()
// formatDate(info.getCreationDate())
// formatDate(info.getModificationDate())
} catch (ParseException e) {
throw e;
} catch (CryptographyException e) {
throw new ParseException("Error decrypting document. "+e);
} catch (InvalidPasswordException e) {
throw new ParseException("Can't decrypt document. "+e);
} catch (Exception e) { // run time exception
throw new ParseException("Can't be handled as pdf document. "+e);
} finally {
try {
if (pdf != null)
pdf.close();
} catch (IOException e) {
// nothing to do
}
}
if (text == null)
text = "";
if (title == null)
title = "";
// collect outlink
Outlink[] outlinks = new Outlink[0];
// collect meta data
Properties metadata = new Properties();
metadata.putAll(content.getMetadata()); // copy through
ParseData parseData = new ParseData(title, outlinks, metadata);
return new ParseImpl(text, parseData);
// any filter?
//return HtmlParseFilters.filter(content, parse, root);
}
// format date
// currently not used. please keep it for future use.
private String formatDate(Calendar date) {
String retval = null;
if(date != null) {
SimpleDateFormat formatter = new SimpleDateFormat();
retval = formatter.format(date.getTime());
}
return retval;
}
}
Richard Braman
mailto:rbraman@taxcodesoftware.org
561.748.4002 (voice)
http://www.taxcodesoftware.org <http://www.taxcodesoftware.org/>
Free Open Source Tax Software
RE: Nutch Parsing PDFs, and general PDF extraction
Posted by Richard Braman <rb...@bramantax.com>.
I dont know it seems to be working now.
-----Original Message-----
From: Jérôme Charron [mailto:jerome.charron@gmail.com]
Sent: Tuesday, February 28, 2006 8:46 AM
To: nutch-dev@lucene.apache.org; rbraman@bramantax.com
Subject: Re: Nutch Parsing PDFs, and general PDF extraction
> Putting the wellformed version of the plugin code you provided
> generated the follwong exception:
Does the nutch-extensionpoints plugin is activated?
Re: Nutch Parsing PDFs, and general PDF extraction
Posted by Jérôme Charron <je...@gmail.com>.
> Putting the wellformed version of the plugin code you provided generated
> the follwong exception:
Does the nutch-extensionpoints plugin is activated?
RE: Nutch Parsing PDFs, and general PDF extraction
Posted by Richard Braman <rb...@bramantax.com>.
Putting the wellformed version of the plugin code you provided generated
the follwong exception:
060228 083159 SEVERE org.apache.nutch.plugin.PluginRuntimeException:
extension point: org.apache.nutch.parse.Parser does not exist.
Exception in thread "main" java.lang.ExceptionInInitializerError
at
org.apache.nutch.db.WebDBInjector.addPage(WebDBInjector.java:437)
at
org.apache.nutch.db.WebDBInjector.injectURLFile(WebDBInjector.java:378)
at
org.apache.nutch.db.WebDBInjector.main(WebDBInjector.java:535)
at org.apache.nutch.tools.CrawlTool.main(CrawlTool.java:134)
Caused by: java.lang.RuntimeException:
org.apache.nutch.plugin.PluginRuntimeException: extension point:
org.apache.nutch.parse.Parser does not exist.
at
org.apache.nutch.plugin.PluginRepository.getInstance(PluginRepository.ja
va:147)
at org.apache.nutch.net.URLFilters.<clinit>(URLFilters.java:40)
... 4 more
Caused by: org.apache.nutch.plugin.PluginRuntimeException: extension
point: org.apache.nutch.parse.Parser does not exist.
at
org.apache.nutch.plugin.PluginRepository.installExtensions(PluginReposit
ory.java:78)
at
org.apache.nutch.plugin.PluginRepository.<init>(PluginRepository.java:61
)
at
org.apache.nutch.plugin.PluginRepository.getInstance(PluginRepository.ja
va:144)
-----Original Message-----
From: YourSoft [mailto:yoursoft@freemail.hu]
Sent: Tuesday, February 28, 2006 8:20 AM
To: nutch-dev@lucene.apache.org
Subject: Re: Nutch Parsing PDFs, and general PDF extraction
Richard Braman wrotte:
No, you should be add to "plugin include" (in nutch-site.xml) e.g.:
<property>
<name>plugin.includes</name>
<value>protocol-http|urlfilter-(regex|prefix)|parse-(text|html|pdf)<desc
ription>Regular
expression naming plugin directory names to
include. Any plugin not matching this expression is excluded. By
default Nutch includes crawling just HTML and plain text via HTTP,
and basic indexing and search plugins.
</description>
</property>
Re: Nutch Parsing PDFs, and general PDF extraction
Posted by YourSoft <yo...@freemail.hu>.
Richard Braman wrotte:
No, you should be add to "plugin include" (in nutch-site.xml) e.g.:
<property>
<name>plugin.includes</name>
<value>protocol-http|urlfilter-(regex|prefix)|parse-(text|html|pdf)<description>Regular
expression naming plugin directory names to
include. Any plugin not matching this expression is excluded. By
default Nutch includes crawling just HTML and plain text via HTTP,
and basic indexing and search plugins.
</description>
</property>
RE: Nutch Parsing PDFs, and general PDF extraction
Posted by Richard Braman <rb...@bramantax.com>.
Should I add this to nutch site:
<plugin
id="parse-pdf"
name="Pdf Parse Plug-in"
version="1.0.0"
provider-name="nutch.org">
<runtime>
<library name="parse-pdf.jar">
<export name="*"/>
</library>
<library name="PDFBox-0.7.2-log4j.jar"/>
<library name="log4j-1.2.9.jar"/>
</runtime>
<extension id="org.apache.nutch.parse.pdf"
name="PdfParse"
point="org.apache.nutch.parse.Parser">
<implementation id="org.apache.nutch.parse.pdf.PdfParser"
class="org.apache.nutch.parse.pdf.PdfParser"
contentType="application/pdf"
pathSuffix=""/>
</extension>
</plugin>
-----Original Message-----
From: Richard Braman [mailto:rbraman@bramantax.com]
Sent: Tuesday, February 28, 2006 7:58 AM
To: nutch-dev@lucene.apache.org; rbraman@bramantax.com
Subject: RE: Nutch Parsing PDFs, and general PDF extraction
I don't have the plugin configured, whats the code for doing that?
-----Original Message-----
From: Richard Braman [mailto:rbraman@bramantax.com]
Sent: Tuesday, February 28, 2006 7:52 AM
To: nutch-dev@lucene.apache.org
Subject: RE: Nutch Parsing PDFs, and general PDF extraction
060228 045534 fetch okay, but can't parse
http://www.irs.gov/pub/irs-pdf/f1040sab.pdf?portlet=3, reason:
failed(2,203): Content-Type not text/html: application/pdf
-----Original Message-----
From: YourSoft [mailto:yoursoft@freemail.hu]
Sent: Tuesday, February 28, 2006 8:00 AM
To: nutch-dev@lucene.apache.org
Subject: Re: Nutch Parsing PDFs, and general PDF extraction
Richard Braman wrotte:
>but my nutch doesn't seem to run the pdf parse class as my log file
>shows it fecthing pdfs, but saying nutch is unable to parse content
>type application/pdf
>
>
Can you send the complette error message?
RE: Nutch Parsing PDFs, and general PDF extraction
Posted by Richard Braman <rb...@bramantax.com>.
I don't have the plugin configured, whats the code for doing that?
-----Original Message-----
From: Richard Braman [mailto:rbraman@bramantax.com]
Sent: Tuesday, February 28, 2006 7:52 AM
To: nutch-dev@lucene.apache.org
Subject: RE: Nutch Parsing PDFs, and general PDF extraction
060228 045534 fetch okay, but can't parse
http://www.irs.gov/pub/irs-pdf/f1040sab.pdf?portlet=3, reason:
failed(2,203): Content-Type not text/html: application/pdf
-----Original Message-----
From: YourSoft [mailto:yoursoft@freemail.hu]
Sent: Tuesday, February 28, 2006 8:00 AM
To: nutch-dev@lucene.apache.org
Subject: Re: Nutch Parsing PDFs, and general PDF extraction
Richard Braman wrotte:
>but my nutch doesn't seem to run the pdf parse class as my log file
>shows it fecthing pdfs, but saying nutch is unable to parse content
>type application/pdf
>
>
Can you send the complette error message?
RE: Nutch Parsing PDFs, and general PDF extraction
Posted by Richard Braman <rb...@bramantax.com>.
060228 045534 fetch okay, but can't parse
http://www.irs.gov/pub/irs-pdf/f1040sab.pdf?portlet=3, reason:
failed(2,203): Content-Type not text/html: application/pdf
-----Original Message-----
From: YourSoft [mailto:yoursoft@freemail.hu]
Sent: Tuesday, February 28, 2006 8:00 AM
To: nutch-dev@lucene.apache.org
Subject: Re: Nutch Parsing PDFs, and general PDF extraction
Richard Braman wrotte:
>but my nutch doesn't seem to run the pdf parse class as my log file
>shows it fecthing pdfs, but saying nutch is unable to parse content
>type application/pdf
>
>
Can you send the complette error message?
Re: Nutch Parsing PDFs, and general PDF extraction
Posted by YourSoft <yo...@freemail.hu>.
Richard Braman wrotte:
>but my nutch doesn't seem to run the pdf parse class as my log file
>shows it fecthing pdfs, but saying nutch is unable to parse content type
>application/pdf
>
>
Can you send the complette error message?
Re: Nutch Parsing PDFs, and general PDF extraction
Posted by Jérôme Charron <je...@gmail.com>.
> <http://lucene.apache.org/nutch/apidocs/org/apache/nutch/parse/pdf/packa
> ge-summary.html> org.apache.nutch.parse.pdf (Nutch 0.7.1 API)
> but I dont see it in the source of 0.7.1 downloaded
>
> I see it on cvs here:
> http://cvs.sourceforge.net/viewcvs.py/nutch/nutch/src/plugin/parse-pdf/s
> rc/java/net/nutch/parse/pdf/
First of all, the nutch source code is no more hosted on sourceforge, but on
apache:
http://svn.apache.org/viewcvs.cgi/lucene/nutch/
The classes packages has also been changed to org.apache.nutch
but my nutch doesn't seem to run the pdf parse class as my log file
> shows it fecthing pdfs, but saying nutch is unable to parse content type
> application/pdf
> Why is this? Was it left out because of performace?
Do you have activated the parse-pdf plugin in conf/nutch-default.xml or
conf/nutch-site.xml ?
Regards
Jérôme
--
http://motrech.free.fr/
http://www.frutch.org/