You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "Håvard W. Kongsgård" <h....@niap.no> on 2006/01/25 16:10:24 UTC

Parsing PDF Nutch Achilles heel?

I have been doing some testing on different nutch configurations to see 
what slows down the fetching process on my servers(nutch 0.7.1).
My general experience is that the PDF parse process is nutchs Achilles heel.

Nutch works fine on older computers, but with the combination of 
|parse-(text|html|pdf)
and http.content.limit = -1(needed to get PDF parsing to work) nutch 
sometimes freezes completely.

Is there planned any improvement to the parsing of PDF files in the next 
version of nutch (0.8)?

Re: Parsing PDF Nutch Achilles heel?

Posted by "Håvard W. Kongsgård" <h....@niap.no>.
I tried it with PDFBox-0.7.3-dev-20060125-log4j(renamed to 
PDFBox-0.7.2-log4j) it worked on some PDFs(50 %)
on the rest "failed with: java.lang.NoClassDefFoundError: 
org/fontbox/afm/AFMParser". But in the end nutch again froze.



Steve Betts wrote:

>I am using PDFBox-0.7.2-log4j.jar. That doesn't make it run a lot faster,
>but it does allow it to complete.
>
>Thanks,
>
>Steve Betts
>sbetts@minethurn.com
>937-477-1797
>
>
>-----Original Message-----
>From: "Håvard W. Kongsgård" [mailto:h.kongsgaard@niap.no]
>Sent: Wednesday, January 25, 2006 10:49 AM
>To: nutch-user@lucene.apache.org
>Subject: Re: Parsing PDF Nutch Achilles heel?
>
>PDFBox-0.7.2 or one of the nightly builds PDFBox-0.7.3-dev...
>
>Steve Betts wrote:
>
>  
>
>>I should have included the link, but I used PDFBox.
>>
>>Thanks,
>>
>>Steve Betts
>>sbetts@minethurn.com
>>937-477-1797
>>
>>
>>-----Original Message-----
>>From: "Håvard W. Kongsgård" [mailto:h.kongsgaard@niap.no]
>>Sent: Wednesday, January 25, 2006 10:34 AM
>>To: nutch-user@lucene.apache.org
>>Subject: Re: Parsing PDF Nutch Achilles heel?
>>
>>>From where do I get the new version http://www.pdfbox.org/ or
>>http://svn.apache.org/viewcvs.cgi/lucene/nutch/
>>
>>
>>
>>Steve Betts wrote:
>>
>>
>>
>>    
>>
>>>There is a bug in the PDF parser tool used with 0.7. You can get a newer
>>>version to replace the jars with the parse-pdf plugin and the freeze will
>>>
>>>
>>>      
>>>
>>go
>>
>>
>>    
>>
>>>away.
>>>
>>>Thanks,
>>>
>>>Steve Betts
>>>sbetts@minethurn.com
>>>937-477-1797
>>>
>>>-----Original Message-----
>>>From: "Håvard W. Kongsgård" [mailto:h.kongsgaard@niap.no]
>>>Sent: Wednesday, January 25, 2006 10:10 AM
>>>To: nutch-user@lucene.apache.org
>>>Subject: Parsing PDF Nutch Achilles heel?
>>>
>>>I have been doing some testing on different nutch configurations to see
>>>what slows down the fetching process on my servers(nutch 0.7.1).
>>>My general experience is that the PDF parse process is nutchs Achilles
>>>
>>>
>>>      
>>>
>>heel.
>>
>>
>>    
>>
>>>Nutch works fine on older computers, but with the combination of
>>>|parse-(text|html|pdf)
>>>and http.content.limit = -1(needed to get PDF parsing to work) nutch
>>>sometimes freezes completely.
>>>
>>>Is there planned any improvement to the parsing of PDF files in the next
>>>version of nutch (0.8)?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>      
>>>
>>
>>
>>
>>    
>>
>
>
>
>  
>

Re: Parsing PDF Nutch Achilles heel?

Posted by "Håvard W. Kongsgård" <h....@niap.no>.
"Cud you create a new version from the latest xpdf version,
I know that the older versions of pdftotext (before October 2005) had 
some issues with PDF 1.6 (acrobat 7)."
Sorry my mistake!

Have now tested pdftotext and it's faster than pdfbox, but it doesn't 
prevent the nutch freezes.



Håvard W. Kongsgård wrote:

> Cud you create a new version from the latest xpdf version,
> I know that the older versions of pdftotext (before October 2005) had 
> some issues with PDF 1.6 (acrobat 7).
>
>
>
> Doug Cutting wrote:
>
>> Steve Betts wrote:
>>
>>> I am using PDFBox-0.7.2-log4j.jar. That doesn't make it run a lot 
>>> faster,
>>> but it does allow it to complete.
>>
>>
>>
>> I find xpdf much faster than PDFBox.
>>
>> http://www.mail-archive.com/nutch-dev@incubator.apache.org/msg00161.html
>>
>> Does this work any better for you?
>>
>> Doug
>>
>
>

Re: Parsing PDF Nutch Achilles heel?

Posted by "Håvard W. Kongsgård" <h....@niap.no>.
Cud you create a new version from the latest xpdf version,
I know that the older versions of pdftotext (before October 2005) had some issues with PDF 1.6 (acrobat 7).



Doug Cutting wrote:

> Steve Betts wrote:
>
>> I am using PDFBox-0.7.2-log4j.jar. That doesn't make it run a lot 
>> faster,
>> but it does allow it to complete.
>
>
> I find xpdf much faster than PDFBox.
>
> http://www.mail-archive.com/nutch-dev@incubator.apache.org/msg00161.html
>
> Does this work any better for you?
>
> Doug
>

Re: Parsing PDF Nutch Achilles heel?

Posted by Doug Cutting <cu...@nutch.org>.
Steve Betts wrote:
> I am using PDFBox-0.7.2-log4j.jar. That doesn't make it run a lot faster,
> but it does allow it to complete.

I find xpdf much faster than PDFBox.

http://www.mail-archive.com/nutch-dev@incubator.apache.org/msg00161.html

Does this work any better for you?

Doug

RE: Parsing PDF Nutch Achilles heel?

Posted by Steve Betts <sb...@minethurn.com>.
I am using PDFBox-0.7.2-log4j.jar. That doesn't make it run a lot faster,
but it does allow it to complete.

Thanks,

Steve Betts
sbetts@minethurn.com
937-477-1797


-----Original Message-----
From: "H�vard W. Kongsg�rd" [mailto:h.kongsgaard@niap.no]
Sent: Wednesday, January 25, 2006 10:49 AM
To: nutch-user@lucene.apache.org
Subject: Re: Parsing PDF Nutch Achilles heel?

PDFBox-0.7.2 or one of the nightly builds PDFBox-0.7.3-dev...

Steve Betts wrote:

>I should have included the link, but I used PDFBox.
>
>Thanks,
>
>Steve Betts
>sbetts@minethurn.com
>937-477-1797
>
>
>-----Original Message-----
>From: "H�vard W. Kongsg�rd" [mailto:h.kongsgaard@niap.no]
>Sent: Wednesday, January 25, 2006 10:34 AM
>To: nutch-user@lucene.apache.org
>Subject: Re: Parsing PDF Nutch Achilles heel?
>
> From where do I get the new version http://www.pdfbox.org/ or
>http://svn.apache.org/viewcvs.cgi/lucene/nutch/
>
>
>
>Steve Betts wrote:
>
>
>
>>There is a bug in the PDF parser tool used with 0.7. You can get a newer
>>version to replace the jars with the parse-pdf plugin and the freeze will
>>
>>
>go
>
>
>>away.
>>
>>Thanks,
>>
>>Steve Betts
>>sbetts@minethurn.com
>>937-477-1797
>>
>>-----Original Message-----
>>From: "H�vard W. Kongsg�rd" [mailto:h.kongsgaard@niap.no]
>>Sent: Wednesday, January 25, 2006 10:10 AM
>>To: nutch-user@lucene.apache.org
>>Subject: Parsing PDF Nutch Achilles heel?
>>
>>I have been doing some testing on different nutch configurations to see
>>what slows down the fetching process on my servers(nutch 0.7.1).
>>My general experience is that the PDF parse process is nutchs Achilles
>>
>>
>heel.
>
>
>>Nutch works fine on older computers, but with the combination of
>>|parse-(text|html|pdf)
>>and http.content.limit = -1(needed to get PDF parsing to work) nutch
>>sometimes freezes completely.
>>
>>Is there planned any improvement to the parsing of PDF files in the next
>>version of nutch (0.8)?
>>
>>
>>
>>
>>
>>
>>
>
>
>
>
>

Re: Parsing PDF Nutch Achilles heel?

Posted by "Håvard W. Kongsgård" <h....@niap.no>.
PDFBox-0.7.2 or one of the nightly builds PDFBox-0.7.3-dev...

Steve Betts wrote:

>I should have included the link, but I used PDFBox.
>
>Thanks,
>
>Steve Betts
>sbetts@minethurn.com
>937-477-1797
>
>
>-----Original Message-----
>From: "Håvard W. Kongsgård" [mailto:h.kongsgaard@niap.no]
>Sent: Wednesday, January 25, 2006 10:34 AM
>To: nutch-user@lucene.apache.org
>Subject: Re: Parsing PDF Nutch Achilles heel?
>
> From where do I get the new version http://www.pdfbox.org/ or
>http://svn.apache.org/viewcvs.cgi/lucene/nutch/
>
>
>
>Steve Betts wrote:
>
>  
>
>>There is a bug in the PDF parser tool used with 0.7. You can get a newer
>>version to replace the jars with the parse-pdf plugin and the freeze will
>>    
>>
>go
>  
>
>>away.
>>
>>Thanks,
>>
>>Steve Betts
>>sbetts@minethurn.com
>>937-477-1797
>>
>>-----Original Message-----
>>From: "Håvard W. Kongsgård" [mailto:h.kongsgaard@niap.no]
>>Sent: Wednesday, January 25, 2006 10:10 AM
>>To: nutch-user@lucene.apache.org
>>Subject: Parsing PDF Nutch Achilles heel?
>>
>>I have been doing some testing on different nutch configurations to see
>>what slows down the fetching process on my servers(nutch 0.7.1).
>>My general experience is that the PDF parse process is nutchs Achilles
>>    
>>
>heel.
>  
>
>>Nutch works fine on older computers, but with the combination of
>>|parse-(text|html|pdf)
>>and http.content.limit = -1(needed to get PDF parsing to work) nutch
>>sometimes freezes completely.
>>
>>Is there planned any improvement to the parsing of PDF files in the next
>>version of nutch (0.8)?
>>
>>
>>
>>
>>
>>    
>>
>
>
>
>  
>

RE: Parsing PDF Nutch Achilles heel?

Posted by Steve Betts <sb...@minethurn.com>.
I should have included the link, but I used PDFBox.

Thanks,

Steve Betts
sbetts@minethurn.com
937-477-1797


-----Original Message-----
From: "H�vard W. Kongsg�rd" [mailto:h.kongsgaard@niap.no]
Sent: Wednesday, January 25, 2006 10:34 AM
To: nutch-user@lucene.apache.org
Subject: Re: Parsing PDF Nutch Achilles heel?

 From where do I get the new version http://www.pdfbox.org/ or
http://svn.apache.org/viewcvs.cgi/lucene/nutch/



Steve Betts wrote:

>There is a bug in the PDF parser tool used with 0.7. You can get a newer
>version to replace the jars with the parse-pdf plugin and the freeze will
go
>away.
>
>Thanks,
>
>Steve Betts
>sbetts@minethurn.com
>937-477-1797
>
>-----Original Message-----
>From: "H�vard W. Kongsg�rd" [mailto:h.kongsgaard@niap.no]
>Sent: Wednesday, January 25, 2006 10:10 AM
>To: nutch-user@lucene.apache.org
>Subject: Parsing PDF Nutch Achilles heel?
>
>I have been doing some testing on different nutch configurations to see
>what slows down the fetching process on my servers(nutch 0.7.1).
>My general experience is that the PDF parse process is nutchs Achilles
heel.
>
>Nutch works fine on older computers, but with the combination of
>|parse-(text|html|pdf)
>and http.content.limit = -1(needed to get PDF parsing to work) nutch
>sometimes freezes completely.
>
>Is there planned any improvement to the parsing of PDF files in the next
>version of nutch (0.8)?
>
>
>
>
>

Re: Parsing PDF Nutch Achilles heel?

Posted by "Håvard W. Kongsgård" <h....@niap.no>.
 From where do I get the new version http://www.pdfbox.org/ or 
http://svn.apache.org/viewcvs.cgi/lucene/nutch/



Steve Betts wrote:

>There is a bug in the PDF parser tool used with 0.7. You can get a newer
>version to replace the jars with the parse-pdf plugin and the freeze will go
>away.
>
>Thanks,
>
>Steve Betts
>sbetts@minethurn.com
>937-477-1797
>
>-----Original Message-----
>From: "Håvard W. Kongsgård" [mailto:h.kongsgaard@niap.no]
>Sent: Wednesday, January 25, 2006 10:10 AM
>To: nutch-user@lucene.apache.org
>Subject: Parsing PDF Nutch Achilles heel?
>
>I have been doing some testing on different nutch configurations to see
>what slows down the fetching process on my servers(nutch 0.7.1).
>My general experience is that the PDF parse process is nutchs Achilles heel.
>
>Nutch works fine on older computers, but with the combination of
>|parse-(text|html|pdf)
>and http.content.limit = -1(needed to get PDF parsing to work) nutch
>sometimes freezes completely.
>
>Is there planned any improvement to the parsing of PDF files in the next
>version of nutch (0.8)?
>
>
>
>  
>

RE: Parsing PDF Nutch Achilles heel?

Posted by Steve Betts <sb...@minethurn.com>.
There is a bug in the PDF parser tool used with 0.7. You can get a newer
version to replace the jars with the parse-pdf plugin and the freeze will go
away.

Thanks,

Steve Betts
sbetts@minethurn.com
937-477-1797

-----Original Message-----
From: "H�vard W. Kongsg�rd" [mailto:h.kongsgaard@niap.no]
Sent: Wednesday, January 25, 2006 10:10 AM
To: nutch-user@lucene.apache.org
Subject: Parsing PDF Nutch Achilles heel?

I have been doing some testing on different nutch configurations to see
what slows down the fetching process on my servers(nutch 0.7.1).
My general experience is that the PDF parse process is nutchs Achilles heel.

Nutch works fine on older computers, but with the combination of
|parse-(text|html|pdf)
and http.content.limit = -1(needed to get PDF parsing to work) nutch
sometimes freezes completely.

Is there planned any improvement to the parsing of PDF files in the next
version of nutch (0.8)?