You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by A Laxmi <a....@gmail.com> on 2014/09/29 00:13:25 UTC
nutch 1.8 pdf crawl issue
Hi,
I have crawled a bunch of PDF urls using Nutch 1.8. It returned empty
"title" and "content" for some of the PDF urls. When I pulled up one such
url, the text seems to be easily selectable and does *not* contain a bunch
of images as in (non-ocr'd pdf), I am confused about why Nutch returned
empty values for "title" and "content" for such a pdf. Example url for
which Nutch returned empty title and content-
http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf
The way I figured out title and content was empty was through Solr Admin.
After it was crawled and indexed in Solr, I search for that url in Solr
Admin UI and it had these values for title, content, url, type fields -
>From Solr Admin:
title,content,url,type
"",,http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf,"application/pdf,application,pdf"
Any thoughts please???
Thanks
Re: nutch 1.8 pdf crawl issue
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi,
> How do we know it has "robots:noindex"?
> http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf
If you take a look into the PDF (using pdfedit, PDFbox, or simply
any text editor): the robots=noindex is defined in the metadata catalog
as embedded XML (XMP metadata, see http://xml.coverpages.org/XMP-Embedding.pdf):
<rdf:Description rdf:about=""
xmlns:pdfx="http://ns.adobe.com/pdfx/1.3/">
<pdfx:ElsevierWebPDFSpecifications>6.2</pdfx:ElsevierWebPDFSpecifications>
<pdfx:doi>10.1016/j.ecolmodel.2012.03.014</pdfx:doi>
<pdfx:robots>noindex</pdfx:robots>
<pdfx:CrossmarkMajorVersionDate>2010-04-23</pdfx:CrossmarkMajorVersionDate>
As said Tika obviously converts this into
<meta name="robots" content="noindex"/>
which causes Nutch not to index the document content.
Although this behavior reasonable, I couldn't find any "official" documentation,
also the document is indexed by Google: that's possibly a bug of Tika and/or Nutch.
Feel free to open a Jira issue. Or, alternatively, ask on the Tika mailing list.
Best,
Sebastian
On 09/29/2014 05:24 PM, A Laxmi wrote:
> Hi Sebastian,
>
> How do we know it has "robots:noindex"? The link I am referring to is -
> http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf
>
> Thanks for your help!
>
>
>
> On Mon, Sep 29, 2014 at 5:38 AM, Sebastian Nagel <wastl.nagel@googlemail.com
>> wrote:
>
>> Hi,
>>
>> that's caused by a "robots:noindex" in the info dict of the PDF.
>> Tika puts this into the metadata
>> <meta name="robots" content="noindex"/>
>> and Nutch then empties title and content.
>>
>> I haven't been aware of this way of excluding non-HTML documents,
>> so we have to check whether this is a but or not.
>> The intention of authors/creators of the PDF is matched,
>> I guess, but the normal way to exclude PDFs from being indexed
>> is via HTTP header, cf.
>>
>>
>> https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
>>
>> Best,
>> Sebastian
>>
>> 2014-09-29 0:13 GMT+02:00 A Laxmi <a....@gmail.com>:
>>
>>> Hi,
>>>
>>> I have crawled a bunch of PDF urls using Nutch 1.8. It returned empty
>>> "title" and "content" for some of the PDF urls. When I pulled up one such
>>> url, the text seems to be easily selectable and does *not* contain a
>> bunch
>>> of images as in (non-ocr'd pdf), I am confused about why Nutch returned
>>> empty values for "title" and "content" for such a pdf. Example url for
>>> which Nutch returned empty title and content-
>>> http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf
>>>
>>> The way I figured out title and content was empty was through Solr Admin.
>>> After it was crawled and indexed in Solr, I search for that url in Solr
>>> Admin UI and it had these values for title, content, url, type fields -
>>>
>>> From Solr Admin:
>>>
>>> title,content,url,type
>>> "",,http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf,
>>> "application/pdf,application,pdf"
>>>
>>>
>>> Any thoughts please???
>>>
>>> Thanks
>>>
>>
>
Re: nutch 1.8 pdf crawl issue
Posted by A Laxmi <a....@gmail.com>.
Hi Sebastian,
How do we know it has "robots:noindex"? The link I am referring to is -
http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf
Thanks for your help!
On Mon, Sep 29, 2014 at 5:38 AM, Sebastian Nagel <wastl.nagel@googlemail.com
> wrote:
> Hi,
>
> that's caused by a "robots:noindex" in the info dict of the PDF.
> Tika puts this into the metadata
> <meta name="robots" content="noindex"/>
> and Nutch then empties title and content.
>
> I haven't been aware of this way of excluding non-HTML documents,
> so we have to check whether this is a but or not.
> The intention of authors/creators of the PDF is matched,
> I guess, but the normal way to exclude PDFs from being indexed
> is via HTTP header, cf.
>
>
> https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
>
> Best,
> Sebastian
>
> 2014-09-29 0:13 GMT+02:00 A Laxmi <a....@gmail.com>:
>
> > Hi,
> >
> > I have crawled a bunch of PDF urls using Nutch 1.8. It returned empty
> > "title" and "content" for some of the PDF urls. When I pulled up one such
> > url, the text seems to be easily selectable and does *not* contain a
> bunch
> > of images as in (non-ocr'd pdf), I am confused about why Nutch returned
> > empty values for "title" and "content" for such a pdf. Example url for
> > which Nutch returned empty title and content-
> > http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf
> >
> > The way I figured out title and content was empty was through Solr Admin.
> > After it was crawled and indexed in Solr, I search for that url in Solr
> > Admin UI and it had these values for title, content, url, type fields -
> >
> > From Solr Admin:
> >
> > title,content,url,type
> > "",,http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf,
> > "application/pdf,application,pdf"
> >
> >
> > Any thoughts please???
> >
> > Thanks
> >
>
Re: nutch 1.8 pdf crawl issue
Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi,
that's caused by a "robots:noindex" in the info dict of the PDF.
Tika puts this into the metadata
<meta name="robots" content="noindex"/>
and Nutch then empties title and content.
I haven't been aware of this way of excluding non-HTML documents,
so we have to check whether this is a but or not.
The intention of authors/creators of the PDF is matched,
I guess, but the normal way to exclude PDFs from being indexed
is via HTTP header, cf.
https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag
Best,
Sebastian
2014-09-29 0:13 GMT+02:00 A Laxmi <a....@gmail.com>:
> Hi,
>
> I have crawled a bunch of PDF urls using Nutch 1.8. It returned empty
> "title" and "content" for some of the PDF urls. When I pulled up one such
> url, the text seems to be easily selectable and does *not* contain a bunch
> of images as in (non-ocr'd pdf), I am confused about why Nutch returned
> empty values for "title" and "content" for such a pdf. Example url for
> which Nutch returned empty title and content-
> http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf
>
> The way I figured out title and content was empty was through Solr Admin.
> After it was crawled and indexed in Solr, I search for that url in Solr
> Admin UI and it had these values for title, content, url, type fields -
>
> From Solr Admin:
>
> title,content,url,type
> "",,http://www.fs.fed.us/global/iitf/pubs/ja_iitf_2012_holm001.pdf,
> "application/pdf,application,pdf"
>
>
> Any thoughts please???
>
> Thanks
>