You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Anas Laffet (JIRA)" <ji...@apache.org> on 2018/07/22 20:27:00 UTC
[jira] [Commented] (NUTCH-2222) re-fetch deletes all metadata
except _csh_ and _rs_
[ https://issues.apache.org/jira/browse/NUTCH-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552146#comment-16552146 ]
Anas Laffet commented on NUTCH-2222:
------------------------------------
Hello all,
I have recently started working with Nutch and I came across the same issue. I just added:
{code:java}
FIELDS.add(WebPage.Field.METADATA);{code}
in
{code:java}
$NUTCH_HOME/src/java/org/apache/nutch/fetcher/FetcherJob.java{code}
and I was able to get the metadata after re-fetch.
I hope this helps.
Cheers!
> re-fetch deletes all metadata except _csh_ and _rs_
> ----------------------------------------------------
>
> Key: NUTCH-2222
> URL: https://issues.apache.org/jira/browse/NUTCH-2222
> Project: Nutch
> Issue Type: Bug
> Components: crawldb
> Affects Versions: 2.3.1
> Environment: Centos 6, mongodb 2.6 and mongodb 3.0 and hbase-0.98.8-hadoop2
> Reporter: Adnane B.
> Assignee: Furkan KAMACI
> Priority: Major
> Fix For: 2.4
>
> Attachments: TestReFetch.java, index.html
>
>
> This problem happens at the the second time I crawl a page
> {code}
> bin/nutch inject urls/
> bin/nutch generate -topN 1000
> bin/nutch fetch -all
> bin/nutch parse -force -all
> bin/nutch updatedb -all
> {code}
> seconde time (re-fetch) :
> {code}
> bin/nutch generate -topN 1000 --> batchid changes for all existing pages
> bin/nutch fetch -all --> *** metadatas are delete for all pages already crawled **
> bin/nutch parse -force -all
> bin/nutch updatedb -all
> {code}
> I reproduce it with mongodb 2.6, mongodb 3.0, and hbase-0.98.8-hadoop2
> It happens only if the page has not changed
> To reproduce easily, please add to nutch-site.xml :
> {code}
> <property>
> <name>db.fetch.interval.default</name>
> <value>60</value>
> <description>The default number of seconds between re-fetches of a page (1 minute)
> </description>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)