[jira] [Comment Edited] (NUTCH-2222) re-fetch deletes all metadata except _csh_ and _rs_

["Lewis John McGibbney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176048#comment-15176048 ] 

Lewis John McGibbney edited comment on NUTCH-2222 at 3/2/16 5:34 PM:
---------------------------------------------------------------------

I don't think there is any workaround no. The way to deal with this is for us to fix the bug. I have been hammered with work and did not get around to fixing it. If you can make an attempt to write a test based upon [TestFetcher|https://github.com/apache/nutch/blob/2.x/src/test/org/apache/nutch/fetcher/TestFetcher.java] and call it testReFetch then we will be able to replicate your issue and narrow it down.


was (Author: lewismc):
I don't think there is any workaround no. The way to deal with this is for us to fix the bug. I have been hammered with work and did not get around to fixing it. If you can make an attempt to write a test based upon [TestFetcher](https://github.com/apache/nutch/blob/2.x/src/test/org/apache/nutch/fetcher/TestFetcher.java) and call it testReFetch then we will be able to replicate your issue and narrow it down.

> re-fetch deletes all  metadata except _csh_ and _rs_
> ----------------------------------------------------
>
>                 Key: NUTCH-2222
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2222
>             Project: Nutch
>          Issue Type: Bug
>          Components: crawldb
>    Affects Versions: 2.3.1
>         Environment: Centos 6, mongodb 2.6 and mongodb 3.0 and hbase-0.98.8-hadoop2
>            Reporter: Adnane B.
>            Assignee: Lewis John McGibbney
>             Fix For: 2.3.2
>
>
> This problem happens at the the second time I crawl a page
> {code}
> bin/nutch inject urls/
> bin/nutch generate -topN 1000
> bin/nutch fetch  -all
> bin/nutch parse -force   -all
> bin/nutch updatedb  -all
> {code}
> seconde time (re-fetch) : 
> {code}
> bin/nutch generate -topN 1000 --> batchid changes for all existing pages
> bin/nutch fetch  -all   -->  *** metadatas are delete for all pages already crawled  **
> bin/nutch parse -force   -all
> bin/nutch updatedb  -all
> {code}
> I reproduce it with mongodb 2.6, mongodb 3.0, and hbase-0.98.8-hadoop2
> It happens only if the page has not changed
> To reproduce easily, please add to nutch-site.xml :
> {code}
> <property>
>   <name>db.fetch.interval.default</name>
>   <value>60</value>
>   <description>The default number of seconds between re-fetches of a page (1 minute)
> </description>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)