[jira] [Resolved] (NUTCH-933) Fetcher does not save a pages Last-Modified value in CrawlDatum

["Julien Nioche (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Julien Nioche resolved NUTCH-933.
---------------------------------

       Resolution: Not a Problem
    Fix Version/s:     (was: 1.9)

Marked as not a problem. PLease reopen if necessary

> Fetcher does not save a pages Last-Modified value in CrawlDatum
> ---------------------------------------------------------------
>
>                 Key: NUTCH-933
>                 URL: https://issues.apache.org/jira/browse/NUTCH-933
>             Project: Nutch
>          Issue Type: Bug
>          Components: fetcher
>    Affects Versions: 1.2
>            Reporter: Joe Kemp
>
> I added the following code in the output method just after the If (content !=null) statement.
>         String lastModified = metadata.get("Last-Modified");
>         if (lastModified !=null && !lastModified.equals("")) {
>         	try {
> 				Date lastModifiedDate = DateUtil.parseDate(lastModified);
> 				datum.setModifiedTime(lastModifiedDate.getTime());
> 			} catch (DateParseException e) {
> 				
> 			}
>         }
> I now get 304 for pages that haven't changed when I recrawl.  Need to do further testing.  Might also need a configuration parameter to turn off this behavior, allowing pages to be forced to be refreshed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)