[jira] [Commented] (CONNECTORS-899) Consider/ignore HTTP header fields when checking for document change

["Florian Schmedding (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CONNECTORS-899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13908414#comment-13908414 ] 

Florian Schmedding commented on CONNECTORS-899:
-----------------------------------------------

Perhaps there is a mor minimal solution as indicated in [CONNECTORS-850|https://issues.apache.org/jira/browse/CONNECTORS-850?focusedCommentId=13901754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13901754].

> Consider/ignore HTTP header fields when checking for document change
> --------------------------------------------------------------------
>
>                 Key: CONNECTORS-899
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-899
>             Project: ManifoldCF
>          Issue Type: Improvement
>          Components: Web connector
>    Affects Versions: ManifoldCF 1.6
>            Reporter: Florian Schmedding
>            Assignee: Karl Wright
>            Priority: Minor
>              Labels: http
>             Fix For: ManifoldCF 1.6
>
>
> The web connector does already ignore certain HTTP header fields that change on every request when checking for document changes. However, this is hardcoded. Some web servers are not properly configured and return even a new last-modified date on each request although the document remains the same. This leads to lots of unncecessary re-ingestions. It would be nice to have the possibility to configure the header fields that should be considerd and ignored.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)