You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Karl Wright (JIRA)" <ji...@apache.org> on 2014/01/05 16:17:52 UTC

[jira] [Commented] (CONNECTORS-850) Maximum interval in dynamic crawling

    [ https://issues.apache.org/jira/browse/CONNECTORS-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13862567#comment-13862567 ] 

Karl Wright commented on CONNECTORS-850:
----------------------------------------

Looked at the code in some depth.

The only time the web connector currently overrides the default interval (determined by history) is when it sees a TTL value in an RSS feed.  So I think that the best way to deal with this is to make it a narrow feature of the web connector, and add a field that allows the upper bound to be specified for a job.  Does this work for you?

> Maximum interval in dynamic crawling
> ------------------------------------
>
>                 Key: CONNECTORS-850
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-850
>             Project: ManifoldCF
>          Issue Type: New Feature
>          Components: Framework crawler agent
>    Affects Versions: ManifoldCF 1.4.1
>            Reporter: Florian Schmedding
>            Assignee: Karl Wright
>            Priority: Minor
>              Labels: features
>             Fix For: ManifoldCF 1.5
>
>
> Currently, the dynamic crawling method used for a continuous job extends the reseed and recrawl intervals when no changes are found in a checked document. However, it should be possible to restrict this extension to a maximum value in order to make sure that new documents are discovered within a certain interval.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)