You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by Shigeki Kobayashi <sh...@g.softbank.co.jp> on 2014/12/24 03:43:07 UTC

"start minimal" option even deletes contents whose links are deleted

Hello guys.




I like to clarify how “minimal start” of job execution work in MCF, using
web contents as a repository connection and Solr as an output connection.



I thought it supposed to skip deletion and it only crawls changed or new
documents. However, there is a guy in my team who tested the minimal start
option and he found out even deletion was done.



To be more specific of how the crawled content changed, he deleted one link
from the root page after full crawling. So the next minimal crawl should
avoid deleting the deleted content from the index.



He used an older version of MCF 1.4.1 but it seems there isn’t any change
list about this issue so I suppose the same thing could happen in the
latest version.



How are we supposed to run MCF without index deletion?



Regards,



Shigeki

Re: "start minimal" option even deletes contents whose links are deleted

Posted by Karl Wright <da...@gmail.com>.
Hi Shigeki,

For hop count calculations, MCF keeps track of links between documents.
When a document is detected to be removed, all documents on the queue that
depend on links to that document are marked to be processed.  When such a
document is processed, the hopcount for it is calculated.  If there is no
other way to reach the document from the seed, the hopcount value is
calculated to be infinity, which is greater than whatever limit you have
set.

Thanks,
Karl

Re: "start minimal" option even deletes contents whose links are deleted

Posted by Shigeki Kobayashi <sh...@g.softbank.co.jp>.
Hi Karl.

Here are what I understand.

Minimal crawl does not do the clean up phase.
The clean up phase removes no-longer-reachable documents.
Even when a link of a page is removed from the root page, minimal crawl is
not supposed to remove the index of the no-longer-reachable page.
If hop count is set to 1, then the no-longer-reachable page should not be
affected because its hop count does not exceed 1.

If I am correct above, then I do not understand why the index of the
non-reachable page is deleted.


2014-12-24 13:59 GMT+09:00 Karl Wright <da...@gmail.com>:
>
> Hi Shigeki,
>
> Minimal crawls do not guarantee that there is no document deletion.  Such
> crawls only do the least amount of work possible based on what model the
> underlying connector implements.  This often just means not doing the
> "cleanup" phase at the end of the job run, which typically removes
> no-longer-reachable documents.  But if, for instance, you are using the web
> connector and you have hop count filtering enabled, then the framework will
> keep track of hop count and will remove all documents that exceed it, which
> does not require the end-of-job cleanup phase.
>
> If your goal is to avoid removing any previously crawled documents, then I
> am afraid that MCF does not have any real support for your model.  "Start
> minimal" is certainly not going to help you.
>
> Thanks,
> karl
>
>

Re: "start minimal" option even deletes contents whose links are deleted

Posted by Karl Wright <da...@gmail.com>.
Hi Shigeki,

Minimal crawls do not guarantee that there is no document deletion.  Such
crawls only do the least amount of work possible based on what model the
underlying connector implements.  This often just means not doing the
"cleanup" phase at the end of the job run, which typically removes
no-longer-reachable documents.  But if, for instance, you are using the web
connector and you have hop count filtering enabled, then the framework will
keep track of hop count and will remove all documents that exceed it, which
does not require the end-of-job cleanup phase.

If your goal is to avoid removing any previously crawled documents, then I
am afraid that MCF does not have any real support for your model.  "Start
minimal" is certainly not going to help you.

Thanks,
karl