You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by Julien Massiera <ju...@francelabs.com> on 2018/10/24 09:26:08 UTC
How documents are deleted
Hi Karl,
I am trying to understand the behavior of ManifoldCF during a re-crawl
and specially how missing documents are deleted and by which process ?
I am focusing on two repository connectors, the JCIFS one and the JDBC
one. Here is what I understand so far :
In the JCIFS connector, the addSeedDocuments method list all the files
found for each configured path. So it seems clear that any previously
crawled files that have not been listed during a re-crawl by this method
should be deleted.
In the JDBC connector, the addSeedDocuments method only list the new or
modified documents during a re-crawl (if, of course, the id query is
correctly using the starttime and endtime variables). So here, there is
a difference between the two connectors. It means that to delete missing
documents, the previously crawled ones need to be 'checked' with the
version query to detect the documents that must be removed.
I am currently unable to tell what is really performed by ManifoldCF to
deal with documents to delete and if any of the assumptions I exposed
above are correct and/or used. Also, I am really interested to know
which part of the code is performing the delete process.
Thanks for your help.
--
Julien MASSIERA
Directeur développement produit
France Labs – Les experts du Search
Retrouvez-nous à l’Enterprise Search & Discovery Summit à Washington DC
www.francelabs.com
Re: How documents are deleted
Posted by Julien Massiera <ju...@francelabs.com>.
Hi Karl,
thanks for your response, I found in the documentation what I need.
Julien
On 24/10/2018 16:06, Karl Wright wrote:
> Hi Julien,
>
> This is a complex question and the framework behaves differently depending
> on the connector model. Please read:
>
> https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs
>
> Karl
>
>
> On Wed, Oct 24, 2018 at 5:26 AM Julien Massiera <
> julien.massiera@francelabs.com> wrote:
>
>> Hi Karl,
>>
>> I am trying to understand the behavior of ManifoldCF during a re-crawl
>> and specially how missing documents are deleted and by which process ?
>>
>> I am focusing on two repository connectors, the JCIFS one and the JDBC
>> one. Here is what I understand so far :
>>
>> In the JCIFS connector, the addSeedDocuments method list all the files
>> found for each configured path. So it seems clear that any previously
>> crawled files that have not been listed during a re-crawl by this method
>> should be deleted.
>>
>> In the JDBC connector, the addSeedDocuments method only list the new or
>> modified documents during a re-crawl (if, of course, the id query is
>> correctly using the starttime and endtime variables). So here, there is
>> a difference between the two connectors. It means that to delete missing
>> documents, the previously crawled ones need to be 'checked' with the
>> version query to detect the documents that must be removed.
>>
>> I am currently unable to tell what is really performed by ManifoldCF to
>> deal with documents to delete and if any of the assumptions I exposed
>> above are correct and/or used. Also, I am really interested to know
>> which part of the code is performing the delete process.
>>
>> Thanks for your help.
>>
>> --
>> Julien MASSIERA
>> Directeur développement produit
>> France Labs – Les experts du Search
>> Retrouvez-nous à l’Enterprise Search & Discovery Summit à Washington DC
>> www.francelabs.com
>>
>>
--
Julien MASSIERA
Directeur développement produit
France Labs – Les experts du Search
Retrouvez-nous à l’Enterprise Search & Discovery Summit à Washington DC
www.francelabs.com
Re: How documents are deleted
Posted by Karl Wright <da...@gmail.com>.
Hi Julien,
This is a complex question and the framework behaves differently depending
on the connector model. Please read:
https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs
Karl
On Wed, Oct 24, 2018 at 5:26 AM Julien Massiera <
julien.massiera@francelabs.com> wrote:
> Hi Karl,
>
> I am trying to understand the behavior of ManifoldCF during a re-crawl
> and specially how missing documents are deleted and by which process ?
>
> I am focusing on two repository connectors, the JCIFS one and the JDBC
> one. Here is what I understand so far :
>
> In the JCIFS connector, the addSeedDocuments method list all the files
> found for each configured path. So it seems clear that any previously
> crawled files that have not been listed during a re-crawl by this method
> should be deleted.
>
> In the JDBC connector, the addSeedDocuments method only list the new or
> modified documents during a re-crawl (if, of course, the id query is
> correctly using the starttime and endtime variables). So here, there is
> a difference between the two connectors. It means that to delete missing
> documents, the previously crawled ones need to be 'checked' with the
> version query to detect the documents that must be removed.
>
> I am currently unable to tell what is really performed by ManifoldCF to
> deal with documents to delete and if any of the assumptions I exposed
> above are correct and/or used. Also, I am really interested to know
> which part of the code is performing the delete process.
>
> Thanks for your help.
>
> --
> Julien MASSIERA
> Directeur développement produit
> France Labs – Les experts du Search
> Retrouvez-nous à l’Enterprise Search & Discovery Summit à Washington DC
> www.francelabs.com
>
>