You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by 李 敏慧(情報システム本部)- Li Minhui - <mi...@g.softbank.co.jp> on 2014/10/29 08:08:38 UTC

About the processing of MCF incremental crawling

Hi guys

I describe my question with following steps.

1, Run a MCF job to crawl a document.
2, The document is indexed to Solr with fields: "id", "name", "content",
"path".
3, Run a batch, this batch update the document index in Solr with SolrJ,
     and it add two fields to the index:   "comment", "bookmark"
4, I modify the document.
5, Run the MCF job again, this time, it is  incremental crawling.
6, I found that the fields "comment" and "bookmark" had been deleted from
the index of Solr.

Dose MCF delete・insert when processing incremental crawling?
Is there any way to let MCF just do update if the document had been indexed
in the Solr when it processing incremental crawling?

Thanks.


Minhui Li


--

Re: About the processing of MCF incremental crawling

Posted by 李 敏慧(情報システム本部)- Li Minhui - <mi...@g.softbank.co.jp>.
Dear Karl

Thanks for your advice.

*Best Regards*
*Minhui Li*

2014-10-29 16:33 GMT+09:00 Karl Wright <da...@gmail.com>:

>
> Hi Minhui,
>
> ManifoldCF is not designed to somehow know about external changes you have
> made to an index outside of ManifoldCF.  It updates entire documents, using
> delete/insert.  If you want to manage other fields, I suggest that the best
> way to do it is to write a transformation connector that adds the "comment"
> and "bookmark" fields directly to a given RepositoryDocument *before* it is
> sent to Solr, and let MCF manage the entire incremental indexing process.
>
> Thanks,
> Karl
>



-- 
****************************************************************
ソフトバンクモバイル株式会社
情報システム本部 システムサービス事業統括部
コミュニケーションサービス部
検索エンジン開発課

李 敏慧
Email  minhui.li@g.softbank.co.jp <mi...@g.softbank.co.jp>
TEL    080-3914-6311
****************************************************************

Re: About the processing of MCF incremental crawling

Posted by Karl Wright <da...@gmail.com>.
Hi Minhui,

ManifoldCF is not designed to somehow know about external changes you have
made to an index outside of ManifoldCF.  It updates entire documents, using
delete/insert.  If you want to manage other fields, I suggest that the best
way to do it is to write a transformation connector that adds the "comment"
and "bookmark" fields directly to a given RepositoryDocument *before* it is
sent to Solr, and let MCF manage the entire incremental indexing process.

Thanks,
Karl