You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Mike Alulin <mi...@yahoo.com> on 2006/01/12 18:51:32 UTC
MapReduce and segment merging
Is it possible to merge segments in the map reduce version of Nutch?
---------------------------------
Yahoo! Photos Showcase holiday pictures in hardcover
Photo Books. You design it and well bind it!
Re: MapReduce and segment merging
Posted by Andrzej Bialecki <ab...@getopt.org>.
Mike Alulin wrote:
> Then how people uses the new version if they need let's say daily crawls of the new/updated pages? I crawl updated pages every 24 hours and if I do not merge the segments, soon I will have hundreds of them. What is the best solution in this case?
>
> Full recrawl is not a good option as i have millions of documents and I DO know which of them were updated without requesting them.
>
This is a development version, nobody said it's feature complete.
Patience, my friend... or spend some effort to improve it. ;-)
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: MapReduce and segment merging
Posted by Mike Alulin <mi...@yahoo.com>.
Then how people uses the new version if they need let's say daily crawls of the new/updated pages? I crawl updated pages every 24 hours and if I do not merge the segments, soon I will have hundreds of them. What is the best solution in this case?
Full recrawl is not a good option as i have millions of documents and I DO know which of them were updated without requesting them.
Andrzej Bialecki <ab...@getopt.org> wrote: Mike Alulin wrote:
> Is it possible to merge segments in the map reduce version of Nutch?
>
Not yet.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------
Yahoo! Photos
Got holiday prints? See all the ways to get quality prints in your hands ASAP.
Re: MapReduce and segment merging
Posted by Andrzej Bialecki <ab...@getopt.org>.
Mike Alulin wrote:
> Is it possible to merge segments in the map reduce version of Nutch?
>
Not yet.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com