You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by ytthet <ye...@gmail.com> on 2010/11/12 19:27:56 UTC
When and how(properly) to delete segments directory (Nutch 1.0)
Hi All,
I like to know when and how to delete segments (directories) in Nutch 1.0.
I searched through mailing list archive, but I can't find the answers.
Following is my background information.
My crawl-fetch-index process is executed once a day by scheduled job. My
"db.fetch.interval.max" is 1, so I am expecting urls to be fetched and
indexed everyday. I am not merging segments in my crawl-fetch-index process
because I can't afford Storage Space and RAM. (Merging segment is one of the
popular discussion in this thread I guess).
On First day, I have 6 folders in /segments/ (because i crawled 6 depth).
Total of 1 GiB. Second day I have another 6 more folders worth of 1 GiB++
Now I have total of 2 GiB. Third day, 1 GiB++ and now I have around 3GIB++.
My question is when can I remove those old folder from /segments/? And how
do I remove it?
I tried deleting previous segment (e.g from first day) by linux "rm" command
and they are gone. But searcher no longer works.
I saw suggestion on one entry "segments are no longer being referenced by
indexes which are
using in searches, simply delete the segments/xxxxxxxxxx directory. " Is
that correct?
If so how exactly?
Thanks for your time,
YT Thet
--
View this message in context: http://lucene.472066.n3.nabble.com/When-and-how-properly-to-delete-segments-directory-Nutch-1-0-tp1890600p1890600.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: When and how(properly) to delete segments directory (Nutch 1.0)
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-11-12 20:33, Alexander Aristov wrote:
> why are you so afraid of segment merger? It appears to be the only
> "official" way to get rid of excessive folders. of course it's time/resource
> consuming but is your system so high loaded?
The truth is that in 99% cases SegmentMerger is not needed, and it is
indeed very resource intensive. Merging segments is completely optional
- NutchBean and Nutch search server will work just fine without it, and
the performance degradation related to using many segments is small. Of
course up to a point - if you have hundreds of tiny segments you should
merge them (at least some of them into larger segments), but if you have
just 30 or so then it should not be a big deal.
>
> Also I might be wrong but if you are not planning to return summaries and
> content from nutch when you can remove folders by rm.
>
> And you can completely get rid of segments by using the solr indexer. After
> that you perform indexing you can delete fetched segments. I presume this is
> what you saw in other threads.
If you keep up with re-fetching, i.e. new fetch cycles can keep up with
the amount of pages that become outdated, then you can remove all
segments older than db.max.fetch.interval, because Nutch forces all
pages older than that to become ready for re-fetching, so they will be
likely found in some newer segment.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: When and how(properly) to delete segments directory (Nutch 1.0)
Posted by Ye T Thet <ye...@gmail.com>.
Alexander,
I can't afford to execute segment merger after every crawl, fetch, index
circle. Because I am using single Linux box with 1GiB RAM and 32 GiB HDD.
I set java heap size to 768 MiB to run Nutch. Still when I reach 3 GiB of
segments, temp files used by hadoop for segment merging just impossible (for
my resources). It took up all HDD space.
That is the reason why I am afraid of segment merger.
I still have the need to return summaries. Thus I would need the segments
that at still in index.
I have not tried solr indexer, I might explore that options.
Thanks for your pointers.
Y.T. Thet
On Sat, Nov 13, 2010 at 3:33 AM, Alexander Aristov <
alexander.aristov@gmail.com> wrote:
> why are you so afraid of segment merger? It appears to be the only
> "official" way to get rid of excessive folders. of course it's
> time/resource
> consuming but is your system so high loaded?
>
> Also I might be wrong but if you are not planning to return summaries and
> content from nutch when you can remove folders by rm.
>
> And you can completely get rid of segments by using the solr indexer. After
> that you perform indexing you can delete fetched segments. I presume this
> is
> what you saw in other threads.
>
> Best Regards
> Alexander Aristov
>
>
> On 12 November 2010 21:27, ytthet <ye...@gmail.com> wrote:
>
> >
> > Hi All,
> >
> > I like to know when and how to delete segments (directories) in Nutch
> 1.0.
> >
> > I searched through mailing list archive, but I can't find the answers.
> >
> > Following is my background information.
> >
> > My crawl-fetch-index process is executed once a day by scheduled job. My
> > "db.fetch.interval.max" is 1, so I am expecting urls to be fetched and
> > indexed everyday. I am not merging segments in my crawl-fetch-index
> process
> > because I can't afford Storage Space and RAM. (Merging segment is one of
> > the
> > popular discussion in this thread I guess).
> >
> > On First day, I have 6 folders in /segments/ (because i crawled 6 depth).
> > Total of 1 GiB. Second day I have another 6 more folders worth of 1 GiB++
> > Now I have total of 2 GiB. Third day, 1 GiB++ and now I have around
> 3GIB++.
> >
> > My question is when can I remove those old folder from /segments/? And
> how
> > do I remove it?
> >
> > I tried deleting previous segment (e.g from first day) by linux "rm"
> > command
> > and they are gone. But searcher no longer works.
> >
> > I saw suggestion on one entry "segments are no longer being referenced by
> > indexes which are
> > using in searches, simply delete the segments/xxxxxxxxxx directory. " Is
> > that correct?
> >
> > If so how exactly?
> >
> > Thanks for your time,
> >
> > YT Thet
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/When-and-how-properly-to-delete-segments-directory-Nutch-1-0-tp1890600p1890600.html
> > Sent from the Nutch - User mailing list archive at Nabble.com.
> >
>
Re: When and how(properly) to delete segments directory (Nutch 1.0)
Posted by Alexander Aristov <al...@gmail.com>.
why are you so afraid of segment merger? It appears to be the only
"official" way to get rid of excessive folders. of course it's time/resource
consuming but is your system so high loaded?
Also I might be wrong but if you are not planning to return summaries and
content from nutch when you can remove folders by rm.
And you can completely get rid of segments by using the solr indexer. After
that you perform indexing you can delete fetched segments. I presume this is
what you saw in other threads.
Best Regards
Alexander Aristov
On 12 November 2010 21:27, ytthet <ye...@gmail.com> wrote:
>
> Hi All,
>
> I like to know when and how to delete segments (directories) in Nutch 1.0.
>
> I searched through mailing list archive, but I can't find the answers.
>
> Following is my background information.
>
> My crawl-fetch-index process is executed once a day by scheduled job. My
> "db.fetch.interval.max" is 1, so I am expecting urls to be fetched and
> indexed everyday. I am not merging segments in my crawl-fetch-index process
> because I can't afford Storage Space and RAM. (Merging segment is one of
> the
> popular discussion in this thread I guess).
>
> On First day, I have 6 folders in /segments/ (because i crawled 6 depth).
> Total of 1 GiB. Second day I have another 6 more folders worth of 1 GiB++
> Now I have total of 2 GiB. Third day, 1 GiB++ and now I have around 3GIB++.
>
> My question is when can I remove those old folder from /segments/? And how
> do I remove it?
>
> I tried deleting previous segment (e.g from first day) by linux "rm"
> command
> and they are gone. But searcher no longer works.
>
> I saw suggestion on one entry "segments are no longer being referenced by
> indexes which are
> using in searches, simply delete the segments/xxxxxxxxxx directory. " Is
> that correct?
>
> If so how exactly?
>
> Thanks for your time,
>
> YT Thet
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/When-and-how-properly-to-delete-segments-directory-Nutch-1-0-tp1890600p1890600.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>