Segment cleanup

[Bai Shen <ba...@gmail.com>]

Is there a reason to keep a segment around after it's been indexed?  When
following the tutorial, I ended up sending the same segment to the solr
server multiple times because I was using segments/* as my argument.

Once I've sent it to the solr server, is there any reason not to delete that
segment?

Thanks.

Re: Segment cleanup

[Markus Jelsma <ma...@openindex.io>]

On Wednesday 26 October 2011 16:24:15 Bai Shen wrote:
> On Tue, Oct 25, 2011 at 1:25 PM, Markus Jelsma
> 
> <ma...@openindex.io>wrote:
> > > Is there a reason to keep a segment around after it's been indexed? 
> > > When following the tutorial, I ended up sending the same segment to
> > > the solr server multiple times because I was using segments/* as my
> > > argument.
> > 
> > Only send the segment(s) that have not been indexed yet unless you have
> > to reindex everything each time.
> 
> -nods-  I didn't mean to send the same segment multiple times.  I just
> didn't quite realize what the index command was doing.
> 
> > > Once I've sent it to the solr server, is there any reason not to delete
> > > that segment?
> > 
> > You can delete a segment if:
> > - you don't do any reindexing;
> > - it's older than fetch interval (default 30 days) and you are sure all
> > URL's
> > in that segment have already been fetched in newer segment(s);
> > - don't need the stored content in the segment.
> 
> What do you mean by reindexing?  Doesn't nutch handle this with it's
> refetching of content after it expires?
> 
> Once I parse the content and update the db, wouldn't the segment be
> irrelevant in regards to whether they get fetched or not?
> 
> The content already gets stored in the solr server to facilitate
> highlighting. So I can't see why we would need to store it in nutch.

Reindexing is useful for development environments or even production when 
Solr's index time analysis changes. If you change for tokens get indexed or 
enable of disable norms, you must reindex from scratch.



-- 
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Re: Segment cleanup

[Bai Shen <ba...@gmail.com>]

On Tue, Oct 25, 2011 at 1:25 PM, Markus Jelsma
<ma...@openindex.io>wrote:

> > Is there a reason to keep a segment around after it's been indexed?  When
> > following the tutorial, I ended up sending the same segment to the solr
> > server multiple times because I was using segments/* as my argument.
>
> Only send the segment(s) that have not been indexed yet unless you have to
> reindex everything each time.
>
>
-nods-  I didn't mean to send the same segment multiple times.  I just
didn't quite realize what the index command was doing.


> >
> > Once I've sent it to the solr server, is there any reason not to delete
> > that segment?
>
> You can delete a segment if:
> - you don't do any reindexing;
> - it's older than fetch interval (default 30 days) and you are sure all
> URL's
> in that segment have already been fetched in newer segment(s);
> - don't need the stored content in the segment.
>
>
What do you mean by reindexing?  Doesn't nutch handle this with it's
refetching of content after it expires?

Once I parse the content and update the db, wouldn't the segment be
irrelevant in regards to whether they get fetched or not?

The content already gets stored in the solr server to facilitate
highlighting. So I can't see why we would need to store it in nutch.

Re: Segment cleanup

[Markus Jelsma <ma...@openindex.io>]

> Is there a reason to keep a segment around after it's been indexed?  When
> following the tutorial, I ended up sending the same segment to the solr
> server multiple times because I was using segments/* as my argument.

Only send the segment(s) that have not been indexed yet unless you have to 
reindex everything each time.

> 
> Once I've sent it to the solr server, is there any reason not to delete
> that segment?

You can delete a segment if:
- you don't do any reindexing;
- it's older than fetch interval (default 30 days) and you are sure all URL's 
in that segment have already been fetched in newer segment(s);
- don't need the stored content in the segment.

> 
> Thanks.