You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Saurabh Gokhale <sa...@gmail.com> on 2011/07/15 21:03:40 UTC
Questions on index Writer
Hi All,
I have following questions about lucene indexWriter. I am using version
3.1.0.
While indexing documents,
1. When is the good time to commit changes? (indexWriter.commit) or just
close the writer after the indexing is done so that commit automatically
happens.
2. When is the good time to merge indexes (indexWriter.maybeMerge()). Is it
just before committing the changes or after indexing say X number of
documents. (I recently upgraded from 2.9.4 to 3.1 and I see 3.1 lucene
generates lot of small index files while indexing document)
Also I have a problem where my lucene index files sometimes gets deleted
from the index folder. I am not sure what code snippet is causing the
existing index files to accidently get removed.
My indexer runs in a thread loop where it indexes file whenever they are
available. When no more files are available, indexer thread closes the
writer and goes to sleep, after specific time, it again creates a new index
on the same folder and starts indexing new files if any available.
A. Is it a wrong way to index files?
B. Because I close the index and open it again later, am I seeing my lucene
index files getting deleted?
Thanks
Saurabh
Re: Questions on index Writer
Posted by Mark Miller <ma...@gmail.com>.
My advice: Don't close the IndexWriter - just call commit. Don't worry about forcing merges - let them happen as they do when you call commit.
If you are going to use the IndexWriter again, you generally do not want to close it. Calling commit is the preferred option.
- Mark Miller
lucidimagination.com
On Jul 15, 2011, at 3:03 PM, Saurabh Gokhale wrote:
> Hi All,
>
> I have following questions about lucene indexWriter. I am using version
> 3.1.0.
>
> While indexing documents,
> 1. When is the good time to commit changes? (indexWriter.commit) or just
> close the writer after the indexing is done so that commit automatically
> happens.
> 2. When is the good time to merge indexes (indexWriter.maybeMerge()). Is it
> just before committing the changes or after indexing say X number of
> documents. (I recently upgraded from 2.9.4 to 3.1 and I see 3.1 lucene
> generates lot of small index files while indexing document)
>
> Also I have a problem where my lucene index files sometimes gets deleted
> from the index folder. I am not sure what code snippet is causing the
> existing index files to accidently get removed.
>
> My indexer runs in a thread loop where it indexes file whenever they are
> available. When no more files are available, indexer thread closes the
> writer and goes to sleep, after specific time, it again creates a new index
> on the same folder and starts indexing new files if any available.
>
> A. Is it a wrong way to index files?
> B. Because I close the index and open it again later, am I seeing my lucene
> index files getting deleted?
>
> Thanks
>
> Saurabh
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Questions on index Writer
Posted by Saurabh Gokhale <sa...@gmail.com>.
Thanks guys, that helped a lot.
Saurabh
On Sat, Jul 16, 2011 at 3:08 PM, Mihai Caraman <ca...@gmail.com>wrote:
> >
> > indexWriter = new IndexWriter(FSDirectory.open(new File(indexDirName)),
> > getAnalyzer(), true, MaxFieldLength.UNLIMITED);
> >
> > Does this statement cleans up existing index files?
>
> yes
>
>
> > If yes, then how do I
> > tackle a scenario where lets say I brought down my application server
> > hosting code to create lucene index and then start it again, this way a
> new
> > JVM will start creating indexWriter instance again which will eventually
> > clean up my existing index files.
> >
>
> indexWriter = new IndexWriter(FSDirectory.open(new File(indexDirName)),
> getAnalyzer(), *false*, MaxFieldLength.UNLIMITED);
>
> Note:
> To avoid using deprecated method use the equivalent code:
>
> Directory idx = new NIOFSDirectory(new File(indexDirName), null);
> IndexWriterConfig iConfig = new
> IndexWriterConfig(Version.LUCENE_31,getAnalyzer());
> iConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
> indexWriter = new IndexWriter(idx, iConfig);
>
Re: Questions on index Writer
Posted by Mihai Caraman <ca...@gmail.com>.
>
> indexWriter = new IndexWriter(FSDirectory.open(new File(indexDirName)),
> getAnalyzer(), true, MaxFieldLength.UNLIMITED);
>
> Does this statement cleans up existing index files?
yes
> If yes, then how do I
> tackle a scenario where lets say I brought down my application server
> hosting code to create lucene index and then start it again, this way a new
> JVM will start creating indexWriter instance again which will eventually
> clean up my existing index files.
>
indexWriter = new IndexWriter(FSDirectory.open(new File(indexDirName)),
getAnalyzer(), *false*, MaxFieldLength.UNLIMITED);
Note:
To avoid using deprecated method use the equivalent code:
Directory idx = new NIOFSDirectory(new File(indexDirName), null);
IndexWriterConfig iConfig = new
IndexWriterConfig(Version.LUCENE_31,getAnalyzer());
iConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
indexWriter = new IndexWriter(idx, iConfig);
Re: Questions on index Writer
Posted by Saurabh Gokhale <sa...@gmail.com>.
Hi Eric
Thanks for the reply.
*Index Code:*
Here is the code snippet that I am using for creating index writer.
indexWriter = new IndexWriter(FSDirectory.open(new File(indexDirName)),
getAnalyzer(), true, MaxFieldLength.UNLIMITED);
Does this statement cleans up existing index files? If yes, then how do I
tackle a scenario where lets say I brought down my application server
hosting code to create lucene index and then start it again, this way a new
JVM will start creating indexWriter instance again which will eventually
clean up my existing index files.
*Code Setup:*
I have a singleton IndexObjectsCreator factory class with indexWriter as an
instance level variable. This factory creates all the objects required like
writer, reader, searcher etc. I create factory instance and indexwriter
instance only once and returns the same instance to the client.
Client can eventually call close on the indexWriter. If it does, then
I create a new indexWriter and return back for the next call for the writer.
My factory instance remains singleton for the duration of the application
run.
*MayBeMerge:*
Yes, as per your suggestion, I will skip this method call, if the segment
merge process operates well within itself automatically then I dont see a
need to call this method as my intetions were to suggest a merge request to
lucene.
thanks again
Saurabh
On Fri, Jul 15, 2011 at 3:03 PM, Erick Erickson <er...@gmail.com>wrote:
> Index files should not be disappearing unless you're using the form
> of opening an indexwriter that creates a new index. We'd need to see
> the code you use top open the IW to provide more help.
>
> If all you're doing is looking at the index directory, segments will
> disappear
> as they are merged so that's also what you might be seeing. Can you give
> us more details?
>
> A good time to commit? Well, that depends. It really depends upon your app.
> Given what you've described, and assuming that you've set reasonable
> numbers for, say, setMaxBufferedDocs, setRAMBufferSizeMB just letting the
> commit happen when you close is probably fine.
>
> What do you hope to accomplish by executing maybeMerge()? Personally I'd
> stay away from that unless you're solving a specific question and just let
> the
> segment merge process operate to handle all that...
>
> Best
> Erick
>
> On Fri, Jul 15, 2011 at 3:03 PM, Saurabh Gokhale
> <sa...@gmail.com> wrote:
> > Hi All,
> >
> > I have following questions about lucene indexWriter. I am using version
> > 3.1.0.
> >
> > While indexing documents,
> > 1. When is the good time to commit changes? (indexWriter.commit) or just
> > close the writer after the indexing is done so that commit automatically
> > happens.
> > 2. When is the good time to merge indexes (indexWriter.maybeMerge()). Is
> it
> > just before committing the changes or after indexing say X number of
> > documents. (I recently upgraded from 2.9.4 to 3.1 and I see 3.1 lucene
> > generates lot of small index files while indexing document)
> >
> > Also I have a problem where my lucene index files sometimes gets deleted
> > from the index folder. I am not sure what code snippet is causing the
> > existing index files to accidently get removed.
> >
> > My indexer runs in a thread loop where it indexes file whenever they are
> > available. When no more files are available, indexer thread closes the
> > writer and goes to sleep, after specific time, it again creates a new
> index
> > on the same folder and starts indexing new files if any available.
> >
> > A. Is it a wrong way to index files?
> > B. Because I close the index and open it again later, am I seeing my
> lucene
> > index files getting deleted?
> >
> > Thanks
> >
> > Saurabh
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Questions on index Writer
Posted by Erick Erickson <er...@gmail.com>.
Index files should not be disappearing unless you're using the form
of opening an indexwriter that creates a new index. We'd need to see
the code you use top open the IW to provide more help.
If all you're doing is looking at the index directory, segments will disappear
as they are merged so that's also what you might be seeing. Can you give
us more details?
A good time to commit? Well, that depends. It really depends upon your app.
Given what you've described, and assuming that you've set reasonable
numbers for, say, setMaxBufferedDocs, setRAMBufferSizeMB just letting the
commit happen when you close is probably fine.
What do you hope to accomplish by executing maybeMerge()? Personally I'd
stay away from that unless you're solving a specific question and just let the
segment merge process operate to handle all that...
Best
Erick
On Fri, Jul 15, 2011 at 3:03 PM, Saurabh Gokhale
<sa...@gmail.com> wrote:
> Hi All,
>
> I have following questions about lucene indexWriter. I am using version
> 3.1.0.
>
> While indexing documents,
> 1. When is the good time to commit changes? (indexWriter.commit) or just
> close the writer after the indexing is done so that commit automatically
> happens.
> 2. When is the good time to merge indexes (indexWriter.maybeMerge()). Is it
> just before committing the changes or after indexing say X number of
> documents. (I recently upgraded from 2.9.4 to 3.1 and I see 3.1 lucene
> generates lot of small index files while indexing document)
>
> Also I have a problem where my lucene index files sometimes gets deleted
> from the index folder. I am not sure what code snippet is causing the
> existing index files to accidently get removed.
>
> My indexer runs in a thread loop where it indexes file whenever they are
> available. When no more files are available, indexer thread closes the
> writer and goes to sleep, after specific time, it again creates a new index
> on the same folder and starts indexing new files if any available.
>
> A. Is it a wrong way to index files?
> B. Because I close the index and open it again later, am I seeing my lucene
> index files getting deleted?
>
> Thanks
>
> Saurabh
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org