You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Yogesh Dabhi <yd...@asite.com> on 2011/06/13 13:14:43 UTC
Slow ness of IndexWriter.close()
Hi,
I try to add and update document in index
At the start its take only 1 to 2 second but after 50 to 60 document add
and update
Its take 40 to 50 second or some time its take more then 1 min
Is there any way to improve performance ?
Please help me
Please check my code
if(documents!=null && documents.size()>0){
IndexWriter writer = null;
boolean createNewIndexDirectory =
!IndexReader.indexExists(FSDirectory.open(indexDir));
int createNewIndexReTries = 0;
Directory directory = FSDirectory.open(indexDir, null);
while(writer==null) {
try{
directory.setLockFactory(new
NativeFSLockFactory(indexDir));
if(analyzer == null){
analyzer = new
StandardAnalyzer(IAsiteIndexConstants.LUCENE_VERSION,AsiteIndexUtil.getS
topWordSet());
}
writer = new IndexWriter(directory, analyzer,
createNewIndexDirectory,IndexWriter.MaxFieldLength.UNLIMITED);
for(Document document : documents){
if(document.get(ICustomFormIndexConstants.INDEX_FIELD_FORM_ID)!=null){
formId =
Integer.parseInt(document.get(ICustomFormIndexConstants.INDEX_FIELD_FORM
_ID));
}
String strValue =
NumericUtils.longToPrefixCoded(Long.parseLong(document.get(ICustomFormIn
dexConstants.INDEX_FIELD_MSG_ID)));
term = new
Term(ICustomFormIndexConstants.INDEX_FIELD_MSG_ID,strValue);
writer.updateDocument(term, document);
}
}catch (LockObtainFailedException ex){
CheckIndex.Status status = new
CheckIndex(directory).checkIndex();
if(status!=null && status.numBadSegments>0){
log.error("Index found corrupted.
status.numBadSegments ::"+status.numBadSegments);
ex.printStackTrace();
throw ex;
}
}finally{
try{
if(writer!=null){
writer.close();
}
}catch(IOException ex){
}
}
}
Thanks & Regards
Yogesh
Re: Slow ness of IndexWriter.close()
Posted by Erick Erickson <er...@gmail.com>.
My first question is "what are you trying to do at a higher level"?
Because asking people to check your code without telling us
what you're trying to accomplish makes it difficult to know what
to look at. You might review:
http://wiki.apache.org/solr/UsingMailingLists
That said, at a guess, your update statement will first delete
all the items in the index where the term matches. Is it possible
that you're deleting/adding the same documents over and
over and over again?
Best
Erick
On Mon, Jun 13, 2011 at 7:14 AM, Yogesh Dabhi <yd...@asite.com> wrote:
>
>
> Hi,
>
> I try to add and update document in index
>
> At the start its take only 1 to 2 second but after 50 to 60 document add
> and update
>
> Its take 40 to 50 second or some time its take more then 1 min
>
>
>
> Is there any way to improve performance ?
>
>
>
> Please help me
>
>
>
> Please check my code
>
>
>
> if(documents!=null && documents.size()>0){
>
> IndexWriter writer = null;
>
> boolean createNewIndexDirectory =
> !IndexReader.indexExists(FSDirectory.open(indexDir));
>
> int createNewIndexReTries = 0;
>
> Directory directory = FSDirectory.open(indexDir, null);
>
> while(writer==null) {
>
> try{
>
> directory.setLockFactory(new
> NativeFSLockFactory(indexDir));
>
>
>
> if(analyzer == null){
>
>
> analyzer = new
> StandardAnalyzer(IAsiteIndexConstants.LUCENE_VERSION,AsiteIndexUtil.getS
> topWordSet());
>
> }
>
> writer = new IndexWriter(directory, analyzer,
> createNewIndexDirectory,IndexWriter.MaxFieldLength.UNLIMITED);
>
> for(Document document : documents){
>
>
> if(document.get(ICustomFormIndexConstants.INDEX_FIELD_FORM_ID)!=null){
>
> formId =
> Integer.parseInt(document.get(ICustomFormIndexConstants.INDEX_FIELD_FORM
> _ID));
>
> }
>
> String strValue =
> NumericUtils.longToPrefixCoded(Long.parseLong(document.get(ICustomFormIn
> dexConstants.INDEX_FIELD_MSG_ID)));
>
> term = new
> Term(ICustomFormIndexConstants.INDEX_FIELD_MSG_ID,strValue);
>
> writer.updateDocument(term, document);
>
> }
>
> }catch (LockObtainFailedException ex){
>
> CheckIndex.Status status = new
> CheckIndex(directory).checkIndex();
>
> if(status!=null && status.numBadSegments>0){
>
> log.error("Index found corrupted.
> status.numBadSegments ::"+status.numBadSegments);
>
> ex.printStackTrace();
>
> throw ex;
>
> }
>
> }finally{
>
> try{
>
> if(writer!=null){
>
> writer.close();
>
> }
>
>
>
> }catch(IOException ex){
>
> }
>
> }
>
>
>
> }
>
>
>
>
> Thanks & Regards
>
> Yogesh
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org