You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by 1world1love <jd...@yahoo.com> on 2008/12/19 05:19:30 UTC
optimize: went from 14488449 to 38449
Ok. This is crazy. I have an index with 14,488,449 docs in it. Today I did a
CheckIndex on it and everything looked fine. I made a copy of the index, ran
a delete on about 1.3 million docs and then did an optimize and now my doc
count is 38449.
The index was originally built with 2.3, but I am now using 2.4.
Of course I have the original index, but what would have cause this?
--
View this message in context: http://www.nabble.com/optimize%3A-went-from-14488449-to-38449-tp21085589p21085589.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: optimize: went from 14488449 to 38449
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK. Thanks for bringing closure here.
Mike
1world1love wrote:
>
>
>
> Michael McCandless-2 wrote:
>>
>>
>> How did you delete the documents? EG, by docID using IndexReader, by
>> Term or Query using IndexWriter?
>>
>> And when you said your previous index had 14488449 docs, was
>> numDocs()
>> or maxDoc()?
>>
>>
>
> I deleted by docid. I got the number by numdocs.
>
> Just to followup, I think that this was caused by some filesystem
> issues. My
> sysadmin said there were numerous nfs errors in the log.
>
> Also, after this happened, I ran a checkindex and got a weird EOF
> exception.
>
> Since then, I made another copy and did a checksum on all of the
> files and
> then did the optimize and it worked fine.
> --
> View this message in context: http://www.nabble.com/optimize%3A-went-from-14488449-to-38449-tp21085589p21134951.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: optimize: went from 14488449 to 38449
Posted by 1world1love <jd...@yahoo.com>.
Michael McCandless-2 wrote:
>
>
> How did you delete the documents? EG, by docID using IndexReader, by
> Term or Query using IndexWriter?
>
> And when you said your previous index had 14488449 docs, was numDocs()
> or maxDoc()?
>
>
I deleted by docid. I got the number by numdocs.
Just to followup, I think that this was caused by some filesystem issues. My
sysadmin said there were numerous nfs errors in the log.
Also, after this happened, I ran a checkindex and got a weird EOF exception.
Since then, I made another copy and did a checksum on all of the files and
then did the optimize and it worked fine.
--
View this message in context: http://www.nabble.com/optimize%3A-went-from-14488449-to-38449-tp21085589p21134951.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: optimize: went from 14488449 to 38449
Posted by Michael McCandless <lu...@mikemccandless.com>.
How did you delete the documents? EG, by docID using IndexReader, by
Term or Query using IndexWriter?
And when you said your previous index had 14488449 docs, was numDocs()
or maxDoc()?
Mike
1world1love wrote:
>
>
>
> Ganesh - yahoo wrote:
>>
>> Optimize will remove the deletes and rearrange the document numbers.
>>
>> Have you done some deletes before deleting 1.3 million docs?
>>
>>
>
> No, that is the crazy part. I haven't done anything to this index
> since it
> was first compiled until I did the deletes. That is why I made a
> copy of it
> first because I had never done deletes before and wanted to be sure
> I didn't
> mess anything up.
>
> When I did the deletes, there were a few of the delete files in the
> index,
> but not that many. My index went from 22GB to 77MB.
> --
> View this message in context: http://www.nabble.com/optimize%3A-went-from-14488449-to-38449-tp21085589p21085775.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: optimize: went from 14488449 to 38449
Posted by 1world1love <jd...@yahoo.com>.
Ganesh - yahoo wrote:
>
> Optimize will remove the deletes and rearrange the document numbers.
>
> Have you done some deletes before deleting 1.3 million docs?
>
>
No, that is the crazy part. I haven't done anything to this index since it
was first compiled until I did the deletes. That is why I made a copy of it
first because I had never done deletes before and wanted to be sure I didn't
mess anything up.
When I did the deletes, there were a few of the delete files in the index,
but not that many. My index went from 22GB to 77MB.
--
View this message in context: http://www.nabble.com/optimize%3A-went-from-14488449-to-38449-tp21085589p21085775.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: optimize: went from 14488449 to 38449
Posted by Ganesh <em...@yahoo.co.in>.
Optimize will remove the deletes and rearrange the document numbers.
Have you done some deletes before deleting 1.3 million docs?
Regards
Ganesh
----- Original Message -----
From: "1world1love" <jd...@yahoo.com>
To: <ja...@lucene.apache.org>
Sent: Friday, December 19, 2008 9:49 AM
Subject: optimize: went from 14488449 to 38449
>
> Ok. This is crazy. I have an index with 14,488,449 docs in it. Today I did
> a
> CheckIndex on it and everything looked fine. I made a copy of the index,
> ran
> a delete on about 1.3 million docs and then did an optimize and now my doc
> count is 38449.
>
> The index was originally built with 2.3, but I am now using 2.4.
>
> Of course I have the original index, but what would have cause this?
> --
> View this message in context:
> http://www.nabble.com/optimize%3A-went-from-14488449-to-38449-tp21085589p21085589.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
Send instant messages to your online friends http://in.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org