You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Phillip Farber <pf...@umich.edu> on 2009/10/06 13:58:10 UTC
solr optimize - no space left on device
I am attempting to optimize a large shard on solr 1.4 and repeatedly get
java.io.IOException: No space left on device. The shard, after a final
commit before optimize, shows a size of about 192GB on a 400GB volume.
I have successfully optimized 2 other shards that were similarly large
without this problem on identical hardware boxes.
Before the optimize I see:
% df -B1 .
Filesystem 1B-blocks Used Available Use% Mounted on
/dev/mapper/internal-solr--build--2
435440427008 205681356800 225335255040 48%
/l/solrs/build-2
slurm-4:/l/solrs/build-2/data/index % du -B1
205441486848 .
There's a slight discrepancy between the du and df which appears to be
orphaned inodes. But the du says there should be enough space to handle
the doubling in size during optimization. However, for the second time
we run out of space and the du and df are wildly different at that point
and the volume is at 100%
% df -B1 .
Filesystem 1B-blocks Used Available Use% Mounted on
/dev/mapper/internal-solr--build--2
435440427008 430985760768 30851072 100%
/l/solrs/build-2
slurm-4:/l/solrs/build-2/data/index % du -B1
252552298496 .
At this point it appears orphaned inodes are consuming space and not
being freed-up. Any clue as to whether this is a lucene bug a solr bug
or some other problem. Error traces follow.
Thanks!
Phil
---
Oct 6, 2009 2:12:37 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {} 0 9110523
Oct 6, 2009 2:12:37 AM org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: background merge hit exception:
_ojl:C151080 _169w:C141302 _1j36:C80405 _1j35:C2043 _1j34:C192 into
_1j37 [optimize]
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2737)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2658)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:401)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:719)
at
org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:96)
at
org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:85)
at
org.apache.lucene.store.BufferedIndexOutput.seek(BufferedIndexOutput.java:124)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.seek(FSDirectory.java:744)
at
org.apache.lucene.index.TermInfosWriter.close(TermInfosWriter.java:220)
at
org.apache.lucene.index.FormatPostingsFieldsWriter.finish(FormatPostingsFieldsWriter.java:70)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:493)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4838)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4429)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
Oct 6, 2009 2:12:37 AM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/build-2 path=/update params={} status=500 QTime=9110523
Re: Dynamically compute document scores...
Posted by Chris Hostetter <ho...@fucit.org>.
: References: <4A...@umich.edu>
: <69...@mail.gmail.com>
: <4A...@umich.edu>
: <Pi...@radix.cryptio.net>
: <4A...@umich.edu> <SN...@phx.gbl>
: <SN...@phx.gbl>
: <87...@mail.gmail.com>
: <SN...@phx.gbl>
: In-Reply-To: <SN...@phx.gbl>
: Subject: Dynamically compute document scores...
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/Thread_hijacking
-Hoss
Re: Dynamically compute document scores...
Posted by Avlesh Singh <av...@gmail.com>.
Options -
1. Can you pre-compute your "business logic" score at index time? If yes,
then this value can be stored in some field and you can use function queries
to use this data plus the score to return a value which you can sort upon.
2. Take a look at -
http://lucene.apache.org/java/2_2_0/api/org/apache/lucene/search/Similarity.html.
Custom similarity implementations can be hooked up into Solr easily.
Cheers
Avlesh
On Tue, Oct 13, 2009 at 9:05 PM, William Pierce <ev...@hotmail.com>wrote:
> Folks:
>
> During query time, I want to dynamically compute a document score as
> follows:
>
> a) Take the SOLR score for the document -- call it S.
> b) Lookup the "business logic" score for this document. Call it L.
> c) Compute a new score T = func(S, L)
> d) Return the documents sorted by T.
>
> I have looked at function queries but in my limited/quick review of it, I
> could not see a quick way of doing this.
>
> Is this possible?
>
> Thanks,
>
> - Bill
>
>
>
Dynamically compute document scores...
Posted by William Pierce <ev...@hotmail.com>.
Folks:
During query time, I want to dynamically compute a document score as
follows:
a) Take the SOLR score for the document -- call it S.
b) Lookup the "business logic" score for this document. Call it L.
c) Compute a new score T = func(S, L)
d) Return the documents sorted by T.
I have looked at function queries but in my limited/quick review of it, I
could not see a quick way of doing this.
Is this possible?
Thanks,
- Bill
Re: Tips on speeding up indexing needed...
Posted by William Pierce <ev...@hotmail.com>.
Thanks, Lance. I already commit at the end. I will take a look at the data
import handler. Thanks again!
-- Bill
--------------------------------------------------
From: "Lance Norskog" <go...@gmail.com>
Sent: Saturday, October 10, 2009 7:58 PM
To: <so...@lucene.apache.org>
Subject: Re: Tips on speeding up indexing needed...
> A few things off the bat:
> 1) do not commit until the end.
> 2) use the DataImportHandler - it runs inside Solr and reads the
> database. This cuts out the HTTP transfer/XML xlation overheads.
> 3) examine your schema. Some of the text analyzers are quite slow.
>
> Solr tips:
> http://wiki.apache.org/solr/SolrPerformanceFactors
>
> Lucene tips:
> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
>
> And, what you don't want to hear: for jobs like this, Solr/Lucene is
> disk-bound. The Windows NTFS file system is much slower than what is
> available for Linux or the Mac, and these numbers are for those
> machines.
>
> Good luck!
>
> Lance Norskog
>
>
> On Sat, Oct 10, 2009 at 5:57 PM, William Pierce <ev...@hotmail.com>
> wrote:
>> Oh and one more thing...For historical reasons our apps run using msft
>> technologies, so using SolrJ would be next to impossible at the present
>> time....
>>
>> Thanks in advance for your help!
>>
>> -- Bill
>>
>> --------------------------------------------------
>> From: "William Pierce" <ev...@hotmail.com>
>> Sent: Saturday, October 10, 2009 5:47 PM
>> To: <so...@lucene.apache.org>
>> Subject: Tips on speeding up indexing needed...
>>
>>> Folks:
>>>
>>> I have a corpus of approx 6 M documents each of approx 4K bytes.
>>> Currently, the way indexing is set up I read documents from a database
>>> and
>>> issue solr post requests in batches (batches are set up so that the
>>> maxPostSize of tomcat which is set to 2MB is adhered to). This means
>>> that
>>> in each batch we write approx 600 or so documents to SOLR. What I am
>>> seeing
>>> is that I am able to push about 2500 docs per minute or approx 40 or so
>>> per
>>> second.
>>>
>>> I saw in Erik's talk on Friday that speeds of 250 docs/sec to 25000
>>> docs/sec have been achieved. Needless to say I am sure that performance
>>> numbers vary widely and are dependent on the domain, machine
>>> configurations,
>>> etc.
>>>
>>> I am running on Windows 2003 server, with 4 GB RAM, dual core xeon.
>>>
>>> Any tips on what I can do to speed this up?
>>>
>>> Thanks,
>>>
>>> Bill
>>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: Tips on speeding up indexing needed...
Posted by Lance Norskog <go...@gmail.com>.
A few things off the bat:
1) do not commit until the end.
2) use the DataImportHandler - it runs inside Solr and reads the
database. This cuts out the HTTP transfer/XML xlation overheads.
3) examine your schema. Some of the text analyzers are quite slow.
Solr tips:
http://wiki.apache.org/solr/SolrPerformanceFactors
Lucene tips:
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
And, what you don't want to hear: for jobs like this, Solr/Lucene is
disk-bound. The Windows NTFS file system is much slower than what is
available for Linux or the Mac, and these numbers are for those
machines.
Good luck!
Lance Norskog
On Sat, Oct 10, 2009 at 5:57 PM, William Pierce <ev...@hotmail.com> wrote:
> Oh and one more thing...For historical reasons our apps run using msft
> technologies, so using SolrJ would be next to impossible at the present
> time....
>
> Thanks in advance for your help!
>
> -- Bill
>
> --------------------------------------------------
> From: "William Pierce" <ev...@hotmail.com>
> Sent: Saturday, October 10, 2009 5:47 PM
> To: <so...@lucene.apache.org>
> Subject: Tips on speeding up indexing needed...
>
>> Folks:
>>
>> I have a corpus of approx 6 M documents each of approx 4K bytes.
>> Currently, the way indexing is set up I read documents from a database and
>> issue solr post requests in batches (batches are set up so that the
>> maxPostSize of tomcat which is set to 2MB is adhered to). This means that
>> in each batch we write approx 600 or so documents to SOLR. What I am seeing
>> is that I am able to push about 2500 docs per minute or approx 40 or so per
>> second.
>>
>> I saw in Erik's talk on Friday that speeds of 250 docs/sec to 25000
>> docs/sec have been achieved. Needless to say I am sure that performance
>> numbers vary widely and are dependent on the domain, machine configurations,
>> etc.
>>
>> I am running on Windows 2003 server, with 4 GB RAM, dual core xeon.
>>
>> Any tips on what I can do to speed this up?
>>
>> Thanks,
>>
>> Bill
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Tips on speeding up indexing needed...
Posted by William Pierce <ev...@hotmail.com>.
Oh and one more thing...For historical reasons our apps run using msft
technologies, so using SolrJ would be next to impossible at the present
time....
Thanks in advance for your help!
-- Bill
--------------------------------------------------
From: "William Pierce" <ev...@hotmail.com>
Sent: Saturday, October 10, 2009 5:47 PM
To: <so...@lucene.apache.org>
Subject: Tips on speeding up indexing needed...
> Folks:
>
> I have a corpus of approx 6 M documents each of approx 4K bytes.
> Currently, the way indexing is set up I read documents from a database and
> issue solr post requests in batches (batches are set up so that the
> maxPostSize of tomcat which is set to 2MB is adhered to). This means that
> in each batch we write approx 600 or so documents to SOLR. What I am
> seeing is that I am able to push about 2500 docs per minute or approx 40
> or so per second.
>
> I saw in Erik's talk on Friday that speeds of 250 docs/sec to 25000
> docs/sec have been achieved. Needless to say I am sure that performance
> numbers vary widely and are dependent on the domain, machine
> configurations, etc.
>
> I am running on Windows 2003 server, with 4 GB RAM, dual core xeon.
>
> Any tips on what I can do to speed this up?
>
> Thanks,
>
> Bill
>
Re: Tips on speeding up indexing needed...
Posted by William Pierce <ev...@hotmail.com>.
Oops....My bad! I didn't realize that by changing the subject line I was
still "part" of the thread whose subject I changed!
Sorry folks! Thanks, Hoss for pointing this out!
- Bill
--------------------------------------------------
From: "Chris Hostetter" <ho...@fucit.org>
Sent: Tuesday, October 13, 2009 11:07 AM
To: <so...@lucene.apache.org>
Subject: Re: Tips on speeding up indexing needed...
>
> : References: <4A...@umich.edu>
> : <69...@mail.gmail.com>
> : <4A...@umich.edu>
> : <Pi...@radix.cryptio.net>
> : <4A...@umich.edu>
> : In-Reply-To: <4A...@umich.edu>
> : Subject: Tips on speeding up indexing needed...
>
> http://people.apache.org/~hossman/#threadhijack
> Thread Hijacking on Mailing Lists
>
> When starting a new discussion on a mailing list, please do not reply to
> an existing message, instead start a fresh email. Even if you change the
> subject line of your email, other mail headers still track which thread
> you replied to and your question is "hidden" in that thread and gets less
> attention. It makes following discussions in the mailing list archives
> particularly difficult.
> See Also: http://en.wikipedia.org/wiki/Thread_hijacking
>
>
>
>
>
> -Hoss
>
>
Re: Tips on speeding up indexing needed...
Posted by Chris Hostetter <ho...@fucit.org>.
: References: <4A...@umich.edu>
: <69...@mail.gmail.com>
: <4A...@umich.edu>
: <Pi...@radix.cryptio.net>
: <4A...@umich.edu>
: In-Reply-To: <4A...@umich.edu>
: Subject: Tips on speeding up indexing needed...
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/Thread_hijacking
-Hoss
Tips on speeding up indexing needed...
Posted by William Pierce <ev...@hotmail.com>.
Folks:
I have a corpus of approx 6 M documents each of approx 4K bytes.
Currently, the way indexing is set up I read documents from a database and
issue solr post requests in batches (batches are set up so that the
maxPostSize of tomcat which is set to 2MB is adhered to). This means that
in each batch we write approx 600 or so documents to SOLR. What I am seeing
is that I am able to push about 2500 docs per minute or approx 40 or so per
second.
I saw in Erik's talk on Friday that speeds of 250 docs/sec to 25000 docs/sec
have been achieved. Needless to say I am sure that performance numbers vary
widely and are dependent on the domain, machine configurations, etc.
I am running on Windows 2003 server, with 4 GB RAM, dual core xeon.
Any tips on what I can do to speed this up?
Thanks,
Bill
Re: solr optimize - no space left on device
Posted by Phillip Farber <pf...@umich.edu>.
Thanks Hoss. Yes, in a separate thread on the list I reported that
doing a multi-stage optimize worked around the out of space problem. We
use mergefactor=10, maxSegments = 16, 8, 4, 2, 1 iteratively starting at
the closest power of two below the number of segments to merge. Works
nicely so far. And compoundFile=false.
Regards,
Phil
Chris Hostetter wrote:
>
>
> : That theory did not work because the error log showed that solr was trying to
> : merge into the _1j37 segment files showing as deleted in the lsof above when
> : it ran out of space so those are a symptom not a cause of the lost space:
>
> Right, you have to keep in mind Solr is always maintaining an open
> searcher, so even as files get deleted because of a merge, the disk space
> is still being used because Solr is maintaining open file handles to those
> deleted files (for searching) so the os can't free up that space.
>
> have you tried doing iterative partial optimizes so Solr can re-open the
> index more frequently (and release some of those file handles sooner) ?
> Also: what does your "useCompoundFile" option look like in solrconfig.xml,
> becuase if it's "true" then you (aparently) need 3X not 2X for an
> optimize...
>
> http://www.nabble.com/How-much-disk-space-does-optimize-really-take-to25790344.html#a25792748
>
>
> -Hoss
>
Re: solr optimize - no space left on device
Posted by Chris Hostetter <ho...@fucit.org>.
: That theory did not work because the error log showed that solr was trying to
: merge into the _1j37 segment files showing as deleted in the lsof above when
: it ran out of space so those are a symptom not a cause of the lost space:
Right, you have to keep in mind Solr is always maintaining an open
searcher, so even as files get deleted because of a merge, the disk space
is still being used because Solr is maintaining open file handles to those
deleted files (for searching) so the os can't free up that space.
have you tried doing iterative partial optimizes so Solr can re-open the
index more frequently (and release some of those file handles sooner) ?
Also: what does your "useCompoundFile" option look like in solrconfig.xml,
becuase if it's "true" then you (aparently) need 3X not 2X for an
optimize...
http://www.nabble.com/How-much-disk-space-does-optimize-really-take-to25790344.html#a25792748
-Hoss
Re: solr optimize - no space left on device
Posted by Phillip Farber <pf...@umich.edu>.
All,
We're puzzled why we're still unable to optimize a 192GB index on a LVM
volume that has 406GB available. We are not using Solr distribution.
There is no snapshooter in the picture. We run out of disk capacity with
a df showing 100% but a du showing just 379GB of files.
Restarting tomcat causes space to be recovered and many segments to be
deleted leaving just 3 from the original 33. Issuing another optimize at
that point causes solr to run for a while and then show no further
activity (cpu,memory consumption) in jconsole. The 3 segments do not
merge into one.
% df -h .
/dev/mapper/internal-solr--build--2
size used avail
406G 402G 30M 100% /l/solrs/build-2
Also suspicious is the 406G vs. 402G vs. 30M for size vs. used vs. avail
Yesterday, after our 2nd try, lsof showed several "deleted" files that
were still open that apparently were consuming space almost 134GB.
jsvc 8381 tomcat 377u REG 253,6 13369098240
1982471 /l/solrs/build-2/data/index/_1j37.tis (deleted)
jsvc 8381 tomcat 378u REG 253,6 184778752
1982472 /l/solrs/build-2/data/index/_1j37.tii (deleted)
jsvc 8381 tomcat 379u REG 253,6 34053685248
1982473 /l/solrs/build-2/data/index/_1j37.frq (deleted)
jsvc 8381 tomcat 380u REG 253,6 130411978752
1982474 /l/solrs/build-2/data/index/_1j37.prx (deleted)
That theory did not work because the error log showed that solr was
trying to merge into the _1j37 segment files showing as deleted in the
lsof above when it ran out of space so those are a symptom not a cause
of the lost space:
SEVERE: java.io.IOException: background merge hit exception:
_ojl:C151080 _169w:C141302 _1j36:C80405 _1j35:C2043 _1j34:C192 into
_1j37 [optimize]]]]e]: java.io.IOException: background merge hit
exception: _ojl:C151080 _169w:C141302 _1j36:C80405 _1j35:C2043
_1j34:C192 into _1j37 [
We restored the pre-optimized index again, restarted tomcat and tried to
optimize using SerialMergePolicy instead of the default
ConcurrentMergePolicy under the theory that concurrent merges could
somehow take more than 2X disk space.
The optimize failed again with "out of space" error. This time there
where no "deleted" files in the lsof output.
This is one shard out of 10. A couple the shards were around 192GB and
merged successfully. Any suggestions on how to debug this would be
greatly appreciated.
Thanks!
Phil
hathitrust.org
University of Michigan
Shalin Shekhar Mangar wrote:
> Not sure but a quick search turned up:
> http://www.walkernews.net/2007/07/13/df-and-du-command-show-different-used-disk-space/
>
> Using upto 2x the index size can happen. Also check if there is a
> snapshooter script running through cron which is making hard links to files
> while a merge is in progress.
>
> Do let us know if you make any progress. This is interesting.
>
> On Tue, Oct 6, 2009 at 5:28 PM, Phillip Farber <pf...@umich.edu> wrote:
>
>> I am attempting to optimize a large shard on solr 1.4 and repeatedly get
>> java.io.IOException: No space left on device. The shard, after a final
>> commit before optimize, shows a size of about 192GB on a 400GB volume. I
>> have successfully optimized 2 other shards that were similarly large without
>> this problem on identical hardware boxes.
>>
>> Before the optimize I see:
>>
>> % df -B1 .
>> Filesystem 1B-blocks Used Available Use% Mounted on
>> /dev/mapper/internal-solr--build--2
>> 435440427008 205681356800 225335255040 48%
>> /l/solrs/build-2
>>
>> slurm-4:/l/solrs/build-2/data/index % du -B1
>> 205441486848 .
>>
>> There's a slight discrepancy between the du and df which appears to be
>> orphaned inodes. But the du says there should be enough space to handle the
>> doubling in size during optimization. However, for the second time we run
>> out of space and the du and df are wildly different at that point and the
>> volume is at 100%
>>
>>
>> % df -B1 .
>>
>> Filesystem 1B-blocks Used Available Use% Mounted on
>> /dev/mapper/internal-solr--build--2
>> 435440427008 430985760768 30851072 100%
>> /l/solrs/build-2
>>
>> slurm-4:/l/solrs/build-2/data/index % du -B1
>> 252552298496 .
>>
>> At this point it appears orphaned inodes are consuming space and not being
>> freed-up. Any clue as to whether this is a lucene bug a solr bug or some
>> other problem. Error traces follow.
>>
>> Thanks!
>>
>> Phil
>>
>> ---
>>
>> Oct 6, 2009 2:12:37 AM org.apache.solr.update.processor.LogUpdateProcessor
>> finish
>> INFO: {} 0 9110523
>> Oct 6, 2009 2:12:37 AM org.apache.solr.common.SolrException log
>> SEVERE: java.io.IOException: background merge hit exception: _ojl:C151080
>> _169w:C141302 _1j36:C80405 _1j35:C2043 _1j34:C192 into _1j37 [optimize]
>> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2737)
>> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2658)
>> at
>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:401)
>> at
>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>> at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
>> at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
>> at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>> at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
>> at
>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>> at
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>> at
>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>> at
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.io.IOException: No space left on device
>> at java.io.RandomAccessFile.writeBytes(Native Method)
>> at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
>> at
>> org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:719)
>> at
>> org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:96)
>> at
>> org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:85)
>> at
>> org.apache.lucene.store.BufferedIndexOutput.seek(BufferedIndexOutput.java:124)
>> at
>> org.apache.lucene.store.FSDirectory$FSIndexOutput.seek(FSDirectory.java:744)
>> at
>> org.apache.lucene.index.TermInfosWriter.close(TermInfosWriter.java:220)
>> at
>> org.apache.lucene.index.FormatPostingsFieldsWriter.finish(FormatPostingsFieldsWriter.java:70)
>> at
>> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:493)
>> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
>> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4838)
>> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4429)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>>
>> Oct 6, 2009 2:12:37 AM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/build-2 path=/update params={} status=500 QTime=9110523
>>
>>
>>
>
>
Re: solr optimize - no space left on device
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Not sure but a quick search turned up:
http://www.walkernews.net/2007/07/13/df-and-du-command-show-different-used-disk-space/
Using upto 2x the index size can happen. Also check if there is a
snapshooter script running through cron which is making hard links to files
while a merge is in progress.
Do let us know if you make any progress. This is interesting.
On Tue, Oct 6, 2009 at 5:28 PM, Phillip Farber <pf...@umich.edu> wrote:
> I am attempting to optimize a large shard on solr 1.4 and repeatedly get
> java.io.IOException: No space left on device. The shard, after a final
> commit before optimize, shows a size of about 192GB on a 400GB volume. I
> have successfully optimized 2 other shards that were similarly large without
> this problem on identical hardware boxes.
>
> Before the optimize I see:
>
> % df -B1 .
> Filesystem 1B-blocks Used Available Use% Mounted on
> /dev/mapper/internal-solr--build--2
> 435440427008 205681356800 225335255040 48%
> /l/solrs/build-2
>
> slurm-4:/l/solrs/build-2/data/index % du -B1
> 205441486848 .
>
> There's a slight discrepancy between the du and df which appears to be
> orphaned inodes. But the du says there should be enough space to handle the
> doubling in size during optimization. However, for the second time we run
> out of space and the du and df are wildly different at that point and the
> volume is at 100%
>
>
> % df -B1 .
>
> Filesystem 1B-blocks Used Available Use% Mounted on
> /dev/mapper/internal-solr--build--2
> 435440427008 430985760768 30851072 100%
> /l/solrs/build-2
>
> slurm-4:/l/solrs/build-2/data/index % du -B1
> 252552298496 .
>
> At this point it appears orphaned inodes are consuming space and not being
> freed-up. Any clue as to whether this is a lucene bug a solr bug or some
> other problem. Error traces follow.
>
> Thanks!
>
> Phil
>
> ---
>
> Oct 6, 2009 2:12:37 AM org.apache.solr.update.processor.LogUpdateProcessor
> finish
> INFO: {} 0 9110523
> Oct 6, 2009 2:12:37 AM org.apache.solr.common.SolrException log
> SEVERE: java.io.IOException: background merge hit exception: _ojl:C151080
> _169w:C141302 _1j36:C80405 _1j35:C2043 _1j34:C192 into _1j37 [optimize]
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2737)
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2658)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:401)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
> at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:168)
> at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:54)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1299)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:548)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: No space left on device
> at java.io.RandomAccessFile.writeBytes(Native Method)
> at java.io.RandomAccessFile.write(RandomAccessFile.java:466)
> at
> org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:719)
> at
> org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:96)
> at
> org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:85)
> at
> org.apache.lucene.store.BufferedIndexOutput.seek(BufferedIndexOutput.java:124)
> at
> org.apache.lucene.store.FSDirectory$FSIndexOutput.seek(FSDirectory.java:744)
> at
> org.apache.lucene.index.TermInfosWriter.close(TermInfosWriter.java:220)
> at
> org.apache.lucene.index.FormatPostingsFieldsWriter.finish(FormatPostingsFieldsWriter.java:70)
> at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:493)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:140)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4838)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4429)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>
> Oct 6, 2009 2:12:37 AM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/build-2 path=/update params={} status=500 QTime=9110523
>
>
>
--
Regards,
Shalin Shekhar Mangar.