You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Carl Austin <Ca...@detica.com> on 2009/07/03 12:12:04 UTC
Insuffient resources during optimize merge
Hi,
I have an issue when optimizing a large index (40GB). Basically the
optimize falls over with an IOException stating there are not enough
system resources, within a RandomAccessFile.readBytes call followed back
to the CompoundFileWriter close method. I assume this is when creating a
new CFS file from the separate other files.
Unfortunately there is no hint at which system resource, and I have no
access to the machine this fails on, otherwise I would run some testing
to see what resource usage is going on.
The process running this, has 1.3GB of memory allowance.
If anyone has any insights they would be much appreciated. Stack trace
included now:
Exception in thread "Lucene Merge Thread #0"
org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
Insufficient system resources exist to complete the requested service
at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Co
ncurrentMergeScheduler.java:309)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
entMergeScheduler.java:286)
Caused by: java.io.IOException: Insufficient system resources exist to
complete the requested service
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(Unknown Source)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
y.java:596)
at
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
java:136)
at
org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.j
ava:228)
at
org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java
:184)
at
org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.j
ava:204)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge
Scheduler.java:205)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
entMergeScheduler.java:260)
Exception in thread "Thread-12957887" java.lang.RuntimeException:
java.io.IOException: background merge hit exception: _cx:c3072736
_d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791 _dn:c4045 _do:c3643
_dp:c3197 into _dq [optimize]
at a.b.c.d.e.f.ImportThread$ImportRunner.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: background merge hit exception:
_cx:c3072736 _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791
_dn:c4045 _do:c3643 _dp:c3197 into _dq [optimize]
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2258)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2203)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2183)
... 2 more
Caused by: java.io.IOException: Insufficient system resources exist to
complete the requested service
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(Unknown Source)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
y.java:596)
at
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
java:136)
at
org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.j
ava:228)
at
org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java
:184)
at
org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.j
ava:204)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
at
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge
Scheduler.java:205)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
entMergeScheduler.java:260)
Thanks
Carl Austin
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Group Limited group of companies.
Detica Group Limited is registered in England under No: 03328242.
Registered offices: Warwick House, PO Box 87, Farnborough Aerospace Centre, Farnborough, Hampshire, GU14 6YU, England.
Re: Insuffient resources during optimize merge
Posted by Manish Joshi <mj...@zedo.com>.
I assume it must be memory related issue.Not sure just a guess.
-Manish
On Fri, Jul 3, 2009 at 3:42 PM, Carl Austin <Ca...@detica.com> wrote:
> Hi,
>
> I have an issue when optimizing a large index (40GB). Basically the
> optimize falls over with an IOException stating there are not enough
> system resources, within a RandomAccessFile.readBytes call followed back
> to the CompoundFileWriter close method. I assume this is when creating a
> new CFS file from the separate other files.
> Unfortunately there is no hint at which system resource, and I have no
> access to the machine this fails on, otherwise I would run some testing
> to see what resource usage is going on.
> The process running this, has 1.3GB of memory allowance.
>
> If anyone has any insights they would be much appreciated. Stack trace
> included now:
>
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
> Insufficient system resources exist to complete the requested service
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Co
> ncurrentMergeScheduler.java:309)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
> entMergeScheduler.java:286)
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(Unknown Source)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:596)
> at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
> at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.j
> ava:228)
> at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java
> :184)
> at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.j
> ava:204)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge
> Scheduler.java:205)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
> entMergeScheduler.java:260)
> Exception in thread "Thread-12957887" java.lang.RuntimeException:
> java.io.IOException: background merge hit exception: _cx:c3072736
> _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791 _dn:c4045 _do:c3643
> _dp:c3197 into _dq [optimize]
> at a.b.c.d.e.f.ImportThread$ImportRunner.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: background merge hit exception:
> _cx:c3072736 _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791
> _dn:c4045 _do:c3643 _dp:c3197 into _dq [optimize]
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2258)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2203)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2183)
> ... 2 more
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(Unknown Source)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:596)
> at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
> at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.j
> ava:228)
> at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java
> :184)
> at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.j
> ava:204)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge
> Scheduler.java:205)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
> entMergeScheduler.java:260)
>
> Thanks
>
> Carl Austin
>
>
>
>
>
> This message should be regarded as confidential. If you have received this
> email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy
> by an authorised signatory. The contents of this email may relate to
> dealings with other companies within the Detica Group Limited group of
> companies.
>
> Detica Group Limited is registered in England under No: 03328242.
>
> Registered offices: Warwick House, PO Box 87, Farnborough Aerospace Centre,
> Farnborough, Hampshire, GU14 6YU, England.
>
>
--
Manish Joshi
Adserving Team
Zedo India
Re: Insuffient resources during optimize merge
Posted by Simon Willnauer <si...@googlemail.com>.
Hey Carl,
do you have any anti-virus running on that machine by any chance?
If so could you try to turn it off and try again.
simon
On Fri, Jul 3, 2009 at 1:53 PM, Carl Austin<Ca...@detica.com> wrote:
> Sorry, I forgot to include that I am using lucene 2.4.0 and Java 1.5
>
> Thanks
>
> Carl
>
> -----Original Message-----
> From: Ian Lea [mailto:ian.lea@gmail.com]
> Sent: 03 July 2009 11:51
> To: java-user@lucene.apache.org
> Subject: Re: Insuffient resources during optimize merge
>
> Hi
>
>
> Google throws up some hits for this error relating to obscure problems on Windows with UNC paths and shared rather than local storage. If that's not relevant it might help to know what OS you are on, and versions of lucene and java.
>
> As a workaround you could perhaps try not using the compound file format.
>
>
>
> --
> Ian.
>
>
>
> On Fri, Jul 3, 2009 at 11:12 AM, Carl Austin<Ca...@detica.com> wrote:
>> Hi,
>>
>> I have an issue when optimizing a large index (40GB). Basically the
>> optimize falls over with an IOException stating there are not enough
>> system resources, within a RandomAccessFile.readBytes call followed
>> back to the CompoundFileWriter close method. I assume this is when
>> creating a new CFS file from the separate other files.
>> Unfortunately there is no hint at which system resource, and I have no
>> access to the machine this fails on, otherwise I would run some
>> testing to see what resource usage is going on.
>> The process running this, has 1.3GB of memory allowance.
>>
>> If anyone has any insights they would be much appreciated. Stack trace
>> included now:
>>
>> Exception in thread "Lucene Merge Thread #0"
>> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
>> Insufficient system resources exist to complete the requested service
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(
>> Co
>> ncurrentMergeScheduler.java:309)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concu
>> rr
>> entMergeScheduler.java:286)
>> Caused by: java.io.IOException: Insufficient system resources exist to
>> complete the requested service
>> at java.io.RandomAccessFile.readBytes(Native Method)
>> at java.io.RandomAccessFile.read(Unknown Source)
>> at
>> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect
>> or
>> y.java:596)
>> at
>> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
>> java:136)
>> at
>> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter
>> .j
>> ava:228)
>> at
>> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.ja
>> va
>> :184)
>> at
>> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger
>> .j
>> ava:204)
>> at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
>> at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMer
>> ge
>> Scheduler.java:205)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concu
>> rr
>> entMergeScheduler.java:260)
>> Exception in thread "Thread-12957887" java.lang.RuntimeException:
>> java.io.IOException: background merge hit exception: _cx:c3072736
>> _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791 _dn:c4045
>> _do:c3643
>> _dp:c3197 into _dq [optimize]
>> at a.b.c.d.e.f.ImportThread$ImportRunner.run(Unknown Source)
>> at java.lang.Thread.run(Unknown Source) Caused by:
>> java.io.IOException: background merge hit exception:
>> _cx:c3072736 _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791
>> _dn:c4045 _do:c3643 _dp:c3197 into _dq [optimize]
>> at
>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2258)
>> at
>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2203)
>> at
>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2183)
>> ... 2 more
>> Caused by: java.io.IOException: Insufficient system resources exist to
>> complete the requested service
>> at java.io.RandomAccessFile.readBytes(Native Method)
>> at java.io.RandomAccessFile.read(Unknown Source)
>> at
>> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect
>> or
>> y.java:596)
>> at
>> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
>> java:136)
>> at
>> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter
>> .j
>> ava:228)
>> at
>> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.ja
>> va
>> :184)
>> at
>> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger
>> .j
>> ava:204)
>> at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
>> at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMer
>> ge
>> Scheduler.java:205)
>> at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concu
>> rr
>> entMergeScheduler.java:260)
>>
>> Thanks
>>
>> Carl Austin
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Group Limited group of companies.
>
> Detica Group Limited is registered in England under No: 03328242.
>
> Registered offices: Warwick House, PO Box 87, Farnborough Aerospace Centre, Farnborough, Hampshire, GU14 6YU, England.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Insuffient resources during optimize merge
Posted by Carl Austin <Ca...@detica.com>.
Thanks Ian,
This is using Windows Server 2000 or 2003, however I believe that the storage is all local. I will of course check, but am pretty sure.
The workaround is also something I may try, so thank you for that.
Carl
-----Original Message-----
From: Ian Lea [mailto:ian.lea@gmail.com]
Sent: 03 July 2009 11:51
To: java-user@lucene.apache.org
Subject: Re: Insuffient resources during optimize merge
Hi
Google throws up some hits for this error relating to obscure problems on Windows with UNC paths and shared rather than local storage. If that's not relevant it might help to know what OS you are on, and versions of lucene and java.
As a workaround you could perhaps try not using the compound file format.
--
Ian.
On Fri, Jul 3, 2009 at 11:12 AM, Carl Austin<Ca...@detica.com> wrote:
> Hi,
>
> I have an issue when optimizing a large index (40GB). Basically the
> optimize falls over with an IOException stating there are not enough
> system resources, within a RandomAccessFile.readBytes call followed
> back to the CompoundFileWriter close method. I assume this is when
> creating a new CFS file from the separate other files.
> Unfortunately there is no hint at which system resource, and I have no
> access to the machine this fails on, otherwise I would run some
> testing to see what resource usage is going on.
> The process running this, has 1.3GB of memory allowance.
>
> If anyone has any insights they would be much appreciated. Stack trace
> included now:
>
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
> Insufficient system resources exist to complete the requested service
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(
> Co
> ncurrentMergeScheduler.java:309)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concu
> rr
> entMergeScheduler.java:286)
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(Unknown Source)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect
> or
> y.java:596)
> at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
> at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter
> .j
> ava:228)
> at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.ja
> va
> :184)
> at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger
> .j
> ava:204)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMer
> ge
> Scheduler.java:205)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concu
> rr
> entMergeScheduler.java:260)
> Exception in thread "Thread-12957887" java.lang.RuntimeException:
> java.io.IOException: background merge hit exception: _cx:c3072736
> _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791 _dn:c4045
> _do:c3643
> _dp:c3197 into _dq [optimize]
> at a.b.c.d.e.f.ImportThread$ImportRunner.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source) Caused by:
> java.io.IOException: background merge hit exception:
> _cx:c3072736 _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791
> _dn:c4045 _do:c3643 _dp:c3197 into _dq [optimize]
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2258)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2203)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2183)
> ... 2 more
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(Unknown Source)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect
> or
> y.java:596)
> at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
> at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter
> .j
> ava:228)
> at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.ja
> va
> :184)
> at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger
> .j
> ava:204)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMer
> ge
> Scheduler.java:205)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concu
> rr
> entMergeScheduler.java:260)
>
> Thanks
>
> Carl Austin
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Group Limited group of companies.
Detica Group Limited is registered in England under No: 03328242.
Registered offices: Warwick House, PO Box 87, Farnborough Aerospace Centre, Farnborough, Hampshire, GU14 6YU, England.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Insuffient resources during optimize merge
Posted by Carl Austin <Ca...@detica.com>.
Sorry, I forgot to include that I am using lucene 2.4.0 and Java 1.5
Thanks
Carl
-----Original Message-----
From: Ian Lea [mailto:ian.lea@gmail.com]
Sent: 03 July 2009 11:51
To: java-user@lucene.apache.org
Subject: Re: Insuffient resources during optimize merge
Hi
Google throws up some hits for this error relating to obscure problems on Windows with UNC paths and shared rather than local storage. If that's not relevant it might help to know what OS you are on, and versions of lucene and java.
As a workaround you could perhaps try not using the compound file format.
--
Ian.
On Fri, Jul 3, 2009 at 11:12 AM, Carl Austin<Ca...@detica.com> wrote:
> Hi,
>
> I have an issue when optimizing a large index (40GB). Basically the
> optimize falls over with an IOException stating there are not enough
> system resources, within a RandomAccessFile.readBytes call followed
> back to the CompoundFileWriter close method. I assume this is when
> creating a new CFS file from the separate other files.
> Unfortunately there is no hint at which system resource, and I have no
> access to the machine this fails on, otherwise I would run some
> testing to see what resource usage is going on.
> The process running this, has 1.3GB of memory allowance.
>
> If anyone has any insights they would be much appreciated. Stack trace
> included now:
>
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
> Insufficient system resources exist to complete the requested service
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(
> Co
> ncurrentMergeScheduler.java:309)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concu
> rr
> entMergeScheduler.java:286)
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(Unknown Source)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect
> or
> y.java:596)
> at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
> at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter
> .j
> ava:228)
> at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.ja
> va
> :184)
> at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger
> .j
> ava:204)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMer
> ge
> Scheduler.java:205)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concu
> rr
> entMergeScheduler.java:260)
> Exception in thread "Thread-12957887" java.lang.RuntimeException:
> java.io.IOException: background merge hit exception: _cx:c3072736
> _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791 _dn:c4045
> _do:c3643
> _dp:c3197 into _dq [optimize]
> at a.b.c.d.e.f.ImportThread$ImportRunner.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source) Caused by:
> java.io.IOException: background merge hit exception:
> _cx:c3072736 _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791
> _dn:c4045 _do:c3643 _dp:c3197 into _dq [optimize]
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2258)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2203)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2183)
> ... 2 more
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(Unknown Source)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirect
> or
> y.java:596)
> at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
> at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter
> .j
> ava:228)
> at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.ja
> va
> :184)
> at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger
> .j
> ava:204)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMer
> ge
> Scheduler.java:205)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concu
> rr
> entMergeScheduler.java:260)
>
> Thanks
>
> Carl Austin
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
This message should be regarded as confidential. If you have received this email in error please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised signatory. The contents of this email may relate to dealings with other companies within the Detica Group Limited group of companies.
Detica Group Limited is registered in England under No: 03328242.
Registered offices: Warwick House, PO Box 87, Farnborough Aerospace Centre, Farnborough, Hampshire, GU14 6YU, England.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Insuffient resources during optimize merge
Posted by Ian Lea <ia...@gmail.com>.
Hi
Google throws up some hits for this error relating to obscure problems
on Windows with UNC paths and shared rather than local storage. If
that's not relevant it might help to know what OS you are on, and
versions of lucene and java.
As a workaround you could perhaps try not using the compound file format.
--
Ian.
On Fri, Jul 3, 2009 at 11:12 AM, Carl Austin<Ca...@detica.com> wrote:
> Hi,
>
> I have an issue when optimizing a large index (40GB). Basically the
> optimize falls over with an IOException stating there are not enough
> system resources, within a RandomAccessFile.readBytes call followed back
> to the CompoundFileWriter close method. I assume this is when creating a
> new CFS file from the separate other files.
> Unfortunately there is no hint at which system resource, and I have no
> access to the machine this fails on, otherwise I would run some testing
> to see what resource usage is going on.
> The process running this, has 1.3GB of memory allowance.
>
> If anyone has any insights they would be much appreciated. Stack trace
> included now:
>
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
> Insufficient system resources exist to complete the requested service
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Co
> ncurrentMergeScheduler.java:309)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
> entMergeScheduler.java:286)
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(Unknown Source)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:596)
> at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
> at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.j
> ava:228)
> at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java
> :184)
> at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.j
> ava:204)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge
> Scheduler.java:205)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
> entMergeScheduler.java:260)
> Exception in thread "Thread-12957887" java.lang.RuntimeException:
> java.io.IOException: background merge hit exception: _cx:c3072736
> _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791 _dn:c4045 _do:c3643
> _dp:c3197 into _dq [optimize]
> at a.b.c.d.e.f.ImportThread$ImportRunner.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: background merge hit exception:
> _cx:c3072736 _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791
> _dn:c4045 _do:c3643 _dp:c3197 into _dq [optimize]
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2258)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2203)
> at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2183)
> ... 2 more
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(Unknown Source)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:596)
> at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
> at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.j
> ava:228)
> at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java
> :184)
> at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.j
> ava:204)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge
> Scheduler.java:205)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
> entMergeScheduler.java:260)
>
> Thanks
>
> Carl Austin
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org