You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Fa...@emc.com on 2009/09/01 12:03:32 UTC
RE: exception to open a large index Insufficient system resources exist
We are running on Windows 2003 Enterprise Edition with NTFS file system on a local disc. JDK version is 1.5.0.12.
The problem was discussed before and there is no clear solution confirmed.
Thanks.
-----Original Message-----
From: Danil ŢORIN [mailto:torindan@gmail.com]
Sent: Tuesday, September 01, 2009 5:56 PM
To: java-user@lucene.apache.org
Subject: Re: exception to open a large index Insufficient system resources exist
There should be no problem with large segments.
Please describe OS, FileSystem and JDK you are running on.
There might be some problems with file >2Gb on Win32/FAT, or in some
ancient Linuxes.
On Tue, Sep 1, 2009 at 12:37, <Fa...@emc.com> wrote:
> I met a problem to open an index bigger than 8GB and the following
> exception was thrown. There is a segment which is bigger than 4GB
> already. After searching internet, it is said that not using compound
> index may solve the problem.
>
> The same exception was thrown when merging with another index happens.
> If the problem is caused by the big segment, we can adjust Lucene
> parameters to control the segment size.
> Anyone knows what's the cause for this exception? As reproducing the
> problem will take a long time, your idea will be save us a lot of
> effort.
>
> java.io.IOException: Insufficient system resources exist to complete the
> requested service
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
> at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:596)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> a:157)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> ava:38)
> at
> org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
> at
> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.jav
> a:73)
> at
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
> at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:228)
> at
> org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.jav
> a:55)
> at
> org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexRead
> er.java:104)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.j
> ava:653)
> at
> org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.j
> ava:115)
> at
> org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> at
> org.apache.lucene.index.IndexReader.open(IndexReader.java:227)
>
> Thanks,
> -Li
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: exception to open a large index Insufficient system resources exist
Posted by Fa...@emc.com.
32 bit JVM, 1.3G allocated heap size, Lucene 2.4.1.
In my option, this exception should not be caused by out of memory or out of system file handle because different exception should be thrown for these two cases.
Any hint?
Thanks,
Fang, Li
-----Original Message-----
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Tuesday, September 01, 2009 6:14 PM
To: java-user@lucene.apache.org
Subject: RE: exception to open a large index Insufficient system resources exist
Which Lucene version, 64 bit JVM?
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Fang_Li@emc.com [mailto:Fang_Li@emc.com]
> Sent: Tuesday, September 01, 2009 12:04 PM
> To: java-user@lucene.apache.org
> Subject: RE: exception to open a large index Insufficient system resources
> exist
>
> We are running on Windows 2003 Enterprise Edition with NTFS file system on
> a local disc. JDK version is 1.5.0.12.
>
> The problem was discussed before and there is no clear solution confirmed.
>
> Thanks.
>
> -----Original Message-----
> From: Danil ŢORIN [mailto:torindan@gmail.com]
> Sent: Tuesday, September 01, 2009 5:56 PM
> To: java-user@lucene.apache.org
> Subject: Re: exception to open a large index Insufficient system resources
> exist
>
> There should be no problem with large segments.
> Please describe OS, FileSystem and JDK you are running on.
>
> There might be some problems with file >2Gb on Win32/FAT, or in some
> ancient Linuxes.
>
> On Tue, Sep 1, 2009 at 12:37, <Fa...@emc.com> wrote:
> > I met a problem to open an index bigger than 8GB and the following
> > exception was thrown. There is a segment which is bigger than 4GB
> > already. After searching internet, it is said that not using compound
> > index may solve the problem.
> >
> > The same exception was thrown when merging with another index happens.
> > If the problem is caused by the big segment, we can adjust Lucene
> > parameters to control the segment size.
> > Anyone knows what's the cause for this exception? As reproducing the
> > problem will take a long time, your idea will be save us a lot of
> > effort.
> >
> > java.io.IOException: Insufficient system resources exist to complete the
> > requested service
> > at java.io.RandomAccessFile.readBytes(Native Method)
> > at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
> > at
> > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> > y.java:596)
> > at
> > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> > a:157)
> > at
> > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> > ava:38)
> > at
> > org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
> > at
> > org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.jav
> > a:73)
> > at
> > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
> > at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> > at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:228)
> > at
> > org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.jav
> > a:55)
> > at
> > org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexRead
> > er.java:104)
> > at
> > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.j
> > ava:653)
> > at
> > org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.j
> > ava:115)
> > at
> > org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> > at
> > org.apache.lucene.index.IndexReader.open(IndexReader.java:227)
> >
> > Thanks,
> > -Li
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: exception to open a large index Insufficient system resources exist
Posted by Uwe Schindler <uw...@thetaphi.de>.
Which Lucene version, 64 bit JVM?
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Fang_Li@emc.com [mailto:Fang_Li@emc.com]
> Sent: Tuesday, September 01, 2009 12:04 PM
> To: java-user@lucene.apache.org
> Subject: RE: exception to open a large index Insufficient system resources
> exist
>
> We are running on Windows 2003 Enterprise Edition with NTFS file system on
> a local disc. JDK version is 1.5.0.12.
>
> The problem was discussed before and there is no clear solution confirmed.
>
> Thanks.
>
> -----Original Message-----
> From: Danil ŢORIN [mailto:torindan@gmail.com]
> Sent: Tuesday, September 01, 2009 5:56 PM
> To: java-user@lucene.apache.org
> Subject: Re: exception to open a large index Insufficient system resources
> exist
>
> There should be no problem with large segments.
> Please describe OS, FileSystem and JDK you are running on.
>
> There might be some problems with file >2Gb on Win32/FAT, or in some
> ancient Linuxes.
>
> On Tue, Sep 1, 2009 at 12:37, <Fa...@emc.com> wrote:
> > I met a problem to open an index bigger than 8GB and the following
> > exception was thrown. There is a segment which is bigger than 4GB
> > already. After searching internet, it is said that not using compound
> > index may solve the problem.
> >
> > The same exception was thrown when merging with another index happens.
> > If the problem is caused by the big segment, we can adjust Lucene
> > parameters to control the segment size.
> > Anyone knows what's the cause for this exception? As reproducing the
> > problem will take a long time, your idea will be save us a lot of
> > effort.
> >
> > java.io.IOException: Insufficient system resources exist to complete the
> > requested service
> > at java.io.RandomAccessFile.readBytes(Native Method)
> > at java.io.RandomAccessFile.read(RandomAccessFile.java:315)
> > at
> > org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> > y.java:596)
> > at
> > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
> > a:157)
> > at
> > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
> > ava:38)
> > at
> > org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:78)
> > at
> > org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.jav
> > a:73)
> > at
> > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
> > at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
> > at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:228)
> > at
> > org.apache.lucene.index.MultiSegmentReader.<init>(MultiSegmentReader.jav
> > a:55)
> > at
> > org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexRead
> > er.java:104)
> > at
> > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.j
> > ava:653)
> > at
> > org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.j
> > ava:115)
> > at
> > org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
> > at
> > org.apache.lucene.index.IndexReader.open(IndexReader.java:227)
> >
> > Thanks,
> > -Li
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org