You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Yang <te...@gmail.com> on 2012/06/13 23:15:08 UTC
OutOfMemoryError when opening the index ?
I got the OutOfMemoryError when I tried to open an Lucene index.
it's very weird since this is only seen when I run this inside an Apache
PIG LATIN script on a particular hadoop cluster of ours,
and it runs fine on an old cluster; also it runs fine if I run the java
code outside of PIG.
as I understand, OutOfMemoryError is not out of heap, so does this mean
that mmap() failed?
I checked the ulimit of the user account running this, it seems to be all
unlimited
any clues?
Thanks
yang
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:265)
at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:86)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
at
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:113)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:29)
at
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:462)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:308)
at
cgm.places.matcher.ListingSearcher.createIndex(ListingSearcher.java:45)
at
cgm.places.matcher.ListingSearcher.<init>(ListingSearcher.java:38)
at
cgm.places.matcher.SearchListingUdf.<init>(SearchListingUdf.java:34)
... 83 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
... 98 more
Re: OutOfMemoryError when opening the index ?
Posted by Yang <te...@gmail.com>.
ok, found it:
we are using Cloudera CDHu3u, they change the ulimit for child jobs.
but I still don't know how to change their default settings yet
On Wed, Jun 13, 2012 at 2:15 PM, Yang <te...@gmail.com> wrote:
> I got the OutOfMemoryError when I tried to open an Lucene index.
>
> it's very weird since this is only seen when I run this inside an Apache
> PIG LATIN script on a particular hadoop cluster of ours,
> and it runs fine on an old cluster; also it runs fine if I run the java
> code outside of PIG.
>
>
> as I understand, OutOfMemoryError is not out of heap, so does this mean
> that mmap() failed?
> I checked the ulimit of the user account running this, it seems to be all
> unlimited
>
>
> any clues?
>
> Thanks
> yang
>
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> at
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:265)
> at
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
> at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:86)
> at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
> at
> org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:113)
> at
> org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:29)
> at
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
> at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:462)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:308)
> at
> cgm.places.matcher.ListingSearcher.createIndex(ListingSearcher.java:45)
> at
> cgm.places.matcher.ListingSearcher.<init>(ListingSearcher.java:38)
> at
> cgm.places.matcher.SearchListingUdf.<init>(SearchListingUdf.java:34)
> ... 83 more
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
> ... 98 more
>
>