You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Yang <te...@gmail.com> on 2012/06/13 23:15:08 UTC

OutOfMemoryError when opening the index ?

I got the OutOfMemoryError when I tried to open an Lucene index.

it's very weird since this is only seen when I run this inside an Apache
PIG LATIN script on a particular hadoop cluster of ours,
and it runs fine on an old cluster; also it runs fine if I run the java
code outside of PIG.


as I understand, OutOfMemoryError is not out of heap, so does this mean
that mmap() failed?
I checked the ulimit of the user account running this, it seems to be all
unlimited


any clues?

Thanks
yang

Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
        at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:265)
        at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
        at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:86)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
        at
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:113)
        at
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:29)
        at
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
        at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
        at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
        at org.apache.lucene.index.IndexReader.open(IndexReader.java:462)
        at org.apache.lucene.index.IndexReader.open(IndexReader.java:308)
        at
cgm.places.matcher.ListingSearcher.createIndex(ListingSearcher.java:45)
        at
cgm.places.matcher.ListingSearcher.<init>(ListingSearcher.java:38)
        at
cgm.places.matcher.SearchListingUdf.<init>(SearchListingUdf.java:34)
        ... 83 more
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
        ... 98 more

Re: OutOfMemoryError when opening the index ?

Posted by Yang <te...@gmail.com>.
ok, found it:

we are using Cloudera CDHu3u, they change the ulimit for child jobs.
but I still don't know how to change their default settings yet


On Wed, Jun 13, 2012 at 2:15 PM, Yang <te...@gmail.com> wrote:

> I got the OutOfMemoryError when I tried to open an Lucene index.
>
> it's very weird since this is only seen when I run this inside an Apache
> PIG LATIN script on a particular hadoop cluster of ours,
> and it runs fine on an old cluster; also it runs fine if I run the java
> code outside of PIG.
>
>
> as I understand, OutOfMemoryError is not out of heap, so does this mean
> that mmap() failed?
> I checked the ulimit of the user account running this, it seems to be all
> unlimited
>
>
> any clues?
>
> Thanks
> yang
>
> Caused by: java.io.IOException: Map failed
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>         at
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:265)
>         at
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
>         at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:86)
>         at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
>         at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
>         at
> org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:113)
>         at
> org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:29)
>         at
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
>         at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
>         at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
>         at org.apache.lucene.index.IndexReader.open(IndexReader.java:462)
>         at org.apache.lucene.index.IndexReader.open(IndexReader.java:308)
>         at
> cgm.places.matcher.ListingSearcher.createIndex(ListingSearcher.java:45)
>         at
> cgm.places.matcher.ListingSearcher.<init>(ListingSearcher.java:38)
>         at
> cgm.places.matcher.SearchListingUdf.<init>(SearchListingUdf.java:34)
>         ... 83 more
> Caused by: java.lang.OutOfMemoryError: Map failed
>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>         ... 98 more
>
>