You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Bernd Fehling <be...@uni-bielefeld.de> on 2012/08/15 13:24:39 UTC
howto run CheckIndex on huge index size
I'm trying to run CheckIndex as seperate tool on a large index to get
nice infos about number of terms, number of tokens, ... but always get OOM exception.
Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
Any idea how to use CheckIndex on huge index size?
Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
Segments file=segments_5u numSegments=1 version=3.6.1 format=FORMAT_3_1 [Lucene 3.1+]
1 of 1: name=_2rhe docCount=37821132
compound=false
hasProx=true
numFiles=8
size (MB)=182.992,35
diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen, os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13 14:08:0
6, source=merge, os.arch=amd64, mergeMaxNumSegments=1, java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
no deletions
test: open reader.........FAILED
WARNING: fixIndex() would remove reference to this segment; full exception:
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:271)
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
at org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:83)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
... 8 more
WARNING: 1 broken segments (containing 37821132 documents) detected
WARNING: would write new segments file, and 37821132 documents would be lost, if -fix were specified
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: howto run CheckIndex on huge index size
Posted by Uwe Schindler <uw...@thetaphi.de>.
Problem not fixed! I contacted infra on IRC already.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Wednesday, August 15, 2012 4:26 PM
> To: java-user@lucene.apache.org
> Subject: RE: howto run CheckIndex on huge index size
>
> I hope the problem is fixed now; this mail is just to check! It was hard
to
> unsubscribe because of the strange eMail. Have no idea at all...
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Wednesday, August 15, 2012 3:13 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: howto run CheckIndex on huge index size
> >
> > I got is, too. As a moderator of this list, I will look into finding
> > the
> root cause
> > and forcefully unsubscribe the failing address!
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> > > -----Original Message-----
> > > From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> > > Sent: Wednesday, August 15, 2012 3:04 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: howto run CheckIndex on huge index size
> > >
> > >
> > > I guess that ulimit could be a default setting of XenServer when it
> > > was
> > first time
> > > setup.
> > > We started with about 27G.
> > > I already raised ulimit -n when setting up XenServer because this
> > > was also limited.
> > >
> > > By the way, am I the only one getting this nasty DELIVERY FAILURE
> > > message from one on this list?
> > >
> > > Bernd
> > >
> > >
> > >
> > >
> > > Am 15.08.2012 14:10, schrieb Uwe Schindler:
> > > > So my blog post, last section, helped? I think the ulimits came
> > > > from
> > there.
> > > > What distribution do you use that ulimit was actually limited - or
> > > > was it some sysadmin doing this? :-)
> > > >
> > > > We should maybe refer to this blog post from docs or create a copy
> > > > of the page inside lucene's distribution!
> > > >
> > > > Uwe
> > > >
> > > > -----
> > > > Uwe Schindler
> > > > H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> > > > eMail: uwe@thetaphi.de
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> > > >> Sent: Wednesday, August 15, 2012 2:07 PM
> > > >> To: java-user@lucene.apache.org
> > > >> Subject: Re: howto run CheckIndex on huge index size
> > > >>
> > > >> Hi Uwe,
> > > >>
> > > >> index size is:
> > > >> -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt
> > > >> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
> > > >> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
> > > >> -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq
> > > >> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
> > > >> -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx
> > > >> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
> > > >> -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis
> > > >> -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u
> > > >> -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen
> > > >>
> > > >> Setting ulimit -v and ulimit -m to unlimited solved this and I
> > > >> can run
> > > > CheckIndex
> > > >> without any java options.
> > > >>
> > > >> The setting with ulimit should be pinned on lucene and solr pages!
> > > >>
> > > >> Thanks for help,
> > > >> Bernd
> > > >>
> > > >>
> > > >> Am 15.08.2012 13:34, schrieb Uwe Schindler:
> > > >>> You don't get a heap-related OOM in your stack trace, it is "Map
> > > >>> failed" - caused by MMapDirectory. You don't have enough virtual
> > > >>> memory to map the index into address space. I think your heap is
> > > >>> way too mch (-Xmx25g is way too big for any existing index and
> > > >>> drives GC
> > > > crazy).
> > > >> How big is your index?
> > > >>>
> > > >>> See
> > > >>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64b
> > > >>> it.h
> > > >>> tm
> > > >>> l
> > > >>>
> > > >>> -----
> > > >>> Uwe Schindler
> > > >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> > > >>> eMail: uwe@thetaphi.de
> > > >>>
> > > >>>
> > > >>>> -----Original Message-----
> > > >>>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> > > >>>> Sent: Wednesday, August 15, 2012 1:25 PM
> > > >>>> To: java-user@lucene.apache.org
> > > >>>> Subject: howto run CheckIndex on huge index size
> > > >>>>
> > > >>>>
> > > >>>> I'm trying to run CheckIndex as seperate tool on a large index
> > > >>>> to
> get
> > > >>>> nice
> > > >>> infos
> > > >>>> about number of terms, number of tokens, ... but always get OOM
> > > >> exception.
> > > >>>>
> > > >>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
> > > >>>>
> > > >>>>
> > > >>>> Any idea how to use CheckIndex on huge index size?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
> > > >>>> Segments file=segments_5u numSegments=1 version=3.6.1
> > > >>>> format=FORMAT_3_1 [Lucene 3.1+]
> > > >>>> 1 of 1: name=_2rhe docCount=37821132
> > > >>>> compound=false
> > > >>>> hasProx=true
> > > >>>> numFiles=8
> > > >>>> size (MB)=182.992,35
> > > >>>> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen,
> > > >>>> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr -
> > > >>>> 2012-04-13
> > > >>>> 14:08:0 6, source=merge, os.arch=amd64,
> mergeMaxNumSegments=1,
> > > >>>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
> > > >>>> no deletions
> > > >>>> test: open reader.........FAILED
> > > >>>> WARNING: fixIndex() would remove reference to this segment;
> full
> > > >>>> exception:
> > > >>>> java.io.IOException: Map failed
> > > >>>> at
sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> > > >>>> at
> > > >>>>
> > > >>
> > >
> >
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec
> > > >>>> t
> > > >>>> ory.java:271)
> > > >>>> at
> > > >>>>
> > > >>
> > >
> > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221
> > )
> > > >>>> at
> > > >>>>
> > > >
> > org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:10
> > 4)
> > > >>>> at
> > > >>>>
> > > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.
> > > >>>> jav
> > > >>>> a:83)
> > > >>>> at
> > > >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116
> > > >>> )
> > > >>>> at
> > > >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
> > > >>>> at
> > > >>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:52
> > > >>> 3)
> > > >>>> at
> > > >>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
> > > >>>> Caused by: java.lang.OutOfMemoryError: Map failed
> > > >>>> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> > > >>>> at
sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
> > > >>>> ... 8 more
> > > >>>> WARNING: 1 broken segments (containing 37821132 documents)
> > detected
> > > >>>> WARNING: would write new segments file, and 37821132 documents
> > > would
> > > >>>> be lost, if -fix were specified
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > >>>> For additional commands, e-mail:
> > > >>>> java-user-help@lucene.apache.org
> > > >>>
> > > >>>
> > > >>>
> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > >>> For additional commands, e-mail:
> > > >>> java-user-help@lucene.apache.org
> > > >>>
> > > >>
> > > >> --
> > > >>
> > *************************************************************
> > > >> Bernd Fehling Universitätsbibliothek Bielefeld
> > > >> Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
> > > >> Universitätsstr. 25 und Wissensmanagement
> > > >> 33615 Bielefeld
> > > >> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
> > > >>
> > > >> BASE - Bielefeld Academic Search Engine - www.base-search.net
> > > >>
> > *************************************************************
> > > >>
> > > >> -----------------------------------------------------------------
> > > >> ---- To unsubscribe, e-mail:
> > > >> java-user-unsubscribe@lucene.apache.org
> > > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --- To unsubscribe, e-mail:
> > > > java-user-unsubscribe@lucene.apache.org
> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >
> > >
> > > --
> > > *************************************************************
> > > Bernd Fehling Universitätsbibliothek Bielefeld
> > > Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
> > > Universitätsstr. 25 und Wissensmanagement
> > > 33615 Bielefeld
> > > Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
> > >
> > > BASE - Bielefeld Academic Search Engine - www.base-search.net
> > > *************************************************************
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: howto run CheckIndex on huge index size
Posted by Uwe Schindler <uw...@thetaphi.de>.
I hope the problem is fixed now; this mail is just to check! It was hard to
unsubscribe because of the strange eMail. Have no idea at all...
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Wednesday, August 15, 2012 3:13 PM
> To: java-user@lucene.apache.org
> Subject: RE: howto run CheckIndex on huge index size
>
> I got is, too. As a moderator of this list, I will look into finding the
root cause
> and forcefully unsubscribe the failing address!
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> > Sent: Wednesday, August 15, 2012 3:04 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: howto run CheckIndex on huge index size
> >
> >
> > I guess that ulimit could be a default setting of XenServer when it
> > was
> first time
> > setup.
> > We started with about 27G.
> > I already raised ulimit -n when setting up XenServer because this was
> > also limited.
> >
> > By the way, am I the only one getting this nasty DELIVERY FAILURE
> > message from one on this list?
> >
> > Bernd
> >
> >
> >
> >
> > Am 15.08.2012 14:10, schrieb Uwe Schindler:
> > > So my blog post, last section, helped? I think the ulimits came from
> there.
> > > What distribution do you use that ulimit was actually limited - or
> > > was it some sysadmin doing this? :-)
> > >
> > > We should maybe refer to this blog post from docs or create a copy
> > > of the page inside lucene's distribution!
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> > > eMail: uwe@thetaphi.de
> > >
> > >
> > >> -----Original Message-----
> > >> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> > >> Sent: Wednesday, August 15, 2012 2:07 PM
> > >> To: java-user@lucene.apache.org
> > >> Subject: Re: howto run CheckIndex on huge index size
> > >>
> > >> Hi Uwe,
> > >>
> > >> index size is:
> > >> -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt
> > >> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
> > >> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
> > >> -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq
> > >> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
> > >> -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx
> > >> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
> > >> -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis
> > >> -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u
> > >> -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen
> > >>
> > >> Setting ulimit -v and ulimit -m to unlimited solved this and I can
> > >> run
> > > CheckIndex
> > >> without any java options.
> > >>
> > >> The setting with ulimit should be pinned on lucene and solr pages!
> > >>
> > >> Thanks for help,
> > >> Bernd
> > >>
> > >>
> > >> Am 15.08.2012 13:34, schrieb Uwe Schindler:
> > >>> You don't get a heap-related OOM in your stack trace, it is "Map
> > >>> failed" - caused by MMapDirectory. You don't have enough virtual
> > >>> memory to map the index into address space. I think your heap is way
> > >>> too mch (-Xmx25g is way too big for any existing index and drives GC
> > > crazy).
> > >> How big is your index?
> > >>>
> > >>> See
> > >>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.h
> > >>> tm
> > >>> l
> > >>>
> > >>> -----
> > >>> Uwe Schindler
> > >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> > >>> eMail: uwe@thetaphi.de
> > >>>
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> > >>>> Sent: Wednesday, August 15, 2012 1:25 PM
> > >>>> To: java-user@lucene.apache.org
> > >>>> Subject: howto run CheckIndex on huge index size
> > >>>>
> > >>>>
> > >>>> I'm trying to run CheckIndex as seperate tool on a large index to
get
> > >>>> nice
> > >>> infos
> > >>>> about number of terms, number of tokens, ... but always get OOM
> > >> exception.
> > >>>>
> > >>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
> > >>>>
> > >>>>
> > >>>> Any idea how to use CheckIndex on huge index size?
> > >>>>
> > >>>>
> > >>>>
> > >>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
> > >>>> Segments file=segments_5u numSegments=1 version=3.6.1
> > >>>> format=FORMAT_3_1 [Lucene 3.1+]
> > >>>> 1 of 1: name=_2rhe docCount=37821132
> > >>>> compound=false
> > >>>> hasProx=true
> > >>>> numFiles=8
> > >>>> size (MB)=182.992,35
> > >>>> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen,
> > >>>> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13
> > >>>> 14:08:0 6, source=merge, os.arch=amd64, mergeMaxNumSegments=1,
> > >>>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
> > >>>> no deletions
> > >>>> test: open reader.........FAILED
> > >>>> WARNING: fixIndex() would remove reference to this segment;
full
> > >>>> exception:
> > >>>> java.io.IOException: Map failed
> > >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec
> > >>>> t
> > >>>> ory.java:271)
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
> > >>>> at
> > >>>>
> > >
> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
> > >>>> at
> > >>>>
> > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.
> > >>>> jav
> > >>>> a:83)
> > >>>> at
> > >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
> > >>>> at
> > >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
> > >>>> at
> > >>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
> > >>>> at
> > >>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
> > >>>> Caused by: java.lang.OutOfMemoryError: Map failed
> > >>>> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> > >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
> > >>>> ... 8 more
> > >>>> WARNING: 1 broken segments (containing 37821132 documents)
> detected
> > >>>> WARNING: would write new segments file, and 37821132 documents
> > would
> > >>>> be lost, if -fix were specified
> > >>>>
> > >>>>
> > >>>>
> > >>>>
---------------------------------------------------------------------
> > >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >>>
> > >>>
> > >>>
---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >>>
> > >>
> > >> --
> > >>
> *************************************************************
> > >> Bernd Fehling Universitätsbibliothek Bielefeld
> > >> Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
> > >> Universitätsstr. 25 und Wissensmanagement
> > >> 33615 Bielefeld
> > >> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
> > >>
> > >> BASE - Bielefeld Academic Search Engine - www.base-search.net
> > >>
> *************************************************************
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> >
> > --
> > *************************************************************
> > Bernd Fehling Universitätsbibliothek Bielefeld
> > Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
> > Universitätsstr. 25 und Wissensmanagement
> > 33615 Bielefeld
> > Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
> >
> > BASE - Bielefeld Academic Search Engine - www.base-search.net
> > *************************************************************
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: howto run CheckIndex on huge index size
Posted by Uwe Schindler <uw...@thetaphi.de>.
I got is, too. As a moderator of this list, I will look into finding the
root cause and forcefully unsubscribe the failing address!
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> Sent: Wednesday, August 15, 2012 3:04 PM
> To: java-user@lucene.apache.org
> Subject: Re: howto run CheckIndex on huge index size
>
>
> I guess that ulimit could be a default setting of XenServer when it was
first time
> setup.
> We started with about 27G.
> I already raised ulimit -n when setting up XenServer because this was also
> limited.
>
> By the way, am I the only one getting this nasty DELIVERY FAILURE message
> from one on this list?
>
> Bernd
>
>
>
>
> Am 15.08.2012 14:10, schrieb Uwe Schindler:
> > So my blog post, last section, helped? I think the ulimits came from
there.
> > What distribution do you use that ulimit was actually limited - or was
> > it some sysadmin doing this? :-)
> >
> > We should maybe refer to this blog post from docs or create a copy of
> > the page inside lucene's distribution!
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> >> -----Original Message-----
> >> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> >> Sent: Wednesday, August 15, 2012 2:07 PM
> >> To: java-user@lucene.apache.org
> >> Subject: Re: howto run CheckIndex on huge index size
> >>
> >> Hi Uwe,
> >>
> >> index size is:
> >> -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt
> >> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
> >> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
> >> -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq
> >> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
> >> -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx
> >> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
> >> -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis
> >> -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u
> >> -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen
> >>
> >> Setting ulimit -v and ulimit -m to unlimited solved this and I can
> >> run
> > CheckIndex
> >> without any java options.
> >>
> >> The setting with ulimit should be pinned on lucene and solr pages!
> >>
> >> Thanks for help,
> >> Bernd
> >>
> >>
> >> Am 15.08.2012 13:34, schrieb Uwe Schindler:
> >>> You don't get a heap-related OOM in your stack trace, it is "Map
> >>> failed" - caused by MMapDirectory. You don't have enough virtual
> >>> memory to map the index into address space. I think your heap is way
> >>> too mch (-Xmx25g is way too big for any existing index and drives GC
> > crazy).
> >> How big is your index?
> >>>
> >>> See
> >>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.h
> >>> tm
> >>> l
> >>>
> >>> -----
> >>> Uwe Schindler
> >>> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> >>> eMail: uwe@thetaphi.de
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> >>>> Sent: Wednesday, August 15, 2012 1:25 PM
> >>>> To: java-user@lucene.apache.org
> >>>> Subject: howto run CheckIndex on huge index size
> >>>>
> >>>>
> >>>> I'm trying to run CheckIndex as seperate tool on a large index to get
> >>>> nice
> >>> infos
> >>>> about number of terms, number of tokens, ... but always get OOM
> >> exception.
> >>>>
> >>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
> >>>>
> >>>>
> >>>> Any idea how to use CheckIndex on huge index size?
> >>>>
> >>>>
> >>>>
> >>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
> >>>> Segments file=segments_5u numSegments=1 version=3.6.1
> >>>> format=FORMAT_3_1 [Lucene 3.1+]
> >>>> 1 of 1: name=_2rhe docCount=37821132
> >>>> compound=false
> >>>> hasProx=true
> >>>> numFiles=8
> >>>> size (MB)=182.992,35
> >>>> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen,
> >>>> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13
> >>>> 14:08:0 6, source=merge, os.arch=amd64, mergeMaxNumSegments=1,
> >>>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
> >>>> no deletions
> >>>> test: open reader.........FAILED
> >>>> WARNING: fixIndex() would remove reference to this segment; full
> >>>> exception:
> >>>> java.io.IOException: Map failed
> >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> >>>> at
> >>>>
> >>
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec
> >>>> t
> >>>> ory.java:271)
> >>>> at
> >>>>
> >>
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
> >>>> at
> >>>>
> > org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
> >>>> at
> >>>>
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.
> >>>> jav
> >>>> a:83)
> >>>> at
> >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
> >>>> at
> >>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
> >>>> at
> >>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
> >>>> at
> >>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
> >>>> Caused by: java.lang.OutOfMemoryError: Map failed
> >>>> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
> >>>> ... 8 more
> >>>> WARNING: 1 broken segments (containing 37821132 documents) detected
> >>>> WARNING: would write new segments file, and 37821132 documents
> would
> >>>> be lost, if -fix were specified
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>
> >> --
> >> *************************************************************
> >> Bernd Fehling Universitätsbibliothek Bielefeld
> >> Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
> >> Universitätsstr. 25 und Wissensmanagement
> >> 33615 Bielefeld
> >> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
> >>
> >> BASE - Bielefeld Academic Search Engine - www.base-search.net
> >> *************************************************************
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> --
> *************************************************************
> Bernd Fehling Universitätsbibliothek Bielefeld
> Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
> Universitätsstr. 25 und Wissensmanagement
> 33615 Bielefeld
> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
>
> BASE - Bielefeld Academic Search Engine - www.base-search.net
> *************************************************************
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: howto run CheckIndex on huge index size
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
I guess that ulimit could be a default setting of XenServer when it was first time setup.
We started with about 27G.
I already raised ulimit -n when setting up XenServer because this was also limited.
By the way, am I the only one getting this nasty DELIVERY FAILURE message from one
on this list?
Bernd
Am 15.08.2012 14:10, schrieb Uwe Schindler:
> So my blog post, last section, helped? I think the ulimits came from there.
> What distribution do you use that ulimit was actually limited - or was it
> some sysadmin doing this? :-)
>
> We should maybe refer to this blog post from docs or create a copy of the
> page inside lucene's distribution!
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
>> Sent: Wednesday, August 15, 2012 2:07 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: howto run CheckIndex on huge index size
>>
>> Hi Uwe,
>>
>> index size is:
>> -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt
>> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
>> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
>> -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq
>> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
>> -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx
>> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
>> -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis
>> -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u
>> -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen
>>
>> Setting ulimit -v and ulimit -m to unlimited solved this and I can run
> CheckIndex
>> without any java options.
>>
>> The setting with ulimit should be pinned on lucene and solr pages!
>>
>> Thanks for help,
>> Bernd
>>
>>
>> Am 15.08.2012 13:34, schrieb Uwe Schindler:
>>> You don't get a heap-related OOM in your stack trace, it is "Map
>>> failed" - caused by MMapDirectory. You don't have enough virtual
>>> memory to map the index into address space. I think your heap is way
>>> too mch (-Xmx25g is way too big for any existing index and drives GC
> crazy).
>> How big is your index?
>>>
>>> See
>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.htm
>>> l
>>>
>>> -----
>>> Uwe Schindler
>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>> http://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>>
>>>
>>>> -----Original Message-----
>>>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
>>>> Sent: Wednesday, August 15, 2012 1:25 PM
>>>> To: java-user@lucene.apache.org
>>>> Subject: howto run CheckIndex on huge index size
>>>>
>>>>
>>>> I'm trying to run CheckIndex as seperate tool on a large index to get
>>>> nice
>>> infos
>>>> about number of terms, number of tokens, ... but always get OOM
>> exception.
>>>>
>>>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
>>>>
>>>>
>>>> Any idea how to use CheckIndex on huge index size?
>>>>
>>>>
>>>>
>>>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
>>>> Segments file=segments_5u numSegments=1 version=3.6.1
>>>> format=FORMAT_3_1 [Lucene 3.1+]
>>>> 1 of 1: name=_2rhe docCount=37821132
>>>> compound=false
>>>> hasProx=true
>>>> numFiles=8
>>>> size (MB)=182.992,35
>>>> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen,
>>>> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13
>>>> 14:08:0 6, source=merge, os.arch=amd64, mergeMaxNumSegments=1,
>>>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
>>>> no deletions
>>>> test: open reader.........FAILED
>>>> WARNING: fixIndex() would remove reference to this segment; full
>>>> exception:
>>>> java.io.IOException: Map failed
>>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>>>> at
>>>>
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec
>>>> t
>>>> ory.java:271)
>>>> at
>>>>
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
>>>> at
>>>>
> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
>>>> at
>>>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.
>>>> jav
>>>> a:83)
>>>> at
>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
>>>> at
>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
>>>> at
>>> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
>>>> at
>>>> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
>>>> Caused by: java.lang.OutOfMemoryError: Map failed
>>>> at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>>>> ... 8 more
>>>> WARNING: 1 broken segments (containing 37821132 documents) detected
>>>> WARNING: would write new segments file, and 37821132 documents would
>>>> be lost, if -fix were specified
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> --
>> *************************************************************
>> Bernd Fehling Universitätsbibliothek Bielefeld
>> Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
>> Universitätsstr. 25 und Wissensmanagement
>> 33615 Bielefeld
>> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
>>
>> BASE - Bielefeld Academic Search Engine - www.base-search.net
>> *************************************************************
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
*************************************************************
Bernd Fehling Universitätsbibliothek Bielefeld
Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
Universitätsstr. 25 und Wissensmanagement
33615 Bielefeld
Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: howto run CheckIndex on huge index size
Posted by Uwe Schindler <uw...@thetaphi.de>.
So my blog post, last section, helped? I think the ulimits came from there.
What distribution do you use that ulimit was actually limited - or was it
some sysadmin doing this? :-)
We should maybe refer to this blog post from docs or create a copy of the
page inside lucene's distribution!
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> Sent: Wednesday, August 15, 2012 2:07 PM
> To: java-user@lucene.apache.org
> Subject: Re: howto run CheckIndex on huge index size
>
> Hi Uwe,
>
> index size is:
> -rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt
> -rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
> -rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
> -rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq
> -rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
> -rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx
> -rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
> -rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis
> -rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u
> -rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen
>
> Setting ulimit -v and ulimit -m to unlimited solved this and I can run
CheckIndex
> without any java options.
>
> The setting with ulimit should be pinned on lucene and solr pages!
>
> Thanks for help,
> Bernd
>
>
> Am 15.08.2012 13:34, schrieb Uwe Schindler:
> > You don't get a heap-related OOM in your stack trace, it is "Map
> > failed" - caused by MMapDirectory. You don't have enough virtual
> > memory to map the index into address space. I think your heap is way
> > too mch (-Xmx25g is way too big for any existing index and drives GC
crazy).
> How big is your index?
> >
> > See
> > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.htm
> > l
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> >> -----Original Message-----
> >> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> >> Sent: Wednesday, August 15, 2012 1:25 PM
> >> To: java-user@lucene.apache.org
> >> Subject: howto run CheckIndex on huge index size
> >>
> >>
> >> I'm trying to run CheckIndex as seperate tool on a large index to get
> >> nice
> > infos
> >> about number of terms, number of tokens, ... but always get OOM
> exception.
> >>
> >> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
> >>
> >>
> >> Any idea how to use CheckIndex on huge index size?
> >>
> >>
> >>
> >> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
> >> Segments file=segments_5u numSegments=1 version=3.6.1
> >> format=FORMAT_3_1 [Lucene 3.1+]
> >> 1 of 1: name=_2rhe docCount=37821132
> >> compound=false
> >> hasProx=true
> >> numFiles=8
> >> size (MB)=182.992,35
> >> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen,
> >> os=Linux, lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13
> >> 14:08:0 6, source=merge, os.arch=amd64, mergeMaxNumSegments=1,
> >> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
> >> no deletions
> >> test: open reader.........FAILED
> >> WARNING: fixIndex() would remove reference to this segment; full
> >> exception:
> >> java.io.IOException: Map failed
> >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> >> at
> >>
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirec
> >> t
> >> ory.java:271)
> >> at
> >>
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
> >> at
> >>
org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
> >> at
> >> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.
> >> jav
> >> a:83)
> >> at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
> >> at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
> >> at
> > org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
> >> at
> >> org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
> >> Caused by: java.lang.OutOfMemoryError: Map failed
> >> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> >> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
> >> ... 8 more
> >> WARNING: 1 broken segments (containing 37821132 documents) detected
> >> WARNING: would write new segments file, and 37821132 documents would
> >> be lost, if -fix were specified
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> --
> *************************************************************
> Bernd Fehling Universitätsbibliothek Bielefeld
> Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
> Universitätsstr. 25 und Wissensmanagement
> 33615 Bielefeld
> Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
>
> BASE - Bielefeld Academic Search Engine - www.base-search.net
> *************************************************************
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: howto run CheckIndex on huge index size
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
Hi Uwe,
index size is:
-rw-r--r-- 1 solr users 82G 15. Aug 07:50 _2rhe.fdt
-rw-r--r-- 1 solr users 303M 15. Aug 07:50 _2rhe.fdx
-rw-r--r-- 1 solr users 1,2k 15. Aug 07:36 _2rhe.fnm
-rw-r--r-- 1 solr users 39G 15. Aug 09:04 _2rhe.frq
-rw-r--r-- 1 solr users 757M 15. Aug 09:05 _2rhe.nrm
-rw-r--r-- 1 solr users 47G 15. Aug 09:04 _2rhe.prx
-rw-r--r-- 1 solr users 257M 15. Aug 09:04 _2rhe.tii
-rw-r--r-- 1 solr users 25G 15. Aug 09:04 _2rhe.tis
-rw-r--r-- 1 solr users 300 15. Aug 09:05 segments_5u
-rw-r--r-- 1 solr users 20 15. Aug 09:05 segments.gen
Setting ulimit -v and ulimit -m to unlimited solved this and I can
run CheckIndex without any java options.
The setting with ulimit should be pinned on lucene and solr pages!
Thanks for help,
Bernd
Am 15.08.2012 13:34, schrieb Uwe Schindler:
> You don't get a heap-related OOM in your stack trace, it is "Map failed" -
> caused by MMapDirectory. You don't have enough virtual memory to map the
> index into address space. I think your heap is way too mch (-Xmx25g is way
> too big for any existing index and drives GC crazy). How big is your index?
>
> See http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
>> Sent: Wednesday, August 15, 2012 1:25 PM
>> To: java-user@lucene.apache.org
>> Subject: howto run CheckIndex on huge index size
>>
>>
>> I'm trying to run CheckIndex as seperate tool on a large index to get nice
> infos
>> about number of terms, number of tokens, ... but always get OOM exception.
>>
>> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
>>
>>
>> Any idea how to use CheckIndex on huge index size?
>>
>>
>>
>> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
>> Segments file=segments_5u numSegments=1 version=3.6.1
>> format=FORMAT_3_1 [Lucene 3.1+]
>> 1 of 1: name=_2rhe docCount=37821132
>> compound=false
>> hasProx=true
>> numFiles=8
>> size (MB)=182.992,35
>> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen, os=Linux,
>> lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13 14:08:0 6,
>> source=merge, os.arch=amd64, mergeMaxNumSegments=1,
>> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
>> no deletions
>> test: open reader.........FAILED
>> WARNING: fixIndex() would remove reference to this segment; full
>> exception:
>> java.io.IOException: Map failed
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>> at
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirect
>> ory.java:271)
>> at
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
>> at
>> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
>> at
>> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.jav
>> a:83)
>> at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
>> at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
>> at
> org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
>> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
>> Caused by: java.lang.OutOfMemoryError: Map failed
>> at sun.nio.ch.FileChannelImpl.map0(Native Method)
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>> ... 8 more
>> WARNING: 1 broken segments (containing 37821132 documents) detected
>> WARNING: would write new segments file, and 37821132 documents would be
>> lost, if -fix were specified
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--
*************************************************************
Bernd Fehling Universitätsbibliothek Bielefeld
Dipl.-Inform. (FH) LibTec - Bibliothekstechnologie
Universitätsstr. 25 und Wissensmanagement
33615 Bielefeld
Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de
BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: howto run CheckIndex on huge index size
Posted by Uwe Schindler <uw...@thetaphi.de>.
You don't get a heap-related OOM in your stack trace, it is "Map failed" -
caused by MMapDirectory. You don't have enough virtual memory to map the
index into address space. I think your heap is way too mch (-Xmx25g is way
too big for any existing index and drives GC crazy). How big is your index?
See http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Bernd Fehling [mailto:bernd.fehling@uni-bielefeld.de]
> Sent: Wednesday, August 15, 2012 1:25 PM
> To: java-user@lucene.apache.org
> Subject: howto run CheckIndex on huge index size
>
>
> I'm trying to run CheckIndex as seperate tool on a large index to get nice
infos
> about number of terms, number of tokens, ... but always get OOM exception.
>
> Already have JAVA_OPTS -d64 -Xmx25g -Xms25g -Xmn6g
>
>
> Any idea how to use CheckIndex on huge index size?
>
>
>
> Opening index @ /srv/www/solr/solr/solrserver/solr/data/index
> Segments file=segments_5u numSegments=1 version=3.6.1
> format=FORMAT_3_1 [Lucene 3.1+]
> 1 of 1: name=_2rhe docCount=37821132
> compound=false
> hasProx=true
> numFiles=8
> size (MB)=182.992,35
> diagnostics = {mergeFactor=1, os.version=2.6.32.23-0.3-xen, os=Linux,
> lucene.version=3.6.1-SNAPSHOT 1325634 - solr - 2012-04-13 14:08:0 6,
> source=merge, os.arch=amd64, mergeMaxNumSegments=1,
> java.version=1.6.0_29, java.vendor=Sun Microsystems Inc.}
> no deletions
> test: open reader.........FAILED
> WARNING: fixIndex() would remove reference to this segment; full
> exception:
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> at
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirect
> ory.java:271)
> at
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
> at
> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:104)
> at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.jav
> a:83)
> at
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
> at
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
> at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:523)
> at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:1064)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
> ... 8 more
> WARNING: 1 broken segments (containing 37821132 documents) detected
> WARNING: would write new segments file, and 37821132 documents would be
> lost, if -fix were specified
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org