You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Justin <th...@gmail.com> on 2008/03/03 20:43:03 UTC
out of memory every time
I'm indexing a large number of documents.
As a server I'm using the /solr/example/start.jar
No matter how much memory I allocate it fails around 7200 documents.
I am committing every 100 docs, and optimizing every 300.
all of my xml's contain on doc, and can range in size from 2k to 700k.
when I restart the start.jar it again reports out of memory.
a sample document looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<add>
<doc>
<field name="PK">1851</field>
<field name="ft:genes.Symbol:1851">TRAJ20</field>
<field name="ft:external_ids.SourceAccession:15531">12049</field>
<field
name="ft:external_ids.SourceAccession:15532">ENSG00000211869</field>
<field name="ft:external_ids.SourceAccession:15533">28735</field>
<field name="ft:external_ids.SourceAccession:15534">HUgn28735</field>
<field name="ft:external_ids.SourceAccession:15535">TRA_</field>
<field name="ft:external_ids.SourceAccession:15536">TRAJ20</field>
<field name="ft:external_ids.SourceAccession:15537">9953837</field>
<field
name="ft:external_ids.SourceAccession:15538">ENSG00000211869</field>
<field name="ft:aliases_and_descriptions.Value:9775">T cell receptor alpha
joining 20</field>
<field name="ft:cytogenetic_locations.Cytoband:4909">14q11.2</field>
<field name="ft:cytogenetic_locations.Cytoband:4910">14q11</field>
<field name="ft:cytogenetic_locations.Cytoband:4911">14q11.2</field>
<field name="ft:location_extras.ContigRefseq:11806">AE000662.1</field>
<field name="ft:location_extras.ContigRefseq:11807">M94081.1</field>
<field name="ft:location_extras.ContigRefseq:11808">CH471078.2</field>
<field name="ft:location_extras.ContigRefseq:11809">NC_000014.7</field>
<field name="ft:location_extras.ContigRefseq:11810">NT_026437.11</field>
<field name="ft:location_extras.ContigRefseq:11811">NG_001332.2</field>
<field name="ft:articles.SourceAccession:192767">8188290</field>
<field name="ft:articles.Title:192767">The human T-cell receptor
TCRAC/TCRDC (C alpha/C delta) region: organization,sequence, and evolution
of 97.6 kb of DNA.</field>
<field name="ft:authors.AuthorName:5909">Koop B.F.</field>
<field name="ft:authors.AuthorName:6912">Rowen L.</field>
<field name="ft:authors.AuthorName:6985">Hood L.</field>
<field name="ft:authors.AuthorName:17109">Wang K.</field>
<field name="ft:authors.AuthorName:72700">Kuo C.L.</field>
<field name="ft:authors.AuthorName:84285">Seto D.</field>
<field name="ft:authors.AuthorName:166156">Lenstra J.A.</field>
<field name="ft:authors.AuthorName:216734">Howard S.</field>
<field name="ft:authors.AuthorName:285493">Shan W.</field>
<field name="ft:authors.AuthorName:346559">Deshpande P.</field>
<field name="ft:probesets.Name:6773">31311_at</field>
<field name="ft:probesets.BinaryPattern:6773">000000000000</field>
</doc>
</add>
the schema is (in summary):
<fields>
<field name="PK" type="sint" indexed="true" stored="true" required="true"
multiValued="false" omitNorms="true"/>
<field name="text" type="text" indexed="true" stored="false"
multiValued="true" omitNorms="true"/>
<dynamicField name="ft:*" type="string" indexed="true"
stored="true" omitNorms="true"/>
<dynamicField name="st:*" type="string" indexed="true" stored="true"
omitNorms="true"/>
</fields>
<uniqueKey>PK</uniqueKey>
<defaultSearchField>text</defaultSearchField>
<solrQueryParser defaultOperator="OR"/>
<copyField source="ft:*" dest="text"/>
<copyField source="st:*" dest="text"/>
and my conf is:
<useCompoundFile>false</useCompoundFile>
<mergeFactor>100</mergeFactor>
<maxBufferedDocs>900</maxBufferedDocs>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength>
RE: out of memory every time
Posted by Jae Joo <jj...@ECNext.com>.
While the job is running, you can monitor the memory usage.
Use the following command - jstat (you can find in the java/bin directory)
jstat -gc <PID> 5s --> every 5 seconds.
Jae
-----Original Message-----
From: Reece [mailto:liquidpele@gmail.com]
Sent: Mon 3/3/2008 8:20 PM
To: solr-user@lucene.apache.org
Subject: Re: out of memory every time
Just guessing, but I'd say it has something to do with the dynamic fields...
I ran a similar operation (docs ranged from 1K to 2MB). For the
initial indexing, I wrote a job to submit about 100,000 documents to
solr, committing after every 10 docs. I never sent any optimize
commands. I also used the example start.jar and didn't specify any
memory constraints.
My job ran for 3 days, and finished without any errors or memory problems.
The only difference I see is that I didn't use any dynamic fields, and
I only stored 2 fields instead of them all.
Just my $0.02
-Reece
On Mon, Mar 3, 2008 at 6:15 PM, Thorsten Scherler <th...@apache.org> wrote:
> On Mon, 2008-03-03 at 21:43 +0200, Justin wrote:
> > I'm indexing a large number of documents.
> >
> > As a server I'm using the /solr/example/start.jar
> >
> > No matter how much memory I allocate it fails around 7200 documents.
>
> How do you allocate the memory?
>
> Something like:
> java -Xms512M -Xmx1500M -jar start.jar
>
> You may have a closer look as well at
> http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html
>
> HTH
>
> salu2
>
>
>
> > I am committing every 100 docs, and optimizing every 300.
> >
> > all of my xml's contain on doc, and can range in size from 2k to 700k.
> >
> > when I restart the start.jar it again reports out of memory.
> >
> >
> > a sample document looks like this:
> > <?xml version="1.0" encoding="UTF-8"?>
> > <add>
> > <doc>
> > <field name="PK">1851</field>
> > <field name="ft:genes.Symbol:1851">TRAJ20</field>
> > <field name="ft:external_ids.SourceAccession:15531">12049</field>
> > <field
> > name="ft:external_ids.SourceAccession:15532">ENSG00000211869</field>
> > <field name="ft:external_ids.SourceAccession:15533">28735</field>
> > <field name="ft:external_ids.SourceAccession:15534">HUgn28735</field>
> > <field name="ft:external_ids.SourceAccession:15535">TRA_</field>
> > <field name="ft:external_ids.SourceAccession:15536">TRAJ20</field>
> > <field name="ft:external_ids.SourceAccession:15537">9953837</field>
> > <field
> > name="ft:external_ids.SourceAccession:15538">ENSG00000211869</field>
> > <field name="ft:aliases_and_descriptions.Value:9775">T cell receptor alpha
> > joining 20</field>
> > <field name="ft:cytogenetic_locations.Cytoband:4909">14q11.2</field>
> > <field name="ft:cytogenetic_locations.Cytoband:4910">14q11</field>
> > <field name="ft:cytogenetic_locations.Cytoband:4911">14q11.2</field>
> > <field name="ft:location_extras.ContigRefseq:11806">AE000662.1</field>
> > <field name="ft:location_extras.ContigRefseq:11807">M94081.1</field>
> > <field name="ft:location_extras.ContigRefseq:11808">CH471078.2</field>
> > <field name="ft:location_extras.ContigRefseq:11809">NC_000014.7</field>
> > <field name="ft:location_extras.ContigRefseq:11810">NT_026437.11</field>
> > <field name="ft:location_extras.ContigRefseq:11811">NG_001332.2</field>
> > <field name="ft:articles.SourceAccession:192767">8188290</field>
> > <field name="ft:articles.Title:192767">The human T-cell receptor
> > TCRAC/TCRDC (C alpha/C delta) region: organization,sequence, and evolution
> > of 97.6 kb of DNA.</field>
> > <field name="ft:authors.AuthorName:5909">Koop B.F.</field>
> > <field name="ft:authors.AuthorName:6912">Rowen L.</field>
> > <field name="ft:authors.AuthorName:6985">Hood L.</field>
> > <field name="ft:authors.AuthorName:17109">Wang K.</field>
> > <field name="ft:authors.AuthorName:72700">Kuo C.L.</field>
> > <field name="ft:authors.AuthorName:84285">Seto D.</field>
> > <field name="ft:authors.AuthorName:166156">Lenstra J.A.</field>
> > <field name="ft:authors.AuthorName:216734">Howard S.</field>
> > <field name="ft:authors.AuthorName:285493">Shan W.</field>
> > <field name="ft:authors.AuthorName:346559">Deshpande P.</field>
> > <field name="ft:probesets.Name:6773">31311_at</field>
> > <field name="ft:probesets.BinaryPattern:6773">000000000000</field>
> > </doc>
> > </add>
> >
> >
> > the schema is (in summary):
> > <fields>
> > <field name="PK" type="sint" indexed="true" stored="true" required="true"
> > multiValued="false" omitNorms="true"/>
> > <field name="text" type="text" indexed="true" stored="false"
> > multiValued="true" omitNorms="true"/>
> >
> > <dynamicField name="ft:*" type="string" indexed="true"
> > stored="true" omitNorms="true"/>
> > <dynamicField name="st:*" type="string" indexed="true" stored="true"
> > omitNorms="true"/>
> > </fields>
> >
> >
> > <uniqueKey>PK</uniqueKey>
> > <defaultSearchField>text</defaultSearchField>
> > <solrQueryParser defaultOperator="OR"/>
> >
> > <copyField source="ft:*" dest="text"/>
> > <copyField source="st:*" dest="text"/>
> >
> >
> > and my conf is:
> > <useCompoundFile>false</useCompoundFile>
> > <mergeFactor>100</mergeFactor>
> > <maxBufferedDocs>900</maxBufferedDocs>
> > <maxMergeDocs>2147483647</maxMergeDocs>
> > <maxFieldLength>10000</maxFieldLength>
> --
> Thorsten Scherler thorsten.at.apache.org
> Open Source Java consulting, training and solutions
>
>
Re: out of memory every time
Posted by Reece <li...@gmail.com>.
Just guessing, but I'd say it has something to do with the dynamic fields...
I ran a similar operation (docs ranged from 1K to 2MB). For the
initial indexing, I wrote a job to submit about 100,000 documents to
solr, committing after every 10 docs. I never sent any optimize
commands. I also used the example start.jar and didn't specify any
memory constraints.
My job ran for 3 days, and finished without any errors or memory problems.
The only difference I see is that I didn't use any dynamic fields, and
I only stored 2 fields instead of them all.
Just my $0.02
-Reece
On Mon, Mar 3, 2008 at 6:15 PM, Thorsten Scherler <th...@apache.org> wrote:
> On Mon, 2008-03-03 at 21:43 +0200, Justin wrote:
> > I'm indexing a large number of documents.
> >
> > As a server I'm using the /solr/example/start.jar
> >
> > No matter how much memory I allocate it fails around 7200 documents.
>
> How do you allocate the memory?
>
> Something like:
> java -Xms512M -Xmx1500M -jar start.jar
>
> You may have a closer look as well at
> http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html
>
> HTH
>
> salu2
>
>
>
> > I am committing every 100 docs, and optimizing every 300.
> >
> > all of my xml's contain on doc, and can range in size from 2k to 700k.
> >
> > when I restart the start.jar it again reports out of memory.
> >
> >
> > a sample document looks like this:
> > <?xml version="1.0" encoding="UTF-8"?>
> > <add>
> > <doc>
> > <field name="PK">1851</field>
> > <field name="ft:genes.Symbol:1851">TRAJ20</field>
> > <field name="ft:external_ids.SourceAccession:15531">12049</field>
> > <field
> > name="ft:external_ids.SourceAccession:15532">ENSG00000211869</field>
> > <field name="ft:external_ids.SourceAccession:15533">28735</field>
> > <field name="ft:external_ids.SourceAccession:15534">HUgn28735</field>
> > <field name="ft:external_ids.SourceAccession:15535">TRA_</field>
> > <field name="ft:external_ids.SourceAccession:15536">TRAJ20</field>
> > <field name="ft:external_ids.SourceAccession:15537">9953837</field>
> > <field
> > name="ft:external_ids.SourceAccession:15538">ENSG00000211869</field>
> > <field name="ft:aliases_and_descriptions.Value:9775">T cell receptor alpha
> > joining 20</field>
> > <field name="ft:cytogenetic_locations.Cytoband:4909">14q11.2</field>
> > <field name="ft:cytogenetic_locations.Cytoband:4910">14q11</field>
> > <field name="ft:cytogenetic_locations.Cytoband:4911">14q11.2</field>
> > <field name="ft:location_extras.ContigRefseq:11806">AE000662.1</field>
> > <field name="ft:location_extras.ContigRefseq:11807">M94081.1</field>
> > <field name="ft:location_extras.ContigRefseq:11808">CH471078.2</field>
> > <field name="ft:location_extras.ContigRefseq:11809">NC_000014.7</field>
> > <field name="ft:location_extras.ContigRefseq:11810">NT_026437.11</field>
> > <field name="ft:location_extras.ContigRefseq:11811">NG_001332.2</field>
> > <field name="ft:articles.SourceAccession:192767">8188290</field>
> > <field name="ft:articles.Title:192767">The human T-cell receptor
> > TCRAC/TCRDC (C alpha/C delta) region: organization,sequence, and evolution
> > of 97.6 kb of DNA.</field>
> > <field name="ft:authors.AuthorName:5909">Koop B.F.</field>
> > <field name="ft:authors.AuthorName:6912">Rowen L.</field>
> > <field name="ft:authors.AuthorName:6985">Hood L.</field>
> > <field name="ft:authors.AuthorName:17109">Wang K.</field>
> > <field name="ft:authors.AuthorName:72700">Kuo C.L.</field>
> > <field name="ft:authors.AuthorName:84285">Seto D.</field>
> > <field name="ft:authors.AuthorName:166156">Lenstra J.A.</field>
> > <field name="ft:authors.AuthorName:216734">Howard S.</field>
> > <field name="ft:authors.AuthorName:285493">Shan W.</field>
> > <field name="ft:authors.AuthorName:346559">Deshpande P.</field>
> > <field name="ft:probesets.Name:6773">31311_at</field>
> > <field name="ft:probesets.BinaryPattern:6773">000000000000</field>
> > </doc>
> > </add>
> >
> >
> > the schema is (in summary):
> > <fields>
> > <field name="PK" type="sint" indexed="true" stored="true" required="true"
> > multiValued="false" omitNorms="true"/>
> > <field name="text" type="text" indexed="true" stored="false"
> > multiValued="true" omitNorms="true"/>
> >
> > <dynamicField name="ft:*" type="string" indexed="true"
> > stored="true" omitNorms="true"/>
> > <dynamicField name="st:*" type="string" indexed="true" stored="true"
> > omitNorms="true"/>
> > </fields>
> >
> >
> > <uniqueKey>PK</uniqueKey>
> > <defaultSearchField>text</defaultSearchField>
> > <solrQueryParser defaultOperator="OR"/>
> >
> > <copyField source="ft:*" dest="text"/>
> > <copyField source="st:*" dest="text"/>
> >
> >
> > and my conf is:
> > <useCompoundFile>false</useCompoundFile>
> > <mergeFactor>100</mergeFactor>
> > <maxBufferedDocs>900</maxBufferedDocs>
> > <maxMergeDocs>2147483647</maxMergeDocs>
> > <maxFieldLength>10000</maxFieldLength>
> --
> Thorsten Scherler thorsten.at.apache.org
> Open Source Java consulting, training and solutions
>
>
Re: out of memory every time
Posted by Thorsten Scherler <th...@apache.org>.
On Mon, 2008-03-03 at 21:43 +0200, Justin wrote:
> I'm indexing a large number of documents.
>
> As a server I'm using the /solr/example/start.jar
>
> No matter how much memory I allocate it fails around 7200 documents.
How do you allocate the memory?
Something like:
java -Xms512M -Xmx1500M -jar start.jar
You may have a closer look as well at
http://java.sun.com/j2se/1.5.0/docs/guide/vm/gc-ergonomics.html
HTH
salu2
> I am committing every 100 docs, and optimizing every 300.
>
> all of my xml's contain on doc, and can range in size from 2k to 700k.
>
> when I restart the start.jar it again reports out of memory.
>
>
> a sample document looks like this:
> <?xml version="1.0" encoding="UTF-8"?>
> <add>
> <doc>
> <field name="PK">1851</field>
> <field name="ft:genes.Symbol:1851">TRAJ20</field>
> <field name="ft:external_ids.SourceAccession:15531">12049</field>
> <field
> name="ft:external_ids.SourceAccession:15532">ENSG00000211869</field>
> <field name="ft:external_ids.SourceAccession:15533">28735</field>
> <field name="ft:external_ids.SourceAccession:15534">HUgn28735</field>
> <field name="ft:external_ids.SourceAccession:15535">TRA_</field>
> <field name="ft:external_ids.SourceAccession:15536">TRAJ20</field>
> <field name="ft:external_ids.SourceAccession:15537">9953837</field>
> <field
> name="ft:external_ids.SourceAccession:15538">ENSG00000211869</field>
> <field name="ft:aliases_and_descriptions.Value:9775">T cell receptor alpha
> joining 20</field>
> <field name="ft:cytogenetic_locations.Cytoband:4909">14q11.2</field>
> <field name="ft:cytogenetic_locations.Cytoband:4910">14q11</field>
> <field name="ft:cytogenetic_locations.Cytoband:4911">14q11.2</field>
> <field name="ft:location_extras.ContigRefseq:11806">AE000662.1</field>
> <field name="ft:location_extras.ContigRefseq:11807">M94081.1</field>
> <field name="ft:location_extras.ContigRefseq:11808">CH471078.2</field>
> <field name="ft:location_extras.ContigRefseq:11809">NC_000014.7</field>
> <field name="ft:location_extras.ContigRefseq:11810">NT_026437.11</field>
> <field name="ft:location_extras.ContigRefseq:11811">NG_001332.2</field>
> <field name="ft:articles.SourceAccession:192767">8188290</field>
> <field name="ft:articles.Title:192767">The human T-cell receptor
> TCRAC/TCRDC (C alpha/C delta) region: organization,sequence, and evolution
> of 97.6 kb of DNA.</field>
> <field name="ft:authors.AuthorName:5909">Koop B.F.</field>
> <field name="ft:authors.AuthorName:6912">Rowen L.</field>
> <field name="ft:authors.AuthorName:6985">Hood L.</field>
> <field name="ft:authors.AuthorName:17109">Wang K.</field>
> <field name="ft:authors.AuthorName:72700">Kuo C.L.</field>
> <field name="ft:authors.AuthorName:84285">Seto D.</field>
> <field name="ft:authors.AuthorName:166156">Lenstra J.A.</field>
> <field name="ft:authors.AuthorName:216734">Howard S.</field>
> <field name="ft:authors.AuthorName:285493">Shan W.</field>
> <field name="ft:authors.AuthorName:346559">Deshpande P.</field>
> <field name="ft:probesets.Name:6773">31311_at</field>
> <field name="ft:probesets.BinaryPattern:6773">000000000000</field>
> </doc>
> </add>
>
>
> the schema is (in summary):
> <fields>
> <field name="PK" type="sint" indexed="true" stored="true" required="true"
> multiValued="false" omitNorms="true"/>
> <field name="text" type="text" indexed="true" stored="false"
> multiValued="true" omitNorms="true"/>
>
> <dynamicField name="ft:*" type="string" indexed="true"
> stored="true" omitNorms="true"/>
> <dynamicField name="st:*" type="string" indexed="true" stored="true"
> omitNorms="true"/>
> </fields>
>
>
> <uniqueKey>PK</uniqueKey>
> <defaultSearchField>text</defaultSearchField>
> <solrQueryParser defaultOperator="OR"/>
>
> <copyField source="ft:*" dest="text"/>
> <copyField source="st:*" dest="text"/>
>
>
> and my conf is:
> <useCompoundFile>false</useCompoundFile>
> <mergeFactor>100</mergeFactor>
> <maxBufferedDocs>900</maxBufferedDocs>
> <maxMergeDocs>2147483647</maxMergeDocs>
> <maxFieldLength>10000</maxFieldLength>
--
Thorsten Scherler thorsten.at.apache.org
Open Source Java consulting, training and solutions