You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mani Kumar <ma...@gmail.com> on 2009/04/13 20:27:55 UTC
DataImporter : Java heap space
Hi All,
I am trying to setup a Solr instance on my macbook.
I get following errors when m trying to do a full db import ... please help
me on this
Apr 13, 2009 11:53:28 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity slideshow with URL:
jdbc:mysql://localhost/mydb_development
Apr 13, 2009 11:53:29 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 319
Apr 13, 2009 11:53:32 PM org.apache.solr.handler.dataimport.DataImporter
doFullImport
SEVERE: Full Import failed
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
My Java version
$ java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)
Is that i need to install a new java version?
my db is also very huge ~15 GB
please do the need full ...
thanks
mani kumar
Re: DataImporter : Java heap space
Posted by Mani Kumar <ma...@gmail.com>.
I am using Tomcat ...
On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar <ma...@gmail.com>wrote:
> Hi All,
> I am trying to setup a Solr instance on my macbook.
>
> I get following errors when m trying to do a full db import ... please help
> me on this
>
> Apr 13, 2009 11:53:28 PM
> org.apache.solr.handler.dataimport.JdbcDataSource$1 call
> INFO: Creating a connection for entity slideshow with URL:
> jdbc:mysql://localhost/mydb_development
> Apr 13, 2009 11:53:29 PM
> org.apache.solr.handler.dataimport.JdbcDataSource$1 call
> INFO: Time taken for getConnection(): 319
> Apr 13, 2009 11:53:32 PM org.apache.solr.handler.dataimport.DataImporter
> doFullImport
> SEVERE: Full Import failed
> org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
>
>
> My Java version
> $ java -version
> java version "1.5.0_16"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
> Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)
>
>
> Is that i need to install a new java version?
> my db is also very huge ~15 GB
>
> please do the need full ...
>
> thanks
> mani kumar
>
>
Re: DataImporter : Java heap space
Posted by Mani Kumar <ma...@gmail.com>.
Hi Noble:
But the question is how much memory? is there any rules or something like
that? so that i can estimate the how much memory it requires?
Yeah i can increase it upto 800MB max.... will try it and let you know
Thanks!
Mani
2009/4/14 Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
> DIH itself may not be consuming so much memory. It also includes the
> memory used by Solr.
>
> Do you have a hard limit on 400MB , is it not possible to increase it?
>
> On Tue, Apr 14, 2009 at 11:09 AM, Mani Kumar <ma...@gmail.com>
> wrote:
> > Hi ILAN:
> >
> > Only one query is required to generate a document ...
> > Here is my data-config.xml
> >
> > <dataConfig>
> > <dataSource type="JdbcDataSource" name="sp"
> > driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost/mydb_development"
> > user="root" password="******" />
> > <document name="items">
> > <entity name="item" dataSource="sp" query="select * from items">
> > <field column="id" name="id" />
> > <field column="title" name="title" />
> > </entity>
> > </document>
> > </dataConfig>
> >
> > and other useful info:
> >
> > mysql> select * from items
> > +----------+
> > | count(*) |
> > +----------+
> > | 900051 |
> > +----------+
> > 1 row in set (0.00 sec)
> >
> > Each record consist of id and title.
> >
> > id is of type int(11) and title's avg. length is 50 chars.
> >
> >
> > I am using tomcat with solr.
> > here is the command i am using to start it
> >
> > ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
> >
> >
> > Thanks! for help. I appreciate it.
> >
> > -Mani Kumar
> >
> > On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch <il...@fonz.net> wrote:
> >
> >> Depending on your dataset and how your queries look you may very likely
> >> need to increase to a larger heap size. How many queries and rows are
> >> required for each of your documents to be generated?
> >>
> >> Ilan
> >>
> >>
> >> On 4/13/09 12:21 PM, Mani Kumar wrote:
> >>
> >>> Hi Shalin:
> >>>
> >>> Thanks for quick response!
> >>>
> >>> By defaults it was set to 1.93 MB.
> >>> But i also tried it with following command:
> >>>
> >>> $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
> >>>
> >>> I also tried tricks given on
> >>> http://wiki.apache.org/solr/DataImportHandlerFaq page.
> >>>
> >>> what should i try next ?
> >>>
> >>> Thanks!
> >>> Mani Kumar
> >>>
> >>> On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar<
> >>> shalinmangar@gmail.com> wrote:
> >>>
> >>> On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar<
> manikumarchauhan@gmail.com
> >>>>
> >>>>> wrote:
> >>>>>
> >>>>
> >>>> Hi All,
> >>>>> I am trying to setup a Solr instance on my macbook.
> >>>>>
> >>>>> I get following errors when m trying to do a full db import ...
> please
> >>>>>
> >>>> help
> >>>>
> >>>>> me on this
> >>>>>
> >>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>> at
> >>>>>
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
> >>>>
> >>>>> at
> >>>>>
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
> >>>>
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
> >>>>
> >>>>> at
> >>>>>
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
> >>>>
> >>>>> at
> >>>>>
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
> >>>>
> >>>>> at
> >>>>>
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
> >>>>
> >>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >>>>> at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
> >>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
> >>>>> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
> >>>>>
> >>>>>
> >>>>> How much heap size have you allocated to the jvm?
> >>>>
> >>>> Also see http://wiki.apache.org/solr/DataImportHandlerFaq
> >>>>
> >>>> --
> >>>> Regards,
> >>>> Shalin Shekhar Mangar.
> >>>>
> >>>>
> >>>
> >>
> >> --
> >> Ilan Rabinovitch
> >> ilan@fonz.net
> >>
> >> ---
> >> SCALE 7x: 2009 Southern California Linux Expo
> >> Los Angeles, CA
> >> http://www.socallinuxexpo.org
> >>
> >>
> >
>
>
>
> --
> --Noble Paul
>
Re: DataImporter : Java heap space
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
DIH itself may not be consuming so much memory. It also includes the
memory used by Solr.
Do you have a hard limit on 400MB , is it not possible to increase it?
On Tue, Apr 14, 2009 at 11:09 AM, Mani Kumar <ma...@gmail.com> wrote:
> Hi ILAN:
>
> Only one query is required to generate a document ...
> Here is my data-config.xml
>
> <dataConfig>
> <dataSource type="JdbcDataSource" name="sp"
> driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost/mydb_development"
> user="root" password="******" />
> <document name="items">
> <entity name="item" dataSource="sp" query="select * from items">
> <field column="id" name="id" />
> <field column="title" name="title" />
> </entity>
> </document>
> </dataConfig>
>
> and other useful info:
>
> mysql> select * from items
> +----------+
> | count(*) |
> +----------+
> | 900051 |
> +----------+
> 1 row in set (0.00 sec)
>
> Each record consist of id and title.
>
> id is of type int(11) and title's avg. length is 50 chars.
>
>
> I am using tomcat with solr.
> here is the command i am using to start it
>
> ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>
>
> Thanks! for help. I appreciate it.
>
> -Mani Kumar
>
> On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch <il...@fonz.net> wrote:
>
>> Depending on your dataset and how your queries look you may very likely
>> need to increase to a larger heap size. How many queries and rows are
>> required for each of your documents to be generated?
>>
>> Ilan
>>
>>
>> On 4/13/09 12:21 PM, Mani Kumar wrote:
>>
>>> Hi Shalin:
>>>
>>> Thanks for quick response!
>>>
>>> By defaults it was set to 1.93 MB.
>>> But i also tried it with following command:
>>>
>>> $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>>>
>>> I also tried tricks given on
>>> http://wiki.apache.org/solr/DataImportHandlerFaq page.
>>>
>>> what should i try next ?
>>>
>>> Thanks!
>>> Mani Kumar
>>>
>>> On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar<
>>> shalinmangar@gmail.com> wrote:
>>>
>>> On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar<manikumarchauhan@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> Hi All,
>>>>> I am trying to setup a Solr instance on my macbook.
>>>>>
>>>>> I get following errors when m trying to do a full db import ... please
>>>>>
>>>> help
>>>>
>>>>> me on this
>>>>>
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
>>>>
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
>>>>
>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>> at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
>>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
>>>>> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
>>>>>
>>>>>
>>>>> How much heap size have you allocated to the jvm?
>>>>
>>>> Also see http://wiki.apache.org/solr/DataImportHandlerFaq
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>>
>>>
>>
>> --
>> Ilan Rabinovitch
>> ilan@fonz.net
>>
>> ---
>> SCALE 7x: 2009 Southern California Linux Expo
>> Los Angeles, CA
>> http://www.socallinuxexpo.org
>>
>>
>
--
--Noble Paul
Re: DataImporter : Java heap space
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Thu, Apr 16, 2009 at 10:31 AM, Mani Kumar <ma...@gmail.com>wrote:
> Aah, Bryan you got it ... Thanks!
> Noble: so i can hope that it'll be fixed soon :) thank you for fixing it
> ...
> please lemme know when its done..
>
This is fixed in trunk. The next nightly build should have this fix.
--
Regards,
Shalin Shekhar Mangar.
Re: DataImporter : Java heap space
Posted by Mani Kumar <ma...@gmail.com>.
Aah, Bryan you got it ... Thanks!
Noble: so i can hope that it'll be fixed soon :) thank you for fixing it ...
please lemme know when its done..
Thanks!
Mani Kumar
2009/4/16 Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
> Hi Bryan,
> Thanks a lot. It is invoking the wrong method
>
> it should have been
> bsz = context.getVariableResolver().replaceTokens(bsz);
>
> it was a silly mistake
>
> --Noble
>
> On Thu, Apr 16, 2009 at 2:13 AM, Bryan Talbot <bt...@aeriagames.com>
> wrote:
> > I think there is a bug in the 1.4 daily builds of data import handler
> which
> > is causing the batchSize parameter to be ignored. This was probably
> > introduced with more recent patches to resolve variables.
> >
> > The affected code is in JdbcDataSource.java
> >
> > String bsz = initProps.getProperty("batchSize");
> > if (bsz != null) {
> > bsz = (String) context.getVariableResolver().resolve(bsz);
> > try {
> > batchSize = Integer.parseInt(bsz);
> > if (batchSize == -1)
> > batchSize = Integer.MIN_VALUE;
> > } catch (NumberFormatException e) {
> > LOG.warn("Invalid batch size: " + bsz);
> > }
> > }
> >
> >
> > The call to context.getVariableResolver().resolve(bsz) is returning null,
> > leading to a NumberFormatException and the batchSize never being set to
> > Integer.MIN_VALUE. MySql won't use streaming result sets in this case
> which
> > can lead to the OOM we're seeing.
> >
> >
> > If your log file contains this entry like mine does, you're being
> affected
> > by this bug too.
> >
> > Apr 15, 2009 1:21:58 PM org.apache.solr.handler.dataimport.JdbcDataSource
> > init
> > WARNING: Invalid batch size: null
> >
> >
> >
> > -Bryan
> >
> >
> >
> >
> > On Apr 13, 2009, at Apr 13, 11:48 PM, Noble Paul നോബിള് नोब्ळ् wrote:
> >
> >> DIH streams 1 row at a time.
> >>
> >> DIH is just a component in Solr. Solr indexing also takes a lot of
> memory
> >>
> >> On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar <
> manikumarchauhan@gmail.com>
> >> wrote:
> >>>
> >>> Yes its throwing the same OOM error and from same place...
> >>> yes i will try increasing the size ... just curious : how this
> dataimport
> >>> works?
> >>>
> >>> Does it loads the whole table into memory?
> >>>
> >>> Is there any estimate about how much memory it needs to create index
> for
> >>> 1GB
> >>> of data.
> >>>
> >>> thx
> >>> mani
> >>>
> >>> On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar <
> >>> shalinmangar@gmail.com> wrote:
> >>>
> >>>> On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar <
> manikumarchauhan@gmail.com
> >>>>>
> >>>>> wrote:
> >>>>
> >>>>> Hi Shalin:
> >>>>> yes i tried with batchSize="-1" parameter as well
> >>>>>
> >>>>> here the config i tried with
> >>>>>
> >>>>> <dataConfig>
> >>>>>
> >>>>> <dataSource type="JdbcDataSource" batchSize="-1" name="sp"
> >>>>> driver="com.mysql.jdbc.Driver"
> >>>>> url="jdbc:mysql://localhost/mydb_development"
> >>>>> user="root" password="******" />
> >>>>>
> >>>>>
> >>>>> I hope i have used batchSize parameter @ right place.
> >>>>>
> >>>>>
> >>>> Yes that is correct. Did it still throw OOM from the same place?
> >>>>
> >>>> I'd suggest you increase the heap and see what works for you. Also try
> >>>> -server on the jvm.
> >>>>
> >>>> --
> >>>> Regards,
> >>>> Shalin Shekhar Mangar.
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> --Noble Paul
> >
> >
>
>
>
> --
> --Noble Paul
>
Re: DataImporter : Java heap space
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
Hi Bryan,
Thanks a lot. It is invoking the wrong method
it should have been
bsz = context.getVariableResolver().replaceTokens(bsz);
it was a silly mistake
--Noble
On Thu, Apr 16, 2009 at 2:13 AM, Bryan Talbot <bt...@aeriagames.com> wrote:
> I think there is a bug in the 1.4 daily builds of data import handler which
> is causing the batchSize parameter to be ignored. This was probably
> introduced with more recent patches to resolve variables.
>
> The affected code is in JdbcDataSource.java
>
> String bsz = initProps.getProperty("batchSize");
> if (bsz != null) {
> bsz = (String) context.getVariableResolver().resolve(bsz);
> try {
> batchSize = Integer.parseInt(bsz);
> if (batchSize == -1)
> batchSize = Integer.MIN_VALUE;
> } catch (NumberFormatException e) {
> LOG.warn("Invalid batch size: " + bsz);
> }
> }
>
>
> The call to context.getVariableResolver().resolve(bsz) is returning null,
> leading to a NumberFormatException and the batchSize never being set to
> Integer.MIN_VALUE. MySql won't use streaming result sets in this case which
> can lead to the OOM we're seeing.
>
>
> If your log file contains this entry like mine does, you're being affected
> by this bug too.
>
> Apr 15, 2009 1:21:58 PM org.apache.solr.handler.dataimport.JdbcDataSource
> init
> WARNING: Invalid batch size: null
>
>
>
> -Bryan
>
>
>
>
> On Apr 13, 2009, at Apr 13, 11:48 PM, Noble Paul നോബിള് नोब्ळ् wrote:
>
>> DIH streams 1 row at a time.
>>
>> DIH is just a component in Solr. Solr indexing also takes a lot of memory
>>
>> On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar <ma...@gmail.com>
>> wrote:
>>>
>>> Yes its throwing the same OOM error and from same place...
>>> yes i will try increasing the size ... just curious : how this dataimport
>>> works?
>>>
>>> Does it loads the whole table into memory?
>>>
>>> Is there any estimate about how much memory it needs to create index for
>>> 1GB
>>> of data.
>>>
>>> thx
>>> mani
>>>
>>> On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar <
>>> shalinmangar@gmail.com> wrote:
>>>
>>>> On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar <manikumarchauhan@gmail.com
>>>>>
>>>>> wrote:
>>>>
>>>>> Hi Shalin:
>>>>> yes i tried with batchSize="-1" parameter as well
>>>>>
>>>>> here the config i tried with
>>>>>
>>>>> <dataConfig>
>>>>>
>>>>> <dataSource type="JdbcDataSource" batchSize="-1" name="sp"
>>>>> driver="com.mysql.jdbc.Driver"
>>>>> url="jdbc:mysql://localhost/mydb_development"
>>>>> user="root" password="******" />
>>>>>
>>>>>
>>>>> I hope i have used batchSize parameter @ right place.
>>>>>
>>>>>
>>>> Yes that is correct. Did it still throw OOM from the same place?
>>>>
>>>> I'd suggest you increase the heap and see what works for you. Also try
>>>> -server on the jvm.
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>
>
--
--Noble Paul
Re: DataImporter : Java heap space
Posted by Bryan Talbot <bt...@aeriagames.com>.
I think there is a bug in the 1.4 daily builds of data import handler
which is causing the batchSize parameter to be ignored. This was
probably introduced with more recent patches to resolve variables.
The affected code is in JdbcDataSource.java
String bsz = initProps.getProperty("batchSize");
if (bsz != null) {
bsz = (String) context.getVariableResolver().resolve(bsz);
try {
batchSize = Integer.parseInt(bsz);
if (batchSize == -1)
batchSize = Integer.MIN_VALUE;
} catch (NumberFormatException e) {
LOG.warn("Invalid batch size: " + bsz);
}
}
The call to context.getVariableResolver().resolve(bsz) is returning
null, leading to a NumberFormatException and the batchSize never being
set to Integer.MIN_VALUE. MySql won't use streaming result sets in
this case which can lead to the OOM we're seeing.
If your log file contains this entry like mine does, you're being
affected by this bug too.
Apr 15, 2009 1:21:58 PM
org.apache.solr.handler.dataimport.JdbcDataSource init
WARNING: Invalid batch size: null
-Bryan
On Apr 13, 2009, at Apr 13, 11:48 PM, Noble Paul നോബിള്
नोब्ळ् wrote:
> DIH streams 1 row at a time.
>
> DIH is just a component in Solr. Solr indexing also takes a lot of
> memory
>
> On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar <manikumarchauhan@gmail.com
> > wrote:
>> Yes its throwing the same OOM error and from same place...
>> yes i will try increasing the size ... just curious : how this
>> dataimport
>> works?
>>
>> Does it loads the whole table into memory?
>>
>> Is there any estimate about how much memory it needs to create
>> index for 1GB
>> of data.
>>
>> thx
>> mani
>>
>> On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar <
>> shalinmangar@gmail.com> wrote:
>>
>>> On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar <manikumarchauhan@gmail.com
>>>> wrote:
>>>
>>>> Hi Shalin:
>>>> yes i tried with batchSize="-1" parameter as well
>>>>
>>>> here the config i tried with
>>>>
>>>> <dataConfig>
>>>>
>>>> <dataSource type="JdbcDataSource" batchSize="-1" name="sp"
>>>> driver="com.mysql.jdbc.Driver"
>>>> url="jdbc:mysql://localhost/mydb_development"
>>>> user="root" password="******" />
>>>>
>>>>
>>>> I hope i have used batchSize parameter @ right place.
>>>>
>>>>
>>> Yes that is correct. Did it still throw OOM from the same place?
>>>
>>> I'd suggest you increase the heap and see what works for you. Also
>>> try
>>> -server on the jvm.
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>
>
>
> --
> --Noble Paul
Re: DataImporter : Java heap space
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
DIH streams 1 row at a time.
DIH is just a component in Solr. Solr indexing also takes a lot of memory
On Tue, Apr 14, 2009 at 12:02 PM, Mani Kumar <ma...@gmail.com> wrote:
> Yes its throwing the same OOM error and from same place...
> yes i will try increasing the size ... just curious : how this dataimport
> works?
>
> Does it loads the whole table into memory?
>
> Is there any estimate about how much memory it needs to create index for 1GB
> of data.
>
> thx
> mani
>
> On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar <
> shalinmangar@gmail.com> wrote:
>
>> On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar <manikumarchauhan@gmail.com
>> >wrote:
>>
>> > Hi Shalin:
>> > yes i tried with batchSize="-1" parameter as well
>> >
>> > here the config i tried with
>> >
>> > <dataConfig>
>> >
>> > <dataSource type="JdbcDataSource" batchSize="-1" name="sp"
>> > driver="com.mysql.jdbc.Driver"
>> > url="jdbc:mysql://localhost/mydb_development"
>> > user="root" password="******" />
>> >
>> >
>> > I hope i have used batchSize parameter @ right place.
>> >
>> >
>> Yes that is correct. Did it still throw OOM from the same place?
>>
>> I'd suggest you increase the heap and see what works for you. Also try
>> -server on the jvm.
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
--
--Noble Paul
Re: DataImporter : Java heap space
Posted by Mani Kumar <ma...@gmail.com>.
Yes its throwing the same OOM error and from same place...
yes i will try increasing the size ... just curious : how this dataimport
works?
Does it loads the whole table into memory?
Is there any estimate about how much memory it needs to create index for 1GB
of data.
thx
mani
On Tue, Apr 14, 2009 at 11:48 AM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar <manikumarchauhan@gmail.com
> >wrote:
>
> > Hi Shalin:
> > yes i tried with batchSize="-1" parameter as well
> >
> > here the config i tried with
> >
> > <dataConfig>
> >
> > <dataSource type="JdbcDataSource" batchSize="-1" name="sp"
> > driver="com.mysql.jdbc.Driver"
> > url="jdbc:mysql://localhost/mydb_development"
> > user="root" password="******" />
> >
> >
> > I hope i have used batchSize parameter @ right place.
> >
> >
> Yes that is correct. Did it still throw OOM from the same place?
>
> I'd suggest you increase the heap and see what works for you. Also try
> -server on the jvm.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: DataImporter : Java heap space
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, Apr 14, 2009 at 11:36 AM, Mani Kumar <ma...@gmail.com>wrote:
> Hi Shalin:
> yes i tried with batchSize="-1" parameter as well
>
> here the config i tried with
>
> <dataConfig>
>
> <dataSource type="JdbcDataSource" batchSize="-1" name="sp"
> driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost/mydb_development"
> user="root" password="******" />
>
>
> I hope i have used batchSize parameter @ right place.
>
>
Yes that is correct. Did it still throw OOM from the same place?
I'd suggest you increase the heap and see what works for you. Also try
-server on the jvm.
--
Regards,
Shalin Shekhar Mangar.
Re: DataImporter : Java heap space
Posted by Mani Kumar <ma...@gmail.com>.
Hi Shalin:
yes i tried with batchSize="-1" parameter as well
here the config i tried with
<dataConfig>
<dataSource type="JdbcDataSource" batchSize="-1" name="sp"
driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost/mydb_development"
user="root" password="******" />
<document name="items">
<entity name="item" dataSource="sp" query="select * from items">
<field column="id" name="id" />
<field column="title" name="title" />
</entity>
</document>
</dataConfig>
I hope i have used batchSize parameter @ right place.
Thanks!
Mani Kumar
On Tue, Apr 14, 2009 at 11:24 AM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> On Tue, Apr 14, 2009 at 11:18 AM, Mani Kumar <manikumarchauhan@gmail.com
> >wrote:
>
> > Here is the stack trace:
> >
> > notice in stack trace * "at
> > com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)"*
> >
> > It looks like that its trying to read whole table into memory at a time.
> n
> > thts y getting OOM.
> >
> >
> Mani, the data-config.xml you posted does not have the batchSize="-1"
> attribute to your data source. Did you try that? This is a known bug in
> MySql jdbc driver.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: DataImporter : Java heap space
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, Apr 14, 2009 at 11:18 AM, Mani Kumar <ma...@gmail.com>wrote:
> Here is the stack trace:
>
> notice in stack trace * "at
> com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)"*
>
> It looks like that its trying to read whole table into memory at a time. n
> thts y getting OOM.
>
>
Mani, the data-config.xml you posted does not have the batchSize="-1"
attribute to your data source. Did you try that? This is a known bug in
MySql jdbc driver.
--
Regards,
Shalin Shekhar Mangar.
Re: DataImporter : Java heap space
Posted by Mani Kumar <ma...@gmail.com>.
Here is the stack trace:
notice in stack trace * "at
com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)"*
It looks like that its trying to read whole table into memory at a time. n
thts y getting OOM.
Apr 14, 2009 11:15:01 AM org.apache.solr.handler.dataimport.DataImporter
doFullImport
SEVERE: Full Import failed
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:468)
at
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2534)
at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1749)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2159)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2548)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2477)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:741)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:587)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:243)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:207)
at
org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:38)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:335)
... 5 more
Apr 14, 2009 11:15:01 AM org.apache.solr.update.DirectUpdateHandler2
rollback
INFO: start rollback
Apr 14, 2009 11:15:01 AM org.apache.solr.update.DirectUpdateHandler2
rollback
INFO: end_rollback
On Tue, Apr 14, 2009 at 11:09 AM, Mani Kumar <ma...@gmail.com>wrote:
> Hi ILAN:
>
> Only one query is required to generate a document ...
> Here is my data-config.xml
>
> <dataConfig>
> <dataSource type="JdbcDataSource" name="sp"
> driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost/mydb_development"
> user="root" password="******" />
> <document name="items">
> <entity name="item" dataSource="sp" query="select * from items">
> <field column="id" name="id" />
> <field column="title" name="title" />
> </entity>
> </document>
> </dataConfig>
>
> and other useful info:
>
> mysql> select * from items
> +----------+
> | count(*) |
> +----------+
> | 900051 |
> +----------+
> 1 row in set (0.00 sec)
>
> Each record consist of id and title.
>
> id is of type int(11) and title's avg. length is 50 chars.
>
>
> I am using tomcat with solr.
> here is the command i am using to start it
>
> ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>
>
> Thanks! for help. I appreciate it.
>
> -Mani Kumar
>
> On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch <il...@fonz.net> wrote:
>
>> Depending on your dataset and how your queries look you may very likely
>> need to increase to a larger heap size. How many queries and rows are
>> required for each of your documents to be generated?
>>
>> Ilan
>>
>>
>> On 4/13/09 12:21 PM, Mani Kumar wrote:
>>
>>> Hi Shalin:
>>>
>>> Thanks for quick response!
>>>
>>> By defaults it was set to 1.93 MB.
>>> But i also tried it with following command:
>>>
>>> $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>>>
>>> I also tried tricks given on
>>> http://wiki.apache.org/solr/DataImportHandlerFaq page.
>>>
>>> what should i try next ?
>>>
>>> Thanks!
>>> Mani Kumar
>>>
>>> On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar<
>>> shalinmangar@gmail.com> wrote:
>>>
>>> On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar<manikumarchauhan@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> Hi All,
>>>>> I am trying to setup a Solr instance on my macbook.
>>>>>
>>>>> I get following errors when m trying to do a full db import ... please
>>>>>
>>>> help
>>>>
>>>>> me on this
>>>>>
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
>>>>
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
>>>>
>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>> at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
>>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
>>>>> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
>>>>>
>>>>>
>>>>> How much heap size have you allocated to the jvm?
>>>>
>>>> Also see http://wiki.apache.org/solr/DataImportHandlerFaq
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>>
>>>
>>
>> --
>> Ilan Rabinovitch
>> ilan@fonz.net
>>
>> ---
>> SCALE 7x: 2009 Southern California Linux Expo
>> Los Angeles, CA
>> http://www.socallinuxexpo.org
>>
>>
>
Re: DataImporter : Java heap space
Posted by Mani Kumar <ma...@gmail.com>.
Hi ILAN:
Only one query is required to generate a document ...
Here is my data-config.xml
<dataConfig>
<dataSource type="JdbcDataSource" name="sp"
driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost/mydb_development"
user="root" password="******" />
<document name="items">
<entity name="item" dataSource="sp" query="select * from items">
<field column="id" name="id" />
<field column="title" name="title" />
</entity>
</document>
</dataConfig>
and other useful info:
mysql> select * from items
+----------+
| count(*) |
+----------+
| 900051 |
+----------+
1 row in set (0.00 sec)
Each record consist of id and title.
id is of type int(11) and title's avg. length is 50 chars.
I am using tomcat with solr.
here is the command i am using to start it
./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
Thanks! for help. I appreciate it.
-Mani Kumar
On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch <il...@fonz.net> wrote:
> Depending on your dataset and how your queries look you may very likely
> need to increase to a larger heap size. How many queries and rows are
> required for each of your documents to be generated?
>
> Ilan
>
>
> On 4/13/09 12:21 PM, Mani Kumar wrote:
>
>> Hi Shalin:
>>
>> Thanks for quick response!
>>
>> By defaults it was set to 1.93 MB.
>> But i also tried it with following command:
>>
>> $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>>
>> I also tried tricks given on
>> http://wiki.apache.org/solr/DataImportHandlerFaq page.
>>
>> what should i try next ?
>>
>> Thanks!
>> Mani Kumar
>>
>> On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar<
>> shalinmangar@gmail.com> wrote:
>>
>> On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar<manikumarchauhan@gmail.com
>>>
>>>> wrote:
>>>>
>>>
>>> Hi All,
>>>> I am trying to setup a Solr instance on my macbook.
>>>>
>>>> I get following errors when m trying to do a full db import ... please
>>>>
>>> help
>>>
>>>> me on this
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at
>>>>
>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
>>>
>>>> at
>>>>
>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
>>>
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
>>>
>>>> at
>>>>
>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
>>>
>>>> at
>>>>
>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
>>>
>>>> at
>>>>
>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
>>>
>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>> at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
>>>> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
>>>>
>>>>
>>>> How much heap size have you allocated to the jvm?
>>>
>>> Also see http://wiki.apache.org/solr/DataImportHandlerFaq
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>>
>>
>
> --
> Ilan Rabinovitch
> ilan@fonz.net
>
> ---
> SCALE 7x: 2009 Southern California Linux Expo
> Los Angeles, CA
> http://www.socallinuxexpo.org
>
>
Re: DataImporter : Java heap space
Posted by Ilan Rabinovitch <il...@fonz.net>.
Depending on your dataset and how your queries look you may very likely
need to increase to a larger heap size. How many queries and rows are
required for each of your documents to be generated?
Ilan
On 4/13/09 12:21 PM, Mani Kumar wrote:
> Hi Shalin:
>
> Thanks for quick response!
>
> By defaults it was set to 1.93 MB.
> But i also tried it with following command:
>
> $ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>
> I also tried tricks given on
> http://wiki.apache.org/solr/DataImportHandlerFaq page.
>
> what should i try next ?
>
> Thanks!
> Mani Kumar
>
> On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar<
> shalinmangar@gmail.com> wrote:
>
>> On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar<manikumarchauhan@gmail.com
>>> wrote:
>>
>>> Hi All,
>>> I am trying to setup a Solr instance on my macbook.
>>>
>>> I get following errors when m trying to do a full db import ... please
>> help
>>> me on this
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>>
>>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
>>> at
>>>
>>>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
>>> at
>>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
>>> at
>>>
>>>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
>>> at
>>>
>>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
>>> at
>>>
>>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>> at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
>>> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
>>>
>>>
>> How much heap size have you allocated to the jvm?
>>
>> Also see http://wiki.apache.org/solr/DataImportHandlerFaq
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
--
Ilan Rabinovitch
ilan@fonz.net
---
SCALE 7x: 2009 Southern California Linux Expo
Los Angeles, CA
http://www.socallinuxexpo.org
Re: DataImporter : Java heap space
Posted by Mani Kumar <ma...@gmail.com>.
Hi Shalin:
Thanks for quick response!
By defaults it was set to 1.93 MB.
But i also tried it with following command:
$ ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
I also tried tricks given on
http://wiki.apache.org/solr/DataImportHandlerFaq page.
what should i try next ?
Thanks!
Mani Kumar
On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
> On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar <manikumarchauhan@gmail.com
> >wrote:
>
> > Hi All,
> > I am trying to setup a Solr instance on my macbook.
> >
> > I get following errors when m trying to do a full db import ... please
> help
> > me on this
> >
> > java.lang.OutOfMemoryError: Java heap space
> > at
> >
> >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
> > at
> >
> >
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
> > at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
> > at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
> > at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
> > Caused by: java.lang.OutOfMemoryError: Java heap space
> > at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
> > at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
> >
> >
> How much heap size have you allocated to the jvm?
>
> Also see http://wiki.apache.org/solr/DataImportHandlerFaq
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: DataImporter : Java heap space
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar <ma...@gmail.com>wrote:
> Hi All,
> I am trying to setup a Solr instance on my macbook.
>
> I get following errors when m trying to do a full db import ... please help
> me on this
>
> java.lang.OutOfMemoryError: Java heap space
> at
>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
> at
>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
> at
>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
> at
>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
> at
>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
>
>
How much heap size have you allocated to the jvm?
Also see http://wiki.apache.org/solr/DataImportHandlerFaq
--
Regards,
Shalin Shekhar Mangar.