You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Richard Marquina Lopez <ri...@gmail.com> on 2014/03/12 21:52:49 UTC
Delta import throws java heap space exception
Hi,
I have some problems when execute the delta import with 2 million of rows
from mysql database:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
at java.nio.CharBuffer.allocate(CharBuffer.java:331)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777)
at java.nio.charset.Charset.decode(Charset.java:810)
at com.mysql.jdbc.StringUtils.toString(StringUtils.java:2010)
at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:820)
at com.mysql.jdbc.BufferRow.getString(BufferRow.java:541)
at com.mysql.jdbc.ResultSetImpl.getStringInternal(
ResultSetImpl.java:5812)
at com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5689)
at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4986)
at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:5175)
at org.apache.solr.handler.dataimport.JdbcDataSource$
ResultSetIterator.getARow(JdbcDataSource.java:315)
at org.apache.solr.handler.dataimport.JdbcDataSource$
ResultSetIterator.access$700(JdbcDataSource.java:254)
at org.apache.solr.handler.dataimport.JdbcDataSource$
ResultSetIterator$1.next(JdbcDataSource.java:294)
at org.apache.solr.handler.dataimport.JdbcDataSource$
ResultSetIterator$1.next(JdbcDataSource.java:286)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(
EntityProcessorBase.java:117)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.
nextModifiedRowKey(SqlEntityProcessor.java:86)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.
nextModifiedRowKey(EntityProcessorWrapper.java:267)
at org.apache.solr.handler.dataimport.DocBuilder.
collectDelta(DocBuilder.java:781)
at org.apache.solr.handler.dataimport.DocBuilder.doDelta(
DocBuilder.java:338)
at org.apache.solr.handler.dataimport.DocBuilder.execute(
DocBuilder.java:223)
at org.apache.solr.handler.dataimport.DataImporter.
doDeltaImport(DataImporter.java:440)
at org.apache.solr.handler.dataimport.DataImporter.
runCmd(DataImporter.java:478)
at org.apache.solr.handler.dataimport.DataImporter$1.run(
DataImporter.java:457)
------------------------------------------------------------
--------------------------------------------------------------------------
java.sql.SQLException: Streaming result set
com.mysql.jdbc.RowDataDynamic@47a034e7
is still active.
No statements may be issued when any streaming result sets are open and in
use on a given connection.
Ensure that you have called .close() on any active streaming result sets
before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingDa
ta(MysqlIO.java:3361)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(
ConnectionImpl.java:5204)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5087)
at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4690)
at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1649)
at org.apache.solr.handler.dataimport.JdbcDataSource.
closeConnection(JdbcDataSource.java:436)
at org.apache.solr.handler.dataimport.JdbcDataSource.
close(JdbcDataSource.java:421)
at org.apache.solr.handler.dataimport.DocBuilder.
closeEntityProcessorWrappers(DocBuilder.java:288)
at org.apache.solr.handler.dataimport.DocBuilder.execute(
DocBuilder.java:277)
at org.apache.solr.handler.dataimport.DataImporter.
doDeltaImport(DataImporter.java:440)
at org.apache.solr.handler.dataimport.DataImporter.
runCmd(DataImporter.java:478)
at org.apache.solr.handler.dataimport.DataImporter$1.run(
DataImporter.java:457)
Currently I have the batchSize parameter stetted to -1
Configuration:
- SOLR 4.4
- Centos 5.5
- 2GB RAM
- 1 Procesosr
Does someone have the same error?
Could someone help me, please?
Thank you,
Richard
Re: Delta import throws java heap space exception
Posted by Richard Marquina Lopez <ri...@gmail.com>.
Hi Furkan,
sure, this is my data-config.xml:
<dataConfig>
<document>
<entity name="item" pk="id" dataSource="store_db" onError="skip"
query="SELECT IT.* FROM item AS IT JOIN order AS ORD ON
IT.order_id=ORD.id WHERE (IT.status=1 AND ORD.status=1)"
deltaQuery="SELECT IT.* FROM item IT, order ORD, customer CUST
WHERE IT.order_id1=ORD.id AND ORD.customer_id=CUST.id AND
(IT.last_modified_date > DATE_SUB('${dataimporter.last_index_time}',
INTERVAL 1 MINUTE) OR ORD.last_modified_date >
DATE_SUB('${dataimporter.last_index_time}', INTERVAL 1 MINUTE) OR
CUST.last_modified_date > DATE_SUB('${dataimporter.last_index_time}',
INTERVAL 1 MINUTE))"
deltaImportQuery="SELECT * FROM item WHERE id='${
dataimporter.delta.id}'"
deletedPkQuery="SELECT IT.id FROM item AS IT JOIN order AS ORD ON
IT.order_id=ORD.id WHERE (IT.status!=1 OR ORD.status!=1) AND
(IT.last_modified_date > DATE_SUB('${dataimporter.last_index_time}',
INTERVAL 1 MINUTE) OR ORD.last_modified_date >
DATE_SUB('${dataimporter.last_index_time}', INTERVAL 1 MINUTE))">
<field column="id" name="id"/>
...
<entity name="product" dataSource="store_db" query="SELECT PR.name,
PR.description FROM product PR, item IT WHERE IT.product_id = PR.id AND
IT.id='${item.id}'" >
<field column="name" name="name"/>
...
</entity>
...
</entity>
</document>
</dataConfig>
Currently I have 2.1 Million of activities.
Thanks a lot,
Richard
2014-03-12 19:16 GMT-04:00 Furkan KAMACI <fu...@gmail.com>:
> Hi;
>
> Could you send your data-config.xml?
>
> Thanks;
> Furkan KAMACI
>
>
> 2014-03-13 1:01 GMT+02:00 Richard Marquina Lopez <
> richard.marquina@gmail.com
> >:
>
> > Hi Ahmet,
> >
> > Thank you for your response, currently I have the next configuration for
> > JVM:
> > -XX:+PrintGCDetails-XX:-UseParallelGC-XX:SurvivorRatio=8-XX:NewRatio=2
> > -XX:+HeapDumpOnOutOfMemoryError-XX:PermSize=128m-XX:MaxPermSize=256m
> > -Xms1024m-Xmx2048m
> > I have 3.67 GB of physical RAM and 2GB is asigned to JVM (-Xmx2048m)
> >
> >
> > 2014-03-12 17:32 GMT-04:00 Ahmet Arslan <io...@yahoo.com>:
> >
> > > Hi Richard,
> > >
> > > How much ram do you assign to java heap? Try increasing it to 1 gb for
> > > example.
> > > Please see : https://wiki.apache.org/solr/ShawnHeisey
> > >
> > > Ahmet
> > >
> > >
> > >
> > > On Wednesday, March 12, 2014 10:53 PM, Richard Marquina Lopez <
> > > richard.marquina@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I have some problems when execute the delta import with 2 million of
> rows
> > > from mysql database:
> > >
> > > java.lang.OutOfMemoryError: Java heap space
> > > at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
> > > at java.nio.CharBuffer.allocate(CharBuffer.java:331)
> > > at
> > java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777)
> > > at java.nio.charset.Charset.decode(Charset.java:810)
> > > at com.mysql.jdbc.StringUtils.toString(StringUtils.java:2010)
> > > at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:820)
> > > at com.mysql.jdbc.BufferRow.getString(BufferRow.java:541)
> > > at com.mysql.jdbc.ResultSetImpl.getStringInternal(
> > > ResultSetImpl.java:5812)
> > > at
> > com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5689)
> > > at
> > com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4986)
> > > at
> > com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:5175)
> > > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > > ResultSetIterator.getARow(JdbcDataSource.java:315)
> > > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > > ResultSetIterator.access$700(JdbcDataSource.java:254)
> > > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > > ResultSetIterator$1.next(JdbcDataSource.java:294)
> > > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > > ResultSetIterator$1.next(JdbcDataSource.java:286)
> > > at
> > org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(
> > > EntityProcessorBase.java:117)
> > > at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> > > nextModifiedRowKey(SqlEntityProcessor.java:86)
> > > at org.apache.solr.handler.dataimport.EntityProcessorWrapper.
> > > nextModifiedRowKey(EntityProcessorWrapper.java:267)
> > > at org.apache.solr.handler.dataimport.DocBuilder.
> > > collectDelta(DocBuilder.java:781)
> > > at org.apache.solr.handler.dataimport.DocBuilder.doDelta(
> > > DocBuilder.java:338)
> > > at org.apache.solr.handler.dataimport.DocBuilder.execute(
> > > DocBuilder.java:223)
> > > at org.apache.solr.handler.dataimport.DataImporter.
> > > doDeltaImport(DataImporter.java:440)
> > > at org.apache.solr.handler.dataimport.DataImporter.
> > > runCmd(DataImporter.java:478)
> > > at org.apache.solr.handler.dataimport.DataImporter$1.run(
> > > DataImporter.java:457)
> > > ------------------------------------------------------------
> > >
> >
> --------------------------------------------------------------------------
> > >
> > > java.sql.SQLException: Streaming result set
> > > com.mysql.jdbc.RowDataDynamic@47a034e7
> > > is still active.
> > > No statements may be issued when any streaming result sets are open and
> > in
> > > use on a given connection.
> > > Ensure that you have called .close() on any active streaming result
> sets
> > > before attempting more queries.
> > > at
> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
> > > at
> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
> > > at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingDa
> > > ta(MysqlIO.java:3361)
> > > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
> > > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> > > at
> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
> > > at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(
> > > ConnectionImpl.java:5204)
> > > at
> > com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5087)
> > > at
> > > com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4690)
> > > at
> com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1649)
> > > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > > closeConnection(JdbcDataSource.java:436)
> > > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > > close(JdbcDataSource.java:421)
> > > at org.apache.solr.handler.dataimport.DocBuilder.
> > > closeEntityProcessorWrappers(DocBuilder.java:288)
> > > at org.apache.solr.handler.dataimport.DocBuilder.execute(
> > > DocBuilder.java:277)
> > > at org.apache.solr.handler.dataimport.DataImporter.
> > > doDeltaImport(DataImporter.java:440)
> > > at org.apache.solr.handler.dataimport.DataImporter.
> > > runCmd(DataImporter.java:478)
> > > at org.apache.solr.handler.dataimport.DataImporter$1.run(
> > > DataImporter.java:457)
> > >
> > > Currently I have the batchSize parameter stetted to -1
> > >
> > > Configuration:
> > > - SOLR 4.4
> > > - Centos 5.5
> > > - 2GB RAM
> > > - 1 Procesosr
> > >
> > > Does someone have the same error?
> > > Could someone help me, please?
> > >
> > > Thank you,
> > > Richard
> > >
>
Re: Delta import throws java heap space exception
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi;
Could you send your data-config.xml?
Thanks;
Furkan KAMACI
2014-03-13 1:01 GMT+02:00 Richard Marquina Lopez <richard.marquina@gmail.com
>:
> Hi Ahmet,
>
> Thank you for your response, currently I have the next configuration for
> JVM:
> -XX:+PrintGCDetails-XX:-UseParallelGC-XX:SurvivorRatio=8-XX:NewRatio=2
> -XX:+HeapDumpOnOutOfMemoryError-XX:PermSize=128m-XX:MaxPermSize=256m
> -Xms1024m-Xmx2048m
> I have 3.67 GB of physical RAM and 2GB is asigned to JVM (-Xmx2048m)
>
>
> 2014-03-12 17:32 GMT-04:00 Ahmet Arslan <io...@yahoo.com>:
>
> > Hi Richard,
> >
> > How much ram do you assign to java heap? Try increasing it to 1 gb for
> > example.
> > Please see : https://wiki.apache.org/solr/ShawnHeisey
> >
> > Ahmet
> >
> >
> >
> > On Wednesday, March 12, 2014 10:53 PM, Richard Marquina Lopez <
> > richard.marquina@gmail.com> wrote:
> >
> > Hi,
> >
> > I have some problems when execute the delta import with 2 million of rows
> > from mysql database:
> >
> > java.lang.OutOfMemoryError: Java heap space
> > at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
> > at java.nio.CharBuffer.allocate(CharBuffer.java:331)
> > at
> java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777)
> > at java.nio.charset.Charset.decode(Charset.java:810)
> > at com.mysql.jdbc.StringUtils.toString(StringUtils.java:2010)
> > at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:820)
> > at com.mysql.jdbc.BufferRow.getString(BufferRow.java:541)
> > at com.mysql.jdbc.ResultSetImpl.getStringInternal(
> > ResultSetImpl.java:5812)
> > at
> com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5689)
> > at
> com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4986)
> > at
> com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:5175)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > ResultSetIterator.getARow(JdbcDataSource.java:315)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > ResultSetIterator.access$700(JdbcDataSource.java:254)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > ResultSetIterator$1.next(JdbcDataSource.java:294)
> > at org.apache.solr.handler.dataimport.JdbcDataSource$
> > ResultSetIterator$1.next(JdbcDataSource.java:286)
> > at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(
> > EntityProcessorBase.java:117)
> > at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> > nextModifiedRowKey(SqlEntityProcessor.java:86)
> > at org.apache.solr.handler.dataimport.EntityProcessorWrapper.
> > nextModifiedRowKey(EntityProcessorWrapper.java:267)
> > at org.apache.solr.handler.dataimport.DocBuilder.
> > collectDelta(DocBuilder.java:781)
> > at org.apache.solr.handler.dataimport.DocBuilder.doDelta(
> > DocBuilder.java:338)
> > at org.apache.solr.handler.dataimport.DocBuilder.execute(
> > DocBuilder.java:223)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > doDeltaImport(DataImporter.java:440)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > runCmd(DataImporter.java:478)
> > at org.apache.solr.handler.dataimport.DataImporter$1.run(
> > DataImporter.java:457)
> > ------------------------------------------------------------
> >
> --------------------------------------------------------------------------
> >
> > java.sql.SQLException: Streaming result set
> > com.mysql.jdbc.RowDataDynamic@47a034e7
> > is still active.
> > No statements may be issued when any streaming result sets are open and
> in
> > use on a given connection.
> > Ensure that you have called .close() on any active streaming result sets
> > before attempting more queries.
> > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
> > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
> > at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingDa
> > ta(MysqlIO.java:3361)
> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
> > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> > at
> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
> > at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(
> > ConnectionImpl.java:5204)
> > at
> com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5087)
> > at
> > com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4690)
> > at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1649)
> > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > closeConnection(JdbcDataSource.java:436)
> > at org.apache.solr.handler.dataimport.JdbcDataSource.
> > close(JdbcDataSource.java:421)
> > at org.apache.solr.handler.dataimport.DocBuilder.
> > closeEntityProcessorWrappers(DocBuilder.java:288)
> > at org.apache.solr.handler.dataimport.DocBuilder.execute(
> > DocBuilder.java:277)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > doDeltaImport(DataImporter.java:440)
> > at org.apache.solr.handler.dataimport.DataImporter.
> > runCmd(DataImporter.java:478)
> > at org.apache.solr.handler.dataimport.DataImporter$1.run(
> > DataImporter.java:457)
> >
> > Currently I have the batchSize parameter stetted to -1
> >
> > Configuration:
> > - SOLR 4.4
> > - Centos 5.5
> > - 2GB RAM
> > - 1 Procesosr
> >
> > Does someone have the same error?
> > Could someone help me, please?
> >
> > Thank you,
> > Richard
> >
>
>
>
> --
> *Lic. Luis Richard Marquina Lopez*
>
>
> *Administrador GeneralMobius IT Solutions**Teléfono: 4551323 - 67406485*
>
Re: Delta import throws java heap space exception
Posted by Richard Marquina Lopez <ri...@gmail.com>.
Hi Ahmet,
Thank you for your response, currently I have the next configuration for
JVM:
-XX:+PrintGCDetails-XX:-UseParallelGC-XX:SurvivorRatio=8-XX:NewRatio=2
-XX:+HeapDumpOnOutOfMemoryError-XX:PermSize=128m-XX:MaxPermSize=256m
-Xms1024m-Xmx2048m
I have 3.67 GB of physical RAM and 2GB is asigned to JVM (-Xmx2048m)
2014-03-12 17:32 GMT-04:00 Ahmet Arslan <io...@yahoo.com>:
> Hi Richard,
>
> How much ram do you assign to java heap? Try increasing it to 1 gb for
> example.
> Please see : https://wiki.apache.org/solr/ShawnHeisey
>
> Ahmet
>
>
>
> On Wednesday, March 12, 2014 10:53 PM, Richard Marquina Lopez <
> richard.marquina@gmail.com> wrote:
>
> Hi,
>
> I have some problems when execute the delta import with 2 million of rows
> from mysql database:
>
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
> at java.nio.CharBuffer.allocate(CharBuffer.java:331)
> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777)
> at java.nio.charset.Charset.decode(Charset.java:810)
> at com.mysql.jdbc.StringUtils.toString(StringUtils.java:2010)
> at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:820)
> at com.mysql.jdbc.BufferRow.getString(BufferRow.java:541)
> at com.mysql.jdbc.ResultSetImpl.getStringInternal(
> ResultSetImpl.java:5812)
> at com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5689)
> at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4986)
> at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:5175)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.getARow(JdbcDataSource.java:315)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.access$700(JdbcDataSource.java:254)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator$1.next(JdbcDataSource.java:294)
> at org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator$1.next(JdbcDataSource.java:286)
> at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(
> EntityProcessorBase.java:117)
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> nextModifiedRowKey(SqlEntityProcessor.java:86)
> at org.apache.solr.handler.dataimport.EntityProcessorWrapper.
> nextModifiedRowKey(EntityProcessorWrapper.java:267)
> at org.apache.solr.handler.dataimport.DocBuilder.
> collectDelta(DocBuilder.java:781)
> at org.apache.solr.handler.dataimport.DocBuilder.doDelta(
> DocBuilder.java:338)
> at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:223)
> at org.apache.solr.handler.dataimport.DataImporter.
> doDeltaImport(DataImporter.java:440)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:478)
> at org.apache.solr.handler.dataimport.DataImporter$1.run(
> DataImporter.java:457)
> ------------------------------------------------------------
> --------------------------------------------------------------------------
>
> java.sql.SQLException: Streaming result set
> com.mysql.jdbc.RowDataDynamic@47a034e7
> is still active.
> No statements may be issued when any streaming result sets are open and in
> use on a given connection.
> Ensure that you have called .close() on any active streaming result sets
> before attempting more queries.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
> at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingDa
> ta(MysqlIO.java:3361)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
> at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(
> ConnectionImpl.java:5204)
> at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5087)
> at
> com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4690)
> at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1649)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> closeConnection(JdbcDataSource.java:436)
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> close(JdbcDataSource.java:421)
> at org.apache.solr.handler.dataimport.DocBuilder.
> closeEntityProcessorWrappers(DocBuilder.java:288)
> at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:277)
> at org.apache.solr.handler.dataimport.DataImporter.
> doDeltaImport(DataImporter.java:440)
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:478)
> at org.apache.solr.handler.dataimport.DataImporter$1.run(
> DataImporter.java:457)
>
> Currently I have the batchSize parameter stetted to -1
>
> Configuration:
> - SOLR 4.4
> - Centos 5.5
> - 2GB RAM
> - 1 Procesosr
>
> Does someone have the same error?
> Could someone help me, please?
>
> Thank you,
> Richard
>
--
*Lic. Luis Richard Marquina Lopez*
*Administrador GeneralMobius IT Solutions**Teléfono: 4551323 - 67406485*
Re: Delta import throws java heap space exception
Posted by Ahmet Arslan <io...@yahoo.com>.
Hi Richard,
How much ram do you assign to java heap? Try increasing it to 1 gb for example.
Please see : https://wiki.apache.org/solr/ShawnHeisey
Ahmet
On Wednesday, March 12, 2014 10:53 PM, Richard Marquina Lopez <ri...@gmail.com> wrote:
Hi,
I have some problems when execute the delta import with 2 million of rows
from mysql database:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
at java.nio.CharBuffer.allocate(CharBuffer.java:331)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777)
at java.nio.charset.Charset.decode(Charset.java:810)
at com.mysql.jdbc.StringUtils.toString(StringUtils.java:2010)
at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:820)
at com.mysql.jdbc.BufferRow.getString(BufferRow.java:541)
at com.mysql.jdbc.ResultSetImpl.getStringInternal(
ResultSetImpl.java:5812)
at com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5689)
at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:4986)
at com.mysql.jdbc.ResultSetImpl.getObject(ResultSetImpl.java:5175)
at org.apache.solr.handler.dataimport.JdbcDataSource$
ResultSetIterator.getARow(JdbcDataSource.java:315)
at org.apache.solr.handler.dataimport.JdbcDataSource$
ResultSetIterator.access$700(JdbcDataSource.java:254)
at org.apache.solr.handler.dataimport.JdbcDataSource$
ResultSetIterator$1.next(JdbcDataSource.java:294)
at org.apache.solr.handler.dataimport.JdbcDataSource$
ResultSetIterator$1.next(JdbcDataSource.java:286)
at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(
EntityProcessorBase.java:117)
at org.apache.solr.handler.dataimport.SqlEntityProcessor.
nextModifiedRowKey(SqlEntityProcessor.java:86)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.
nextModifiedRowKey(EntityProcessorWrapper.java:267)
at org.apache.solr.handler.dataimport.DocBuilder.
collectDelta(DocBuilder.java:781)
at org.apache.solr.handler.dataimport.DocBuilder.doDelta(
DocBuilder.java:338)
at org.apache.solr.handler.dataimport.DocBuilder.execute(
DocBuilder.java:223)
at org.apache.solr.handler.dataimport.DataImporter.
doDeltaImport(DataImporter.java:440)
at org.apache.solr.handler.dataimport.DataImporter.
runCmd(DataImporter.java:478)
at org.apache.solr.handler.dataimport.DataImporter$1.run(
DataImporter.java:457)
------------------------------------------------------------
--------------------------------------------------------------------------
java.sql.SQLException: Streaming result set
com.mysql.jdbc.RowDataDynamic@47a034e7
is still active.
No statements may be issued when any streaming result sets are open and in
use on a given connection.
Ensure that you have called .close() on any active streaming result sets
before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:927)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingDa
ta(MysqlIO.java:3361)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(
ConnectionImpl.java:5204)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5087)
at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4690)
at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1649)
at org.apache.solr.handler.dataimport.JdbcDataSource.
closeConnection(JdbcDataSource.java:436)
at org.apache.solr.handler.dataimport.JdbcDataSource.
close(JdbcDataSource.java:421)
at org.apache.solr.handler.dataimport.DocBuilder.
closeEntityProcessorWrappers(DocBuilder.java:288)
at org.apache.solr.handler.dataimport.DocBuilder.execute(
DocBuilder.java:277)
at org.apache.solr.handler.dataimport.DataImporter.
doDeltaImport(DataImporter.java:440)
at org.apache.solr.handler.dataimport.DataImporter.
runCmd(DataImporter.java:478)
at org.apache.solr.handler.dataimport.DataImporter$1.run(
DataImporter.java:457)
Currently I have the batchSize parameter stetted to -1
Configuration:
- SOLR 4.4
- Centos 5.5
- 2GB RAM
- 1 Procesosr
Does someone have the same error?
Could someone help me, please?
Thank you,
Richard