Solr 6.4.1 Issue

[Islam Omar <is...@gmail.com>]

Hi All ,

I have a trouble problem when doing *full import in solr 6.4.1 using MySQL
DB , the problem is : *

i need to create 1 core which will be around 9,500,000 documents , when i
do full import with *batchSize* *= -1* in datasource  , everything was ok
but when the solr finish fetching data from database it *can't stop* running
full import command and continue trying to fetch another data.

*the log throws this exception *

Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
java.sql.SQLException: Operation not allowed after ResultSet closed
at
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:61)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:464)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:133)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
... 7 more
Caused by: java.sql.SQLException: Operation not allowed after ResultSet
closed
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:988)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:974)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:803)
at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:6985)
at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2232)
at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2216)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)

*Best regards* ,
*Islam omar*
*Java developer*

Re: Solr 6.4.1 Issue

[Shawn Heisey <ap...@elyograg.org>]

On 4/3/2017 9:55 AM, Islam Omar wrote:
> I have a trouble problem when doing *full import in solr 6.4.1 using MySQL
> DB , the problem is : *
>
> i need to create 1 core which will be around 9,500,000 documents , when i
> do full import with *batchSize* *= -1* in datasource  , everything was ok
> but when the solr finish fetching data from database it *can't stop* running
> full import command and continue trying to fetch another data.
>
> *the log throws this exception *
>
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.sql.SQLException: Operation not allowed after ResultSet closed
<snip>
> Caused by: java.sql.SQLException: Operation not allowed after ResultSet
> closed
<snip>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:458)

This error is happening in code added by SOLR-2199, in version 6.1.0:

https://issues.apache.org/jira/browse/SOLR-2199

Based on the error message and what I know about JDBC, I believe that it
is isolated to the specific new code added by that issue.

I have done DIH imports from MySQL with version 6.3.0, creating indexes
much larger than your 9.5 million rows, without any problems.

This is the first time I've seen this particular problem mentioned.  If
it WERE a bug in the SOLR-2199 code, it seems like it would have already
been noticed in the nine months since 6.1.0 was released.  It is more
likely that there's an issue with the MySQL JDBC driver, or possibly a
result of the specific SQL statements that are being used in the import.

It is always possible that a new bug was introduced in 6.4.x, but DIH is
heavily used by Solr users, and 6.4.x has been out for some time, so I
think that a general problem with DIH would have already been reported
at least once.  At this point, I think Solr probably does not have a
bug, but I'm not completely ruling that possibility out.

What version of the MySQL JDBC driver are you using with Solr?  I would
strongly recommend that you use the latest GA version.  That is
currently version 5.1.41.  There are development versions of the 6.0
driver available, but I do not recommend using those except in test
environments where you can make changes without affecting production
systems.

https://dev.mysql.com/downloads/connector/j/5.1.html

If a new driver doesn't help, we can look deeper into whether Solr has a
bug.  You may need to share your dataimport config, schema, and
solrconfig.xml, after redacting any sensitive information.

Thanks,
Shawn

Re: Solr 6.4.1 Issue

[Mikhail Khludnev <mk...@apache.org>]

Hello,

I can't understand what does actually means "everything was ok
> but when the solr finish fetching data from database it *can't stop*
running
> full import command and continue trying to fetch another data.".
Islam,
How do you know that it finishes fetching the data?
Are you sure that there was no any error or interrupt command which can
cause the exception?

On Mon, Apr 3, 2017 at 6:55 PM, Islam Omar <is...@gmail.com> wrote:

> Hi All ,
>
> I have a trouble problem when doing *full import in solr 6.4.1 using MySQL
> DB , the problem is : *
>
> i need to create 1 core which will be around 9,500,000 documents , when i
> do full import with *batchSize* *= -1* in datasource  , everything was ok
> but when the solr finish fetching data from database it *can't stop*
> running
> full import command and continue trying to fetch another data.
>
> *the log throws this exception *
>
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.sql.SQLException: Operation not allowed after ResultSet closed
> at
> org.apache.solr.handler.dataimport.DataImportHandlerException.
> wrapAndThrow(DataImportHandlerException.java:61)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.hasnext(JdbcDataSource.java:464)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator$1.hasNext(JdbcDataSource.java:377)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(
> EntityProcessorBase.java:133)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.
> nextRow(SqlEntityProcessor.java:75)
> at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(
> EntityProcessorWrapper.java:244)
> at
> org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:475)
> at
> org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:414)
> ... 7 more
> Caused by: java.sql.SQLException: Operation not allowed after ResultSet
> closed
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:988)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:974)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
> at com.mysql.jdbc.ResultSetImpl.checkClosed(ResultSetImpl.java:803)
> at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:6985)
> at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2232)
> at com.mysql.jdbc.StatementImpl.getMoreResults(StatementImpl.java:2216)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$
> ResultSetIterator.hasnext(JdbcDataSource.java:458)
>
> *Best regards* ,
> *Islam omar*
> *Java developer*
>



-- 
Sincerely yours
Mikhail Khludnev