You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marc Sturlese <ma...@gmail.com> on 2009/01/05 10:06:29 UTC
Deduplication patch not working in nightly build
Hey there,
I was using the Deduplication patch with Solr 1.3 release and everything was
working perfectly. Now I upgraded to a nigthly build (20th december) to be
able to use new facet algorithm and other stuff and DeDuplication is not
working any more. I have followed exactly the same steps to apply the patch
to the source code. I am geting this error:
WARNING: Error reading data
com.mysql.jdbc.CommunicationsException: Communications link failure due to
underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
** END NESTED EXCEPTION **
Last packet sent to the server was 202481 ms ago.
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
Jan 5, 2009 10:06:16 AM org.apache.solr.handler.dataimport.JdbcDataSource
logError
WARNING: Exception while closing result set
com.mysql.jdbc.CommunicationsException: Communications link failure due to
underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
** END NESTED EXCEPTION **
Last packet sent to the server was 202482 ms ago.
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
Mark do you know if I should to some extra hack to run DeDuplication patch
in the new solr version (nightly)? Has something changed in the source that
make it not work or it's simply that i am doing something wrong?
Thanks in advanced
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Deduplication patch not working in nightly build
Posted by Marc Sturlese <ma...@gmail.com>.
Yeah looks like but... if I don't use the DeDuplication patch everything
works perfect. I can create my indexed using full import and delta import
without problems. The JdbcDataSource of the nightly is pretty similar to the
1.3 release...
The DeDuplication patch doesn't touch the dataimporthandler classes... it's
coz I thought the problem was not there (but can't say it for sure...)
I was thinking that the problem has something to do with the
UpdateRequestProcessorChain but don't know how this part of the source
works...
I am really interested in updating to the nightly build as I think new facet
algorithm and SolrDeletionPolicy are really great stuff!
>>Marc, I've just committed a fix which may have caused the bug. Can you use
>>svn trunk (or the next nightly build) and confirm?
You meann the last nightly build?
Thanks
Noble Paul നോബിള് नोब्ळ् wrote:
>
> looks like a bug w/ DIH with the recent fixes.
> --Noble
>
> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese <ma...@gmail.com>
> wrote:
>>
>> Hey there,
>> I was using the Deduplication patch with Solr 1.3 release and everything
>> was
>> working perfectly. Now I upgraded to a nigthly build (20th december) to
>> be
>> able to use new facet algorithm and other stuff and DeDuplication is not
>> working any more. I have followed exactly the same steps to apply the
>> patch
>> to the source code. I am geting this error:
>>
>> WARNING: Error reading data
>> com.mysql.jdbc.CommunicationsException: Communications link failure due
>> to
>> underlying exception:
>>
>> ** BEGIN NESTED EXCEPTION **
>>
>> java.io.EOFException
>>
>> STACKTRACE:
>>
>> java.io.EOFException
>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>
>>
>> ** END NESTED EXCEPTION **
>> Last packet sent to the server was 202481 ms ago.
>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> Jan 5, 2009 10:06:16 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> logError
>> WARNING: Exception while closing result set
>> com.mysql.jdbc.CommunicationsException: Communications link failure due
>> to
>> underlying exception:
>>
>> ** BEGIN NESTED EXCEPTION **
>>
>> java.io.EOFException
>>
>> STACKTRACE:
>>
>> java.io.EOFException
>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>
>>
>> ** END NESTED EXCEPTION **
>>
>> Last packet sent to the server was 202482 ms ago.
>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>
>> Mark do you know if I should to some extra hack to run DeDuplication
>> patch
>> in the new solr version (nightly)? Has something changed in the source
>> that
>> make it not work or it's simply that i am doing something wrong?
>>
>> Thanks in advanced
>> --
>> View this message in context:
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288872.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Deduplication patch not working in nightly build
Posted by Marc Sturlese <ma...@gmail.com>.
Yeah looks like but... if I don't use the DeDuplication patch everything
works perfect. I can create my indexed using full import and delta import
without problems. The JdbcDataSource of the nightly is pretty similar to the
1.3 release...
The DeDuplication patch doesn't touch the dataimporthandler classes... it's
coz I thought the problem was not there (but can't say it for sure...)
I was thinking that the problem has something to do with the
UpdateRequestProcessorChain but don't know how this part of the source
works...
Any advice how could I sort it? I am really interested in updating to the
nightly build as I think new facet algorithm and SolrDeletionPolicy are
really great stuff!
Thanks
Noble Paul നോബിള് नोब्ळ् wrote:
>
> looks like a bug w/ DIH with the recent fixes.
> --Noble
>
> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese <ma...@gmail.com>
> wrote:
>>
>> Hey there,
>> I was using the Deduplication patch with Solr 1.3 release and everything
>> was
>> working perfectly. Now I upgraded to a nigthly build (20th december) to
>> be
>> able to use new facet algorithm and other stuff and DeDuplication is not
>> working any more. I have followed exactly the same steps to apply the
>> patch
>> to the source code. I am geting this error:
>>
>> WARNING: Error reading data
>> com.mysql.jdbc.CommunicationsException: Communications link failure due
>> to
>> underlying exception:
>>
>> ** BEGIN NESTED EXCEPTION **
>>
>> java.io.EOFException
>>
>> STACKTRACE:
>>
>> java.io.EOFException
>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>
>>
>> ** END NESTED EXCEPTION **
>> Last packet sent to the server was 202481 ms ago.
>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> Jan 5, 2009 10:06:16 AM org.apache.solr.handler.dataimport.JdbcDataSource
>> logError
>> WARNING: Exception while closing result set
>> com.mysql.jdbc.CommunicationsException: Communications link failure due
>> to
>> underlying exception:
>>
>> ** BEGIN NESTED EXCEPTION **
>>
>> java.io.EOFException
>>
>> STACKTRACE:
>>
>> java.io.EOFException
>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>
>>
>> ** END NESTED EXCEPTION **
>>
>> Last packet sent to the server was 202482 ms ago.
>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> at
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> at
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> at
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>
>> Mark do you know if I should to some extra hack to run DeDuplication
>> patch
>> in the new solr version (nightly)? Has something changed in the source
>> that
>> make it not work or it's simply that i am doing something wrong?
>>
>> Thanks in advanced
>> --
>> View this message in context:
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288838.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Deduplication patch not working in nightly build
Posted by Mark Miller <ma...@gmail.com>.
I can't imagine why dedupe would have anything to do with this, other
than what was said, it perhaps is taking a bit longer to get a document
to the db, and it times out (maybe a long signature calculation?). Have
you tried changing your MySql settings to allow for a longer timeout?
(sorry, I'm not to up to date on what you have tried).
Also, are you using autocommit during the import? If so, you might try
turning it off for the full import.
- Mark
Marc Sturlese wrote:
> Hey there,
> I am stack in this problem sine 3 days ago and no idea how to sort it.
>
> I am using the nighlty from a week ago, mysql and this driver and url:
> driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost/my_db"
>
> I can use deduplication patch with indexs of 200.000 docs and no problem.
> When I try a full-import with a db of 1.500.000 it stops indexing at doc
> number 15.000 aprox showing me the error posted above.
> Once I get the exception, i restart tomcat and start a delta-import... this
> time everything works fine!
> I need to avoid this error in the full import, i have tryed:
>
> url="jdbc:mysql://localhost/my_db?autoReconnect=true to sort it in case the
> connection was closed due to long time until next doc was indexed, but
> nothing changed... I keep having this:
> Jan 9, 2009 1:38:18 PM org.apache.solr.handler.dataimport.JdbcDataSource
> logError
> WARNING: Error reading data
> com.mysql.jdbc.CommunicationsException: Communications link failure due to
> underlying exception:
>
> ** BEGIN NESTED EXCEPTION **
>
> java.io.EOFException
>
> STACKTRACE:
>
> java.io.EOFException
> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:279)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)
>
>
> ** END NESTED EXCEPTION **
>
>
>
> Last packet sent to the server was 206097 ms ago.
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:279)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)
> Jan 9, 2009 1:38:18 PM org.apache.solr.handler.dataimport.JdbcDataSource
> logError
> WARNING: Exception while closing result set
> com.mysql.jdbc.CommunicationsException: Communications link failure due to
> underlying exception:
>
> ** BEGIN NESTED EXCEPTION **
>
> java.io.EOFException
>
> STACKTRACE:
>
> java.io.EOFException
> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:295)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:287)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)
>
>
> ** END NESTED EXCEPTION **
>
>
>
> Last packet sent to the server was 206097 ms ago.
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:295)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:287)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)
>
>
> Has anyone experienced this problem? Any advice what could i do...
> Thanks
>
>
> Marc Sturlese wrote:
>
>> Thanks I will have a look to my JdbcDataSource. Anyway it's weird because
>> using the 1.3 release I don't have that problem...
>>
>> Shalin Shekhar Mangar wrote:
>>
>>> Yes, initially I figured that we are accidentally re-using a closed data
>>> source. But Noble has pinned it right. I guess you can try looking into
>>> your
>>> JDBC driver's documentation for a setting which increases the connection
>>> alive-ness.
>>>
>>> On Mon, Jan 5, 2009 at 5:29 PM, Noble Paul നോബിള് नोब्ळ् <
>>> noble.paul@gmail.com> wrote:
>>>
>>>
>>>> I guess the indexing of a doc is taking too long (may be because of
>>>> the de-dup patch) and the resultset gets closed automaticallly (timed
>>>> out)
>>>> --Noble
>>>>
>>>> On Mon, Jan 5, 2009 at 5:14 PM, Marc Sturlese <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Donig this fix I get the same error :(
>>>>>
>>>>> I am going to try to set up the last nigthly build... let's see if I
>>>>>
>>>> have
>>>>
>>>>> better luck.
>>>>>
>>>>> The thing is it stop indexing at the doc num 150.000 aprox... and give
>>>>>
>>>> me
>>>>
>>>>> that mysql exception error... Without DeDuplication patch I can index
>>>>>
>>>> 2
>>>>
>>>>> milion docs without problems...
>>>>>
>>>>> I am pretty lost with this... :(
>>>>>
>>>>>
>>>>> Shalin Shekhar Mangar wrote:
>>>>>
>>>>>> Yes I meant the 05/01/2008 build. The fix is a one line change
>>>>>>
>>>>>> Add the following as the last line of DataConfig.Entity.clearCache()
>>>>>> dataSrc = null;
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 5, 2009 at 4:22 PM, Marc Sturlese
>>>>>> <ma...@gmail.com>wrote:
>>>>>>
>>>>>>
>>>>>>> Shalin you mean I should test the 05/01/2008 nighlty? maybe with
>>>>>>>
>>>> this
>>>> one
>>>>
>>>>>>> works? If the fix you did is not really big can u tell me where in
>>>>>>>
>>>> the
>>>>
>>>>>>> source is and what is it for? (I have been debuging and tracing a
>>>>>>>
>>>> lot
>>>> the
>>>>
>>>>>>> dataimporthandler source and I I would like to know what the
>>>>>>>
>>>> imporovement
>>>>
>>>>>>> is
>>>>>>> about if it is not a problem...)
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>> Shalin Shekhar Mangar wrote:
>>>>>>>
>>>>>>>> Marc, I've just committed a fix which may have caused the bug. Can
>>>>>>>>
>>>> you
>>>>
>>>>>>> use
>>>>>>>
>>>>>>>> svn trunk (or the next nightly build) and confirm?
>>>>>>>>
>>>>>>>> On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള് नोब्ळ् <
>>>>>>>> noble.paul@gmail.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> looks like a bug w/ DIH with the recent fixes.
>>>>>>>>> --Noble
>>>>>>>>>
>>>>>>>>> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese
>>>>>>>>>
>>>>>>> <ma...@gmail.com>
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hey there,
>>>>>>>>>> I was using the Deduplication patch with Solr 1.3 release and
>>>>>>>>>>
>>>>>>>>> everything
>>>>>>>>> was
>>>>>>>>>
>>>>>>>>>> working perfectly. Now I upgraded to a nigthly build (20th
>>>>>>>>>>
>>>> december)
>>>>
>>>>>>> to
>>>>>>>
>>>>>>>>> be
>>>>>>>>>
>>>>>>>>>> able to use new facet algorithm and other stuff and
>>>>>>>>>>
>>>> DeDuplication
>>>> is
>>>>
>>>>>>>>> not
>>>>>>>>>
>>>>>>>>>> working any more. I have followed exactly the same steps to
>>>>>>>>>>
>>>> apply
>>>>
>>>>>>> the
>>>>>>>
>>>>>>>>> patch
>>>>>>>>>
>>>>>>>>>> to the source code. I am geting this error:
>>>>>>>>>>
>>>>>>>>>> WARNING: Error reading data
>>>>>>>>>> com.mysql.jdbc.CommunicationsException: Communications link
>>>>>>>>>>
>>>> failure
>>>>
>>>>>>> due
>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>> underlying exception:
>>>>>>>>>>
>>>>>>>>>> ** BEGIN NESTED EXCEPTION **
>>>>>>>>>>
>>>>>>>>>> java.io.EOFException
>>>>>>>>>>
>>>>>>>>>> STACKTRACE:
>>>>>>>>>>
>>>>>>>>>> java.io.EOFException
>>>>>>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>>>
>>>>>>>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>>>>>>
>>>>>>>>>> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>>>
>>>>>>>>>> ** END NESTED EXCEPTION **
>>>>>>>>>> Last packet sent to the server was 202481 ms ago.
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>>>
>>>>>>>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>>>>>>
>>>>>>>>>> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>>>
>>>>>>>>>> Jan 5, 2009 10:06:16 AM
>>>>>>>>>>
>>>>>>>>> org.apache.solr.handler.dataimport.JdbcDataSource
>>>>>>>>>
>>>>>>>>>> logError
>>>>>>>>>> WARNING: Exception while closing result set
>>>>>>>>>> com.mysql.jdbc.CommunicationsException: Communications link
>>>>>>>>>>
>>>> failure
>>>>
>>>>>>> due
>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>> underlying exception:
>>>>>>>>>>
>>>>>>>>>> ** BEGIN NESTED EXCEPTION **
>>>>>>>>>>
>>>>>>>>>> java.io.EOFException
>>>>>>>>>>
>>>>>>>>>> STACKTRACE:
>>>>>>>>>>
>>>>>>>>>> java.io.EOFException
>>>>>>>>>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>>>
>>>>>>>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>>>>
>>>>>>>>>> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>>>
>>>>>>>>>> ** END NESTED EXCEPTION **
>>>>>>>>>>
>>>>>>>>>> Last packet sent to the server was 202482 ms ago.
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>>>
>>>>>>>>>> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>>>>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>> com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>>>>
>>>>>>>>>> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>>>
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>>>
>>>>>>>>>> Mark do you know if I should to some extra hack to run
>>>>>>>>>>
>>>> DeDuplication
>>>>
>>>>>>>>> patch
>>>>>>>>>
>>>>>>>>>> in the new solr version (nightly)? Has something changed in the
>>>>>>>>>>
>>>>>>> source
>>>>>>>
>>>>>>>>> that
>>>>>>>>>
>>>>>>>>>> make it not work or it's simply that i am doing something
>>>>>>>>>>
>>>> wrong?
>>>>
>>>>>>>>>> Thanks in advanced
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>>
>>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
>>>>
>>>>>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> --Noble Paul
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Shalin Shekhar Mangar.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>>
>>>>>>>
>>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html
>>>>
>>>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shalin Shekhar Mangar.
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> View this message in context:
>>>>>
>>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21289643.html
>>>>
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>>
>>>
>>
>
>
Re: Deduplication patch not working in nightly build
Posted by Marc Sturlese <ma...@gmail.com>.
Hey there,
I am stack in this problem sine 3 days ago and no idea how to sort it.
I am using the nighlty from a week ago, mysql and this driver and url:
driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/my_db"
I can use deduplication patch with indexs of 200.000 docs and no problem.
When I try a full-import with a db of 1.500.000 it stops indexing at doc
number 15.000 aprox showing me the error posted above.
Once I get the exception, i restart tomcat and start a delta-import... this
time everything works fine!
I need to avoid this error in the full import, i have tryed:
url="jdbc:mysql://localhost/my_db?autoReconnect=true to sort it in case the
connection was closed due to long time until next doc was indexed, but
nothing changed... I keep having this:
Jan 9, 2009 1:38:18 PM org.apache.solr.handler.dataimport.JdbcDataSource
logError
WARNING: Error reading data
com.mysql.jdbc.CommunicationsException: Communications link failure due to
underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:279)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)
** END NESTED EXCEPTION **
Last packet sent to the server was 206097 ms ago.
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:279)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)
Jan 9, 2009 1:38:18 PM org.apache.solr.handler.dataimport.JdbcDataSource
logError
WARNING: Exception while closing result set
com.mysql.jdbc.CommunicationsException: Communications link failure due to
underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:295)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:287)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)
** END NESTED EXCEPTION **
Last packet sent to the server was 206097 ms ago.
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:295)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:287)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:167)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:205)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:387)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:209)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:160)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:368)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:437)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:428)
Has anyone experienced this problem? Any advice what could i do...
Thanks
Marc Sturlese wrote:
>
> Thanks I will have a look to my JdbcDataSource. Anyway it's weird because
> using the 1.3 release I don't have that problem...
>
> Shalin Shekhar Mangar wrote:
>>
>> Yes, initially I figured that we are accidentally re-using a closed data
>> source. But Noble has pinned it right. I guess you can try looking into
>> your
>> JDBC driver's documentation for a setting which increases the connection
>> alive-ness.
>>
>> On Mon, Jan 5, 2009 at 5:29 PM, Noble Paul നോബിള് नोब्ळ् <
>> noble.paul@gmail.com> wrote:
>>
>>> I guess the indexing of a doc is taking too long (may be because of
>>> the de-dup patch) and the resultset gets closed automaticallly (timed
>>> out)
>>> --Noble
>>>
>>> On Mon, Jan 5, 2009 at 5:14 PM, Marc Sturlese <ma...@gmail.com>
>>> wrote:
>>> >
>>> > Donig this fix I get the same error :(
>>> >
>>> > I am going to try to set up the last nigthly build... let's see if I
>>> have
>>> > better luck.
>>> >
>>> > The thing is it stop indexing at the doc num 150.000 aprox... and give
>>> me
>>> > that mysql exception error... Without DeDuplication patch I can index
>>> 2
>>> > milion docs without problems...
>>> >
>>> > I am pretty lost with this... :(
>>> >
>>> >
>>> > Shalin Shekhar Mangar wrote:
>>> >>
>>> >> Yes I meant the 05/01/2008 build. The fix is a one line change
>>> >>
>>> >> Add the following as the last line of DataConfig.Entity.clearCache()
>>> >> dataSrc = null;
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Jan 5, 2009 at 4:22 PM, Marc Sturlese
>>> >> <ma...@gmail.com>wrote:
>>> >>
>>> >>>
>>> >>> Shalin you mean I should test the 05/01/2008 nighlty? maybe with
>>> this
>>> one
>>> >>> works? If the fix you did is not really big can u tell me where in
>>> the
>>> >>> source is and what is it for? (I have been debuging and tracing a
>>> lot
>>> the
>>> >>> dataimporthandler source and I I would like to know what the
>>> imporovement
>>> >>> is
>>> >>> about if it is not a problem...)
>>> >>>
>>> >>> Thanks!
>>> >>>
>>> >>>
>>> >>> Shalin Shekhar Mangar wrote:
>>> >>> >
>>> >>> > Marc, I've just committed a fix which may have caused the bug. Can
>>> you
>>> >>> use
>>> >>> > svn trunk (or the next nightly build) and confirm?
>>> >>> >
>>> >>> > On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള് नोब्ळ् <
>>> >>> > noble.paul@gmail.com> wrote:
>>> >>> >
>>> >>> >> looks like a bug w/ DIH with the recent fixes.
>>> >>> >> --Noble
>>> >>> >>
>>> >>> >> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese
>>> >>> <ma...@gmail.com>
>>> >>> >> wrote:
>>> >>> >> >
>>> >>> >> > Hey there,
>>> >>> >> > I was using the Deduplication patch with Solr 1.3 release and
>>> >>> >> everything
>>> >>> >> was
>>> >>> >> > working perfectly. Now I upgraded to a nigthly build (20th
>>> december)
>>> >>> to
>>> >>> >> be
>>> >>> >> > able to use new facet algorithm and other stuff and
>>> DeDuplication
>>> is
>>> >>> >> not
>>> >>> >> > working any more. I have followed exactly the same steps to
>>> apply
>>> >>> the
>>> >>> >> patch
>>> >>> >> > to the source code. I am geting this error:
>>> >>> >> >
>>> >>> >> > WARNING: Error reading data
>>> >>> >> > com.mysql.jdbc.CommunicationsException: Communications link
>>> failure
>>> >>> due
>>> >>> >> to
>>> >>> >> > underlying exception:
>>> >>> >> >
>>> >>> >> > ** BEGIN NESTED EXCEPTION **
>>> >>> >> >
>>> >>> >> > java.io.EOFException
>>> >>> >> >
>>> >>> >> > STACKTRACE:
>>> >>> >> >
>>> >>> >> > java.io.EOFException
>>> >>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>>> >>> >> > at
>>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
>>> >>> >> > at
>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>> >>> >> > at
>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>> >>> >> > at
>>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>> >>> >> > at
>>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>> >>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>> >>> >> >
>>> >>> >> >
>>> >>> >> > ** END NESTED EXCEPTION **
>>> >>> >> > Last packet sent to the server was 202481 ms ago.
>>> >>> >> > at
>>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>>> >>> >> > at
>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>> >>> >> > at
>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>> >>> >> > at
>>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>> >>> >> > at
>>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>> >>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>> >>> >> > Jan 5, 2009 10:06:16 AM
>>> >>> >> org.apache.solr.handler.dataimport.JdbcDataSource
>>> >>> >> > logError
>>> >>> >> > WARNING: Exception while closing result set
>>> >>> >> > com.mysql.jdbc.CommunicationsException: Communications link
>>> failure
>>> >>> due
>>> >>> >> to
>>> >>> >> > underlying exception:
>>> >>> >> >
>>> >>> >> > ** BEGIN NESTED EXCEPTION **
>>> >>> >> >
>>> >>> >> > java.io.EOFException
>>> >>> >> >
>>> >>> >> > STACKTRACE:
>>> >>> >> >
>>> >>> >> > java.io.EOFException
>>> >>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>>> >>> >> > at
>>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
>>> >>> >> > at
>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>> >>> >> > at
>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>> >>> >> > at
>>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>> >>> >> > at
>>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>> >>> >> > at
>>> >>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>>> >>> >> > at
>>> com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>>> >>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>> >>> >> >
>>> >>> >> >
>>> >>> >> > ** END NESTED EXCEPTION **
>>> >>> >> >
>>> >>> >> > Last packet sent to the server was 202482 ms ago.
>>> >>> >> > at
>>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>>> >>> >> > at
>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>> >>> >> > at
>>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>> >>> >> > at
>>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>> >>> >> > at
>>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>> >>> >> > at
>>> >>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>>> >>> >> > at
>>> com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>>> >>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>> >>> >> > at
>>> >>> >> >
>>> >>> >>
>>> >>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>> >>> >> >
>>> >>> >> > Mark do you know if I should to some extra hack to run
>>> DeDuplication
>>> >>> >> patch
>>> >>> >> > in the new solr version (nightly)? Has something changed in the
>>> >>> source
>>> >>> >> that
>>> >>> >> > make it not work or it's simply that i am doing something
>>> wrong?
>>> >>> >> >
>>> >>> >> > Thanks in advanced
>>> >>> >> > --
>>> >>> >> > View this message in context:
>>> >>> >>
>>> >>>
>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
>>> >>> >> > Sent from the Solr - User mailing list archive at Nabble.com.
>>> >>> >> >
>>> >>> >> >
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >> --
>>> >>> >> --Noble Paul
>>> >>> >>
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > --
>>> >>> > Regards,
>>> >>> > Shalin Shekhar Mangar.
>>> >>> >
>>> >>> >
>>> >>>
>>> >>> --
>>> >>> View this message in context:
>>> >>>
>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html
>>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >> Regards,
>>> >> Shalin Shekhar Mangar.
>>> >>
>>> >>
>>> >
>>> > --
>>> > View this message in context:
>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21289643.html
>>> > Sent from the Solr - User mailing list archive at Nabble.com.
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21371780.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Deduplication patch not working in nightly build
Posted by Marc Sturlese <ma...@gmail.com>.
Thanks I will have a look to my JdbcDataSource. Anyway it's weird because
using the 1.3 release I don't have that problem...
Shalin Shekhar Mangar wrote:
>
> Yes, initially I figured that we are accidentally re-using a closed data
> source. But Noble has pinned it right. I guess you can try looking into
> your
> JDBC driver's documentation for a setting which increases the connection
> alive-ness.
>
> On Mon, Jan 5, 2009 at 5:29 PM, Noble Paul നോബിള് नोब्ळ् <
> noble.paul@gmail.com> wrote:
>
>> I guess the indexing of a doc is taking too long (may be because of
>> the de-dup patch) and the resultset gets closed automaticallly (timed
>> out)
>> --Noble
>>
>> On Mon, Jan 5, 2009 at 5:14 PM, Marc Sturlese <ma...@gmail.com>
>> wrote:
>> >
>> > Donig this fix I get the same error :(
>> >
>> > I am going to try to set up the last nigthly build... let's see if I
>> have
>> > better luck.
>> >
>> > The thing is it stop indexing at the doc num 150.000 aprox... and give
>> me
>> > that mysql exception error... Without DeDuplication patch I can index 2
>> > milion docs without problems...
>> >
>> > I am pretty lost with this... :(
>> >
>> >
>> > Shalin Shekhar Mangar wrote:
>> >>
>> >> Yes I meant the 05/01/2008 build. The fix is a one line change
>> >>
>> >> Add the following as the last line of DataConfig.Entity.clearCache()
>> >> dataSrc = null;
>> >>
>> >>
>> >>
>> >> On Mon, Jan 5, 2009 at 4:22 PM, Marc Sturlese
>> >> <ma...@gmail.com>wrote:
>> >>
>> >>>
>> >>> Shalin you mean I should test the 05/01/2008 nighlty? maybe with this
>> one
>> >>> works? If the fix you did is not really big can u tell me where in
>> the
>> >>> source is and what is it for? (I have been debuging and tracing a lot
>> the
>> >>> dataimporthandler source and I I would like to know what the
>> imporovement
>> >>> is
>> >>> about if it is not a problem...)
>> >>>
>> >>> Thanks!
>> >>>
>> >>>
>> >>> Shalin Shekhar Mangar wrote:
>> >>> >
>> >>> > Marc, I've just committed a fix which may have caused the bug. Can
>> you
>> >>> use
>> >>> > svn trunk (or the next nightly build) and confirm?
>> >>> >
>> >>> > On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള് नोब्ळ् <
>> >>> > noble.paul@gmail.com> wrote:
>> >>> >
>> >>> >> looks like a bug w/ DIH with the recent fixes.
>> >>> >> --Noble
>> >>> >>
>> >>> >> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese
>> >>> <ma...@gmail.com>
>> >>> >> wrote:
>> >>> >> >
>> >>> >> > Hey there,
>> >>> >> > I was using the Deduplication patch with Solr 1.3 release and
>> >>> >> everything
>> >>> >> was
>> >>> >> > working perfectly. Now I upgraded to a nigthly build (20th
>> december)
>> >>> to
>> >>> >> be
>> >>> >> > able to use new facet algorithm and other stuff and
>> DeDuplication
>> is
>> >>> >> not
>> >>> >> > working any more. I have followed exactly the same steps to
>> apply
>> >>> the
>> >>> >> patch
>> >>> >> > to the source code. I am geting this error:
>> >>> >> >
>> >>> >> > WARNING: Error reading data
>> >>> >> > com.mysql.jdbc.CommunicationsException: Communications link
>> failure
>> >>> due
>> >>> >> to
>> >>> >> > underlying exception:
>> >>> >> >
>> >>> >> > ** BEGIN NESTED EXCEPTION **
>> >>> >> >
>> >>> >> > java.io.EOFException
>> >>> >> >
>> >>> >> > STACKTRACE:
>> >>> >> >
>> >>> >> > java.io.EOFException
>> >>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> >>> >> > at
>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
>> >>> >> > at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >>> >> > at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >>> >> > at
>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >>> >> > at
>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >>> >> >
>> >>> >> >
>> >>> >> > ** END NESTED EXCEPTION **
>> >>> >> > Last packet sent to the server was 202481 ms ago.
>> >>> >> > at
>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> >>> >> > at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >>> >> > at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >>> >> > at
>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >>> >> > at
>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >>> >> > Jan 5, 2009 10:06:16 AM
>> >>> >> org.apache.solr.handler.dataimport.JdbcDataSource
>> >>> >> > logError
>> >>> >> > WARNING: Exception while closing result set
>> >>> >> > com.mysql.jdbc.CommunicationsException: Communications link
>> failure
>> >>> due
>> >>> >> to
>> >>> >> > underlying exception:
>> >>> >> >
>> >>> >> > ** BEGIN NESTED EXCEPTION **
>> >>> >> >
>> >>> >> > java.io.EOFException
>> >>> >> >
>> >>> >> > STACKTRACE:
>> >>> >> >
>> >>> >> > java.io.EOFException
>> >>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> >>> >> > at
>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
>> >>> >> > at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >>> >> > at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >>> >> > at
>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >>> >> > at
>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >>> >> > at
>> >>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> >>> >> > at
>> com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> >>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >>> >> >
>> >>> >> >
>> >>> >> > ** END NESTED EXCEPTION **
>> >>> >> >
>> >>> >> > Last packet sent to the server was 202482 ms ago.
>> >>> >> > at
>> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> >>> >> > at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >>> >> > at
>> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >>> >> > at
>> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >>> >> > at
>> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >>> >> > at
>> >>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> >>> >> > at
>> com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> >>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >>> >> > at
>> >>> >> >
>> >>> >>
>> >>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >>> >> >
>> >>> >> > Mark do you know if I should to some extra hack to run
>> DeDuplication
>> >>> >> patch
>> >>> >> > in the new solr version (nightly)? Has something changed in the
>> >>> source
>> >>> >> that
>> >>> >> > make it not work or it's simply that i am doing something wrong?
>> >>> >> >
>> >>> >> > Thanks in advanced
>> >>> >> > --
>> >>> >> > View this message in context:
>> >>> >>
>> >>>
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
>> >>> >> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >>> >> >
>> >>> >> >
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> --Noble Paul
>> >>> >>
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Regards,
>> >>> > Shalin Shekhar Mangar.
>> >>> >
>> >>> >
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Shalin Shekhar Mangar.
>> >>
>> >>
>> >
>> > --
>> > View this message in context:
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21289643.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>> >
>>
>>
>>
>> --
>> --Noble Paul
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21290661.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Deduplication patch not working in nightly build
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Yes, initially I figured that we are accidentally re-using a closed data
source. But Noble has pinned it right. I guess you can try looking into your
JDBC driver's documentation for a setting which increases the connection
alive-ness.
On Mon, Jan 5, 2009 at 5:29 PM, Noble Paul നോബിള് नोब्ळ् <
noble.paul@gmail.com> wrote:
> I guess the indexing of a doc is taking too long (may be because of
> the de-dup patch) and the resultset gets closed automaticallly (timed
> out)
> --Noble
>
> On Mon, Jan 5, 2009 at 5:14 PM, Marc Sturlese <ma...@gmail.com>
> wrote:
> >
> > Donig this fix I get the same error :(
> >
> > I am going to try to set up the last nigthly build... let's see if I have
> > better luck.
> >
> > The thing is it stop indexing at the doc num 150.000 aprox... and give me
> > that mysql exception error... Without DeDuplication patch I can index 2
> > milion docs without problems...
> >
> > I am pretty lost with this... :(
> >
> >
> > Shalin Shekhar Mangar wrote:
> >>
> >> Yes I meant the 05/01/2008 build. The fix is a one line change
> >>
> >> Add the following as the last line of DataConfig.Entity.clearCache()
> >> dataSrc = null;
> >>
> >>
> >>
> >> On Mon, Jan 5, 2009 at 4:22 PM, Marc Sturlese
> >> <ma...@gmail.com>wrote:
> >>
> >>>
> >>> Shalin you mean I should test the 05/01/2008 nighlty? maybe with this
> one
> >>> works? If the fix you did is not really big can u tell me where in the
> >>> source is and what is it for? (I have been debuging and tracing a lot
> the
> >>> dataimporthandler source and I I would like to know what the
> imporovement
> >>> is
> >>> about if it is not a problem...)
> >>>
> >>> Thanks!
> >>>
> >>>
> >>> Shalin Shekhar Mangar wrote:
> >>> >
> >>> > Marc, I've just committed a fix which may have caused the bug. Can
> you
> >>> use
> >>> > svn trunk (or the next nightly build) and confirm?
> >>> >
> >>> > On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള് नोब्ळ् <
> >>> > noble.paul@gmail.com> wrote:
> >>> >
> >>> >> looks like a bug w/ DIH with the recent fixes.
> >>> >> --Noble
> >>> >>
> >>> >> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese
> >>> <ma...@gmail.com>
> >>> >> wrote:
> >>> >> >
> >>> >> > Hey there,
> >>> >> > I was using the Deduplication patch with Solr 1.3 release and
> >>> >> everything
> >>> >> was
> >>> >> > working perfectly. Now I upgraded to a nigthly build (20th
> december)
> >>> to
> >>> >> be
> >>> >> > able to use new facet algorithm and other stuff and DeDuplication
> is
> >>> >> not
> >>> >> > working any more. I have followed exactly the same steps to apply
> >>> the
> >>> >> patch
> >>> >> > to the source code. I am geting this error:
> >>> >> >
> >>> >> > WARNING: Error reading data
> >>> >> > com.mysql.jdbc.CommunicationsException: Communications link
> failure
> >>> due
> >>> >> to
> >>> >> > underlying exception:
> >>> >> >
> >>> >> > ** BEGIN NESTED EXCEPTION **
> >>> >> >
> >>> >> > java.io.EOFException
> >>> >> >
> >>> >> > STACKTRACE:
> >>> >> >
> >>> >> > java.io.EOFException
> >>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> >>> >> > at
> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
> >>> >> > at
> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> >>> >> > at
> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> >>> >> > at
> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> >>> >> > at
> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> >>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >>> >> >
> >>> >> >
> >>> >> > ** END NESTED EXCEPTION **
> >>> >> > Last packet sent to the server was 202481 ms ago.
> >>> >> > at
> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> >>> >> > at
> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> >>> >> > at
> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> >>> >> > at
> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> >>> >> > at
> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> >>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >>> >> > Jan 5, 2009 10:06:16 AM
> >>> >> org.apache.solr.handler.dataimport.JdbcDataSource
> >>> >> > logError
> >>> >> > WARNING: Exception while closing result set
> >>> >> > com.mysql.jdbc.CommunicationsException: Communications link
> failure
> >>> due
> >>> >> to
> >>> >> > underlying exception:
> >>> >> >
> >>> >> > ** BEGIN NESTED EXCEPTION **
> >>> >> >
> >>> >> > java.io.EOFException
> >>> >> >
> >>> >> > STACKTRACE:
> >>> >> >
> >>> >> > java.io.EOFException
> >>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> >>> >> > at
> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
> >>> >> > at
> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> >>> >> > at
> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> >>> >> > at
> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> >>> >> > at
> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> >>> >> > at
> >>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> >>> >> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> >>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >>> >> >
> >>> >> >
> >>> >> > ** END NESTED EXCEPTION **
> >>> >> >
> >>> >> > Last packet sent to the server was 202482 ms ago.
> >>> >> > at
> >>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> >>> >> > at
> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> >>> >> > at
> com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> >>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> >>> >> > at
> >>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> >>> >> > at
> >>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> >>> >> > at
> >>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> >>> >> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> >>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> >>> >> > at
> >>> >> >
> >>> >>
> >>>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >>> >> >
> >>> >> > Mark do you know if I should to some extra hack to run
> DeDuplication
> >>> >> patch
> >>> >> > in the new solr version (nightly)? Has something changed in the
> >>> source
> >>> >> that
> >>> >> > make it not work or it's simply that i am doing something wrong?
> >>> >> >
> >>> >> > Thanks in advanced
> >>> >> > --
> >>> >> > View this message in context:
> >>> >>
> >>>
> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
> >>> >> > Sent from the Solr - User mailing list archive at Nabble.com.
> >>> >> >
> >>> >> >
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> --Noble Paul
> >>> >>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Regards,
> >>> > Shalin Shekhar Mangar.
> >>> >
> >>> >
> >>>
> >>> --
> >>> View this message in context:
> >>>
> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html
> >>> Sent from the Solr - User mailing list archive at Nabble.com.
> >>>
> >>>
> >>
> >>
> >> --
> >> Regards,
> >> Shalin Shekhar Mangar.
> >>
> >>
> >
> > --
> > View this message in context:
> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21289643.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
>
>
>
> --
> --Noble Paul
>
--
Regards,
Shalin Shekhar Mangar.
Re: Deduplication patch not working in nightly build
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
I guess the indexing of a doc is taking too long (may be because of
the de-dup patch) and the resultset gets closed automaticallly (timed
out)
--Noble
On Mon, Jan 5, 2009 at 5:14 PM, Marc Sturlese <ma...@gmail.com> wrote:
>
> Donig this fix I get the same error :(
>
> I am going to try to set up the last nigthly build... let's see if I have
> better luck.
>
> The thing is it stop indexing at the doc num 150.000 aprox... and give me
> that mysql exception error... Without DeDuplication patch I can index 2
> milion docs without problems...
>
> I am pretty lost with this... :(
>
>
> Shalin Shekhar Mangar wrote:
>>
>> Yes I meant the 05/01/2008 build. The fix is a one line change
>>
>> Add the following as the last line of DataConfig.Entity.clearCache()
>> dataSrc = null;
>>
>>
>>
>> On Mon, Jan 5, 2009 at 4:22 PM, Marc Sturlese
>> <ma...@gmail.com>wrote:
>>
>>>
>>> Shalin you mean I should test the 05/01/2008 nighlty? maybe with this one
>>> works? If the fix you did is not really big can u tell me where in the
>>> source is and what is it for? (I have been debuging and tracing a lot the
>>> dataimporthandler source and I I would like to know what the imporovement
>>> is
>>> about if it is not a problem...)
>>>
>>> Thanks!
>>>
>>>
>>> Shalin Shekhar Mangar wrote:
>>> >
>>> > Marc, I've just committed a fix which may have caused the bug. Can you
>>> use
>>> > svn trunk (or the next nightly build) and confirm?
>>> >
>>> > On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള് नोब्ळ् <
>>> > noble.paul@gmail.com> wrote:
>>> >
>>> >> looks like a bug w/ DIH with the recent fixes.
>>> >> --Noble
>>> >>
>>> >> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese
>>> <ma...@gmail.com>
>>> >> wrote:
>>> >> >
>>> >> > Hey there,
>>> >> > I was using the Deduplication patch with Solr 1.3 release and
>>> >> everything
>>> >> was
>>> >> > working perfectly. Now I upgraded to a nigthly build (20th december)
>>> to
>>> >> be
>>> >> > able to use new facet algorithm and other stuff and DeDuplication is
>>> >> not
>>> >> > working any more. I have followed exactly the same steps to apply
>>> the
>>> >> patch
>>> >> > to the source code. I am geting this error:
>>> >> >
>>> >> > WARNING: Error reading data
>>> >> > com.mysql.jdbc.CommunicationsException: Communications link failure
>>> due
>>> >> to
>>> >> > underlying exception:
>>> >> >
>>> >> > ** BEGIN NESTED EXCEPTION **
>>> >> >
>>> >> > java.io.EOFException
>>> >> >
>>> >> > STACKTRACE:
>>> >> >
>>> >> > java.io.EOFException
>>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>>> >> > at
>>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
>>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>> >> > at
>>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>> >> > at
>>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>> >> >
>>> >> >
>>> >> > ** END NESTED EXCEPTION **
>>> >> > Last packet sent to the server was 202481 ms ago.
>>> >> > at
>>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>> >> > at
>>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>> >> > at
>>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>> >> > Jan 5, 2009 10:06:16 AM
>>> >> org.apache.solr.handler.dataimport.JdbcDataSource
>>> >> > logError
>>> >> > WARNING: Exception while closing result set
>>> >> > com.mysql.jdbc.CommunicationsException: Communications link failure
>>> due
>>> >> to
>>> >> > underlying exception:
>>> >> >
>>> >> > ** BEGIN NESTED EXCEPTION **
>>> >> >
>>> >> > java.io.EOFException
>>> >> >
>>> >> > STACKTRACE:
>>> >> >
>>> >> > java.io.EOFException
>>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>>> >> > at
>>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
>>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>> >> > at
>>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>> >> > at
>>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>> >> > at
>>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>>> >> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>> >> >
>>> >> >
>>> >> > ** END NESTED EXCEPTION **
>>> >> >
>>> >> > Last packet sent to the server was 202482 ms ago.
>>> >> > at
>>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>>> >> > at
>>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>>> >> > at
>>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>>> >> > at
>>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>>> >> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>>> >> > at
>>> >> >
>>> >>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>>> >> >
>>> >> > Mark do you know if I should to some extra hack to run DeDuplication
>>> >> patch
>>> >> > in the new solr version (nightly)? Has something changed in the
>>> source
>>> >> that
>>> >> > make it not work or it's simply that i am doing something wrong?
>>> >> >
>>> >> > Thanks in advanced
>>> >> > --
>>> >> > View this message in context:
>>> >>
>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
>>> >> > Sent from the Solr - User mailing list archive at Nabble.com.
>>> >> >
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> --Noble Paul
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Regards,
>>> > Shalin Shekhar Mangar.
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21289643.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Deduplication patch not working in nightly build
Posted by Marc Sturlese <ma...@gmail.com>.
Donig this fix I get the same error :(
I am going to try to set up the last nigthly build... let's see if I have
better luck.
The thing is it stop indexing at the doc num 150.000 aprox... and give me
that mysql exception error... Without DeDuplication patch I can index 2
milion docs without problems...
I am pretty lost with this... :(
Shalin Shekhar Mangar wrote:
>
> Yes I meant the 05/01/2008 build. The fix is a one line change
>
> Add the following as the last line of DataConfig.Entity.clearCache()
> dataSrc = null;
>
>
>
> On Mon, Jan 5, 2009 at 4:22 PM, Marc Sturlese
> <ma...@gmail.com>wrote:
>
>>
>> Shalin you mean I should test the 05/01/2008 nighlty? maybe with this one
>> works? If the fix you did is not really big can u tell me where in the
>> source is and what is it for? (I have been debuging and tracing a lot the
>> dataimporthandler source and I I would like to know what the imporovement
>> is
>> about if it is not a problem...)
>>
>> Thanks!
>>
>>
>> Shalin Shekhar Mangar wrote:
>> >
>> > Marc, I've just committed a fix which may have caused the bug. Can you
>> use
>> > svn trunk (or the next nightly build) and confirm?
>> >
>> > On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള് नोब्ळ् <
>> > noble.paul@gmail.com> wrote:
>> >
>> >> looks like a bug w/ DIH with the recent fixes.
>> >> --Noble
>> >>
>> >> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese
>> <ma...@gmail.com>
>> >> wrote:
>> >> >
>> >> > Hey there,
>> >> > I was using the Deduplication patch with Solr 1.3 release and
>> >> everything
>> >> was
>> >> > working perfectly. Now I upgraded to a nigthly build (20th december)
>> to
>> >> be
>> >> > able to use new facet algorithm and other stuff and DeDuplication is
>> >> not
>> >> > working any more. I have followed exactly the same steps to apply
>> the
>> >> patch
>> >> > to the source code. I am geting this error:
>> >> >
>> >> > WARNING: Error reading data
>> >> > com.mysql.jdbc.CommunicationsException: Communications link failure
>> due
>> >> to
>> >> > underlying exception:
>> >> >
>> >> > ** BEGIN NESTED EXCEPTION **
>> >> >
>> >> > java.io.EOFException
>> >> >
>> >> > STACKTRACE:
>> >> >
>> >> > java.io.EOFException
>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> >> > at
>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >> > at
>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >> > at
>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >> >
>> >> >
>> >> > ** END NESTED EXCEPTION **
>> >> > Last packet sent to the server was 202481 ms ago.
>> >> > at
>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >> > at
>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >> > at
>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >> > Jan 5, 2009 10:06:16 AM
>> >> org.apache.solr.handler.dataimport.JdbcDataSource
>> >> > logError
>> >> > WARNING: Exception while closing result set
>> >> > com.mysql.jdbc.CommunicationsException: Communications link failure
>> due
>> >> to
>> >> > underlying exception:
>> >> >
>> >> > ** BEGIN NESTED EXCEPTION **
>> >> >
>> >> > java.io.EOFException
>> >> >
>> >> > STACKTRACE:
>> >> >
>> >> > java.io.EOFException
>> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> >> > at
>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >> > at
>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >> > at
>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >> > at
>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> >> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >> >
>> >> >
>> >> > ** END NESTED EXCEPTION **
>> >> >
>> >> > Last packet sent to the server was 202482 ms ago.
>> >> > at
>> com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> >> > at
>> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> >> > at
>> com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> >> > at
>> com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> >> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >> >
>> >> > Mark do you know if I should to some extra hack to run DeDuplication
>> >> patch
>> >> > in the new solr version (nightly)? Has something changed in the
>> source
>> >> that
>> >> > make it not work or it's simply that i am doing something wrong?
>> >> >
>> >> > Thanks in advanced
>> >> > --
>> >> > View this message in context:
>> >>
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
>> >> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> --Noble Paul
>> >>
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Shalin Shekhar Mangar.
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21289643.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Deduplication patch not working in nightly build
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Yes I meant the 05/01/2008 build. The fix is a one line change
Add the following as the last line of DataConfig.Entity.clearCache()
dataSrc = null;
On Mon, Jan 5, 2009 at 4:22 PM, Marc Sturlese <ma...@gmail.com>wrote:
>
> Shalin you mean I should test the 05/01/2008 nighlty? maybe with this one
> works? If the fix you did is not really big can u tell me where in the
> source is and what is it for? (I have been debuging and tracing a lot the
> dataimporthandler source and I I would like to know what the imporovement
> is
> about if it is not a problem...)
>
> Thanks!
>
>
> Shalin Shekhar Mangar wrote:
> >
> > Marc, I've just committed a fix which may have caused the bug. Can you
> use
> > svn trunk (or the next nightly build) and confirm?
> >
> > On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള് नोब्ळ् <
> > noble.paul@gmail.com> wrote:
> >
> >> looks like a bug w/ DIH with the recent fixes.
> >> --Noble
> >>
> >> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese <ma...@gmail.com>
> >> wrote:
> >> >
> >> > Hey there,
> >> > I was using the Deduplication patch with Solr 1.3 release and
> >> everything
> >> was
> >> > working perfectly. Now I upgraded to a nigthly build (20th december)
> to
> >> be
> >> > able to use new facet algorithm and other stuff and DeDuplication is
> >> not
> >> > working any more. I have followed exactly the same steps to apply the
> >> patch
> >> > to the source code. I am geting this error:
> >> >
> >> > WARNING: Error reading data
> >> > com.mysql.jdbc.CommunicationsException: Communications link failure
> due
> >> to
> >> > underlying exception:
> >> >
> >> > ** BEGIN NESTED EXCEPTION **
> >> >
> >> > java.io.EOFException
> >> >
> >> > STACKTRACE:
> >> >
> >> > java.io.EOFException
> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> >> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> >> > at
> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> >> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >> >
> >> >
> >> > ** END NESTED EXCEPTION **
> >> > Last packet sent to the server was 202481 ms ago.
> >> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> >> > at
> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> >> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> >> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >> > Jan 5, 2009 10:06:16 AM
> >> org.apache.solr.handler.dataimport.JdbcDataSource
> >> > logError
> >> > WARNING: Exception while closing result set
> >> > com.mysql.jdbc.CommunicationsException: Communications link failure
> due
> >> to
> >> > underlying exception:
> >> >
> >> > ** BEGIN NESTED EXCEPTION **
> >> >
> >> > java.io.EOFException
> >> >
> >> > STACKTRACE:
> >> >
> >> > java.io.EOFException
> >> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> >> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> >> > at
> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> >> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> >> > at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> >> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >> >
> >> >
> >> > ** END NESTED EXCEPTION **
> >> >
> >> > Last packet sent to the server was 202482 ms ago.
> >> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> >> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> >> > at
> >> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> >> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> >> > at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> >> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> >> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> >> > at
> >> >
> >>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >> >
> >> > Mark do you know if I should to some extra hack to run DeDuplication
> >> patch
> >> > in the new solr version (nightly)? Has something changed in the source
> >> that
> >> > make it not work or it's simply that i am doing something wrong?
> >> >
> >> > Thanks in advanced
> >> > --
> >> > View this message in context:
> >>
> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
> >> > Sent from the Solr - User mailing list archive at Nabble.com.
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> --Noble Paul
> >>
> >
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: Deduplication patch not working in nightly build
Posted by Marc Sturlese <ma...@gmail.com>.
Shalin you mean I should test the 05/01/2008 nighlty? maybe with this one
works? If the fix you did is not really big can u tell me where in the
source is and what is it for? (I have been debuging and tracing a lot the
dataimporthandler source and I I would like to know what the imporovement is
about if it is not a problem...)
Thanks!
Shalin Shekhar Mangar wrote:
>
> Marc, I've just committed a fix which may have caused the bug. Can you use
> svn trunk (or the next nightly build) and confirm?
>
> On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള് नोब्ळ् <
> noble.paul@gmail.com> wrote:
>
>> looks like a bug w/ DIH with the recent fixes.
>> --Noble
>>
>> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese <ma...@gmail.com>
>> wrote:
>> >
>> > Hey there,
>> > I was using the Deduplication patch with Solr 1.3 release and
>> everything
>> was
>> > working perfectly. Now I upgraded to a nigthly build (20th december) to
>> be
>> > able to use new facet algorithm and other stuff and DeDuplication is
>> not
>> > working any more. I have followed exactly the same steps to apply the
>> patch
>> > to the source code. I am geting this error:
>> >
>> > WARNING: Error reading data
>> > com.mysql.jdbc.CommunicationsException: Communications link failure due
>> to
>> > underlying exception:
>> >
>> > ** BEGIN NESTED EXCEPTION **
>> >
>> > java.io.EOFException
>> >
>> > STACKTRACE:
>> >
>> > java.io.EOFException
>> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
>> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> > at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> > at
>> >
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> > at
>> >
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >
>> >
>> > ** END NESTED EXCEPTION **
>> > Last packet sent to the server was 202481 ms ago.
>> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> > at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> > at
>> >
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> > at
>> >
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> > Jan 5, 2009 10:06:16 AM
>> org.apache.solr.handler.dataimport.JdbcDataSource
>> > logError
>> > WARNING: Exception while closing result set
>> > com.mysql.jdbc.CommunicationsException: Communications link failure due
>> to
>> > underlying exception:
>> >
>> > ** BEGIN NESTED EXCEPTION **
>> >
>> > java.io.EOFException
>> >
>> > STACKTRACE:
>> >
>> > java.io.EOFException
>> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
>> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
>> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> > at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> > at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> > at
>> >
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> > at
>> >
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >
>> >
>> > ** END NESTED EXCEPTION **
>> >
>> > Last packet sent to the server was 202482 ms ago.
>> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
>> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
>> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
>> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
>> > at
>> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
>> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
>> > at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
>> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
>> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
>> > at
>> >
>> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
>> > at
>> >
>> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
>> > at
>> >
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
>> > at
>> >
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>> >
>> > Mark do you know if I should to some extra hack to run DeDuplication
>> patch
>> > in the new solr version (nightly)? Has something changed in the source
>> that
>> > make it not work or it's simply that i am doing something wrong?
>> >
>> > Thanks in advanced
>> > --
>> > View this message in context:
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
>> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>> >
>>
>>
>>
>> --
>> --Noble Paul
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21288939.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Deduplication patch not working in nightly build
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Marc, I've just committed a fix which may have caused the bug. Can you use
svn trunk (or the next nightly build) and confirm?
On Mon, Jan 5, 2009 at 3:10 PM, Noble Paul നോബിള് नोब्ळ् <
noble.paul@gmail.com> wrote:
> looks like a bug w/ DIH with the recent fixes.
> --Noble
>
> On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese <ma...@gmail.com>
> wrote:
> >
> > Hey there,
> > I was using the Deduplication patch with Solr 1.3 release and everything
> was
> > working perfectly. Now I upgraded to a nigthly build (20th december) to
> be
> > able to use new facet algorithm and other stuff and DeDuplication is not
> > working any more. I have followed exactly the same steps to apply the
> patch
> > to the source code. I am geting this error:
> >
> > WARNING: Error reading data
> > com.mysql.jdbc.CommunicationsException: Communications link failure due
> to
> > underlying exception:
> >
> > ** BEGIN NESTED EXCEPTION **
> >
> > java.io.EOFException
> >
> > STACKTRACE:
> >
> > java.io.EOFException
> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> > at
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> > at
> >
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> > at
> >
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >
> >
> > ** END NESTED EXCEPTION **
> > Last packet sent to the server was 202481 ms ago.
> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> > at
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> > at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> > at
> >
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> > at
> >
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> > Jan 5, 2009 10:06:16 AM org.apache.solr.handler.dataimport.JdbcDataSource
> > logError
> > WARNING: Exception while closing result set
> > com.mysql.jdbc.CommunicationsException: Communications link failure due
> to
> > underlying exception:
> >
> > ** BEGIN NESTED EXCEPTION **
> >
> > java.io.EOFException
> >
> > STACKTRACE:
> >
> > java.io.EOFException
> > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> > at
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> > at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> > at
> >
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> > at
> >
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >
> >
> > ** END NESTED EXCEPTION **
> >
> > Last packet sent to the server was 202482 ms ago.
> > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> > at
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> > at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> > at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> > at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> > at
> >
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> > at
> >
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> > at
> >
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> > at
> >
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> >
> > Mark do you know if I should to some extra hack to run DeDuplication
> patch
> > in the new solr version (nightly)? Has something changed in the source
> that
> > make it not work or it's simply that i am doing something wrong?
> >
> > Thanks in advanced
> > --
> > View this message in context:
> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
>
>
>
> --
> --Noble Paul
>
--
Regards,
Shalin Shekhar Mangar.
Re: Deduplication patch not working in nightly build
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
looks like a bug w/ DIH with the recent fixes.
--Noble
On Mon, Jan 5, 2009 at 2:36 PM, Marc Sturlese <ma...@gmail.com> wrote:
>
> Hey there,
> I was using the Deduplication patch with Solr 1.3 release and everything was
> working perfectly. Now I upgraded to a nigthly build (20th december) to be
> able to use new facet algorithm and other stuff and DeDuplication is not
> working any more. I have followed exactly the same steps to apply the patch
> to the source code. I am geting this error:
>
> WARNING: Error reading data
> com.mysql.jdbc.CommunicationsException: Communications link failure due to
> underlying exception:
>
> ** BEGIN NESTED EXCEPTION **
>
> java.io.EOFException
>
> STACKTRACE:
>
> java.io.EOFException
> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>
>
> ** END NESTED EXCEPTION **
> Last packet sent to the server was 202481 ms ago.
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> Jan 5, 2009 10:06:16 AM org.apache.solr.handler.dataimport.JdbcDataSource
> logError
> WARNING: Exception while closing result set
> com.mysql.jdbc.CommunicationsException: Communications link failure due to
> underlying exception:
>
> ** BEGIN NESTED EXCEPTION **
>
> java.io.EOFException
>
> STACKTRACE:
>
> java.io.EOFException
> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>
>
> ** END NESTED EXCEPTION **
>
> Last packet sent to the server was 202482 ms ago.
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>
> Mark do you know if I should to some extra hack to run DeDuplication patch
> in the new solr version (nightly)? Has something changed in the source that
> make it not work or it's simply that i am doing something wrong?
>
> Thanks in advanced
> --
> View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21287327.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Deduplication patch not working in nightly build
Posted by Grant Ingersoll <gs...@apache.org>.
I've seen similar errors when large background merges happen while
looping in a result set. See http://lucene.grantingersoll.com/2008/07/16/mysql-solr-and-communications-link-failure/
On Jan 9, 2009, at 12:50 PM, Mark Miller wrote:
> Your basically writing segments more often now, and somehow avoiding
> a longer merge I think. Also, likely, deduplication is probably
> adding enough extra data to your index to hit a sweet spot where a
> merge is too long. Or something to that effect - MySql is especially
> sensitive to timeouts when doing a select * on a huge db in my
> testing. I didnt understand your answer on the autocommit - I take
> it you are using it? Or no?
>
> All a guess, but it def points to a merge taking a bit long and
> causing a timeout. I think you can relax the MySql timeout settings
> if that is it.
>
> I'd like to get to the bottom of this as well, so any other info you
> can provide would be great.
>
> - Mark
>
> Marc Sturlese wrote:
>> Hey Shalin,
>>
>> In the begining (when the error was appearing) i had
>> <ramBufferSizeMB>32</ramBufferSizeMB>
>> and no maxBufferedDocs set
>>
>> Now I have:
>> <ramBufferSizeMB>32</ramBufferSizeMB>
>> <maxBufferedDocs>50</maxBufferedDocs>
>>
>> I think taht setting maxBufferedDocs to 50 I am forcing more disk
>> writting
>> than I would like... but at least it works fine (but a bit
>> slower,opiously).
>>
>> I keep saying that the most weird thing is that I don't have that
>> problem
>> using solr1.3, just with the nightly...
>>
>> Even that it's good that it works well now, would be great if
>> someone can
>> give me an explanation why this is happening
>>
>>
>> Shalin Shekhar Mangar wrote:
>>
>>> On Fri, Jan 9, 2009 at 9:23 PM, Marc Sturlese
>>> <ma...@gmail.com>wrote:
>>>
>>>
>>>> hey there,
>>>> I hadn't autoCommit set to true but I have it sorted!!!!! The error
>>>> stopped
>>>> appearing after setting the property maxBufferedDocs in
>>>> solrconfig.xml. I
>>>> can't exactly undersand why but it just worked.
>>>> Anyway, maxBufferedDocs is deprecaded, would ramBufferSizeMB do
>>>> the same?
>>>>
>>>>
>>>>
>>> What I find strange is this line in the exception:
>>> "Last packet sent to the server was 202481 ms ago."
>>>
>>> Something took very very long to complete and the connection got
>>> closed by
>>> the time the next row was fetched from the opened resultset.
>>>
>>> Just curious, what was the previous value of maxBufferedDocs and
>>> what did
>>> you change it to?
>>>
>>>
>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21374908.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>>
>>>
>>
>>
>
--------------------------
Grant Ingersoll
Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ
Re: Deduplication patch not working in nightly build
Posted by Mark Miller <ma...@gmail.com>.
Your basically writing segments more often now, and somehow avoiding a
longer merge I think. Also, likely, deduplication is probably adding
enough extra data to your index to hit a sweet spot where a merge is too
long. Or something to that effect - MySql is especially sensitive to
timeouts when doing a select * on a huge db in my testing. I didnt
understand your answer on the autocommit - I take it you are using it?
Or no?
All a guess, but it def points to a merge taking a bit long and causing
a timeout. I think you can relax the MySql timeout settings if that is it.
I'd like to get to the bottom of this as well, so any other info you can
provide would be great.
- Mark
Marc Sturlese wrote:
> Hey Shalin,
>
> In the begining (when the error was appearing) i had
> <ramBufferSizeMB>32</ramBufferSizeMB>
> and no maxBufferedDocs set
>
> Now I have:
> <ramBufferSizeMB>32</ramBufferSizeMB>
> <maxBufferedDocs>50</maxBufferedDocs>
>
> I think taht setting maxBufferedDocs to 50 I am forcing more disk writting
> than I would like... but at least it works fine (but a bit slower,opiously).
>
> I keep saying that the most weird thing is that I don't have that problem
> using solr1.3, just with the nightly...
>
> Even that it's good that it works well now, would be great if someone can
> give me an explanation why this is happening
>
>
>
> Shalin Shekhar Mangar wrote:
>
>> On Fri, Jan 9, 2009 at 9:23 PM, Marc Sturlese
>> <ma...@gmail.com>wrote:
>>
>>
>>> hey there,
>>> I hadn't autoCommit set to true but I have it sorted!!!!! The error
>>> stopped
>>> appearing after setting the property maxBufferedDocs in solrconfig.xml. I
>>> can't exactly undersand why but it just worked.
>>> Anyway, maxBufferedDocs is deprecaded, would ramBufferSizeMB do the same?
>>>
>>>
>>>
>> What I find strange is this line in the exception:
>> "Last packet sent to the server was 202481 ms ago."
>>
>> Something took very very long to complete and the connection got closed by
>> the time the next row was fetched from the opened resultset.
>>
>> Just curious, what was the previous value of maxBufferedDocs and what did
>> you change it to?
>>
>>
>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21374908.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>>
>
>
Re: Deduplication patch not working in nightly build
Posted by Marc Sturlese <ma...@gmail.com>.
Hey Mark,
Sorry I was not enough especific, I wanted to mean that I have and I always
had autoCommit=false.
I will do some more traces and test. Will post if I have any new important
thing to mention.
Thanks.
Marc Sturlese wrote:
>
> Hey Shalin,
>
> In the begining (when the error was appearing) i had
> <ramBufferSizeMB>32</ramBufferSizeMB>
> and no maxBufferedDocs set
>
> Now I have:
> <ramBufferSizeMB>32</ramBufferSizeMB>
> <maxBufferedDocs>50</maxBufferedDocs>
>
> I think taht setting maxBufferedDocs to 50 I am forcing more disk writting
> than I would like... but at least it works fine (but a bit
> slower,opiously).
>
> I keep saying that the most weird thing is that I don't have that problem
> using solr1.3, just with the nightly...
>
> Even that it's good that it works well now, would be great if someone can
> give me an explanation why this is happening
>
>
>
> Shalin Shekhar Mangar wrote:
>>
>> On Fri, Jan 9, 2009 at 9:23 PM, Marc Sturlese
>> <ma...@gmail.com>wrote:
>>
>>>
>>> hey there,
>>> I hadn't autoCommit set to true but I have it sorted!!!!! The error
>>> stopped
>>> appearing after setting the property maxBufferedDocs in solrconfig.xml.
>>> I
>>> can't exactly undersand why but it just worked.
>>> Anyway, maxBufferedDocs is deprecaded, would ramBufferSizeMB do the
>>> same?
>>>
>>>
>> What I find strange is this line in the exception:
>> "Last packet sent to the server was 202481 ms ago."
>>
>> Something took very very long to complete and the connection got closed
>> by
>> the time the next row was fetched from the opened resultset.
>>
>> Just curious, what was the previous value of maxBufferedDocs and what did
>> you change it to?
>>
>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21374908.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21378069.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Deduplication patch not working in nightly build
Posted by Marc Sturlese <ma...@gmail.com>.
Hey Shalin,
In the begining (when the error was appearing) i had
<ramBufferSizeMB>32</ramBufferSizeMB>
and no maxBufferedDocs set
Now I have:
<ramBufferSizeMB>32</ramBufferSizeMB>
<maxBufferedDocs>50</maxBufferedDocs>
I think taht setting maxBufferedDocs to 50 I am forcing more disk writting
than I would like... but at least it works fine (but a bit slower,opiously).
I keep saying that the most weird thing is that I don't have that problem
using solr1.3, just with the nightly...
Even that it's good that it works well now, would be great if someone can
give me an explanation why this is happening
Shalin Shekhar Mangar wrote:
>
> On Fri, Jan 9, 2009 at 9:23 PM, Marc Sturlese
> <ma...@gmail.com>wrote:
>
>>
>> hey there,
>> I hadn't autoCommit set to true but I have it sorted!!!!! The error
>> stopped
>> appearing after setting the property maxBufferedDocs in solrconfig.xml. I
>> can't exactly undersand why but it just worked.
>> Anyway, maxBufferedDocs is deprecaded, would ramBufferSizeMB do the same?
>>
>>
> What I find strange is this line in the exception:
> "Last packet sent to the server was 202481 ms ago."
>
> Something took very very long to complete and the connection got closed by
> the time the next row was fetched from the opened resultset.
>
> Just curious, what was the previous value of maxBufferedDocs and what did
> you change it to?
>
>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21374908.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21376235.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Deduplication patch not working in nightly build
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Fri, Jan 9, 2009 at 9:23 PM, Marc Sturlese <ma...@gmail.com>wrote:
>
> hey there,
> I hadn't autoCommit set to true but I have it sorted!!!!! The error stopped
> appearing after setting the property maxBufferedDocs in solrconfig.xml. I
> can't exactly undersand why but it just worked.
> Anyway, maxBufferedDocs is deprecaded, would ramBufferSizeMB do the same?
>
>
What I find strange is this line in the exception:
"Last packet sent to the server was 202481 ms ago."
Something took very very long to complete and the connection got closed by
the time the next row was fetched from the opened resultset.
Just curious, what was the previous value of maxBufferedDocs and what did
you change it to?
>
> --
> View this message in context:
> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21374908.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: Deduplication patch not working in nightly build
Posted by Marc Sturlese <ma...@gmail.com>.
hey there,
I hadn't autoCommit set to true but I have it sorted!!!!! The error stopped
appearing after setting the property maxBufferedDocs in solrconfig.xml. I
can't exactly undersand why but it just worked.
Anyway, maxBufferedDocs is deprecaded, would ramBufferSizeMB do the same?
Thanks
Marc Sturlese wrote:
>
> Hey there,
> I was using the Deduplication patch with Solr 1.3 release and everything
> was working perfectly. Now I upgraded to a nigthly build (20th december)
> to be able to use new facet algorithm and other stuff and DeDuplication is
> not working any more. I have followed exactly the same steps to apply the
> patch to the source code. I am geting this error:
>
> WARNING: Error reading data
> com.mysql.jdbc.CommunicationsException: Communications link failure due to
> underlying exception:
>
> ** BEGIN NESTED EXCEPTION **
>
> java.io.EOFException
>
> STACKTRACE:
>
> java.io.EOFException
> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2404)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>
>
> ** END NESTED EXCEPTION **
> Last packet sent to the server was 202481 ms ago.
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.ResultSet.next(ResultSet.java:6144)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:294)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
> Jan 5, 2009 10:06:16 AM org.apache.solr.handler.dataimport.JdbcDataSource
> logError
> WARNING: Exception while closing result set
> com.mysql.jdbc.CommunicationsException: Communications link failure due to
> underlying exception:
>
> ** BEGIN NESTED EXCEPTION **
>
> java.io.EOFException
>
> STACKTRACE:
>
> java.io.EOFException
> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>
>
> ** END NESTED EXCEPTION **
>
> Last packet sent to the server was 202482 ms ago.
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2563)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
> at
> com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
> at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
> at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:150)
> at com.mysql.jdbc.ResultSet.realClose(ResultSet.java:6488)
> at com.mysql.jdbc.ResultSet.close(ResultSet.java:736)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.close(JdbcDataSource.java:312)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:304)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$400(JdbcDataSource.java:189)
> at
> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:225)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:76)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:351)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:193)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:144)
> at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:407)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:388)
>
> Mark do you know if I should to some extra hack to run DeDuplication patch
> in the new solr version (nightly)? Has something changed in the source
> that make it not work or it's simply that i am doing something wrong?
> I have been checking DirectUpdateHandler2 and have seen few diferences
> compared with the release 1.3... Don't know if the key is there...
>
> Thanks in advanced
>
--
View this message in context: http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21374908.html
Sent from the Solr - User mailing list archive at Nabble.com.