dataimport.properties still updated on error

[Barry Harding <ba...@systemax.co.uk>]

Hi I am currently using the DIH to connect to and import data from a MS SQL Server, and in general doing full, delta or deletes seems to work perfectly.

The issue is that I spotted some errors being logged in the tomcat logs for SOLR which are :

19-Sep-2011 07:45:25 org.apache.solr.common.SolrException log
SEVERE: Exception in entity : product:org.apache.solr.handler.dataimport.DataImportHandlerException: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 125) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
	at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:64)
	at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:339)
	at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(JdbcDataSource.java:228)
	at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:262)
	at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:77)
	at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:75)
	at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:238)
	at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:591)
	at org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:302)
	at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:178)
	at org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:390)
	at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:429)
	at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 125) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:213)
	at com.microsoft.sqlserver.jdbc.SQLServerResultSet$FetchBuffer.nextRow(SQLServerResultSet.java:4713)
	at com.microsoft.sqlserver.jdbc.SQLServerResultSet.fetchBufferNext(SQLServerResultSet.java:1671)
	at com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:944)
	at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:331)
	... 11 more

19-Sep-2011 07:45:25 org.apache.solr.handler.dataimport.DocBuilder doDelta
INFO: Delta Import completed successfully
19-Sep-2011 07:45:25 org.apache.solr.handler.dataimport.DocBuilder finish


Now this SQL error I can deal with and I will probably switch to snapshot Isolation as these are constantly updated tables, but my issue is not the sql error but the fact that the delta import still reported that it had imported successfully and still wrote out the last updated time to the dataimport.properties file, so the next time it ran it missed a bunch of documents that should have been indexed.

If it had failed and just rolled back the changes and not updated the dataimport.properties file it would (assuming no more deadlocks) have caught all of the missed documents on the next delta import.

My connection to MS SQL is using the "responseBuffering=adaptive" setting to reduce memory overhead, So I guess what I am asking is there any way I can cause the DIH to roll back the import if an error occurs and to not update the "dataimport.properties" file.

Any help or suggestions would be appreciated

Thanks

Barry H
 
DISCLAIMER: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Misco UK Ltd. Any unauthorised use or dissemination of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation. Misco UK Ltd, registered in Scotland Number 114143. Registered Office: Caledonian Exchange, 19a Canning Street, Edinburgh EH3 8EG. Telephone +44 (0)1933 686000. This e-mail message has been scanned by CA Gateway Security.