Solr 3.5 DIH Delta Import hanging

[Arcadius Ahouansou <ar...@menelic.com>]

Hello.

 We have a Solr 3.5 master randomly hanging during delta import.
It does not happen very often.

When it does, the output of
dataimport?command=status
is shown below.

After each page refresh, the only thing that changes is the "Time Elapsed".

I have also done a thread dump (see below).
It shows that the DIH is hanging on
java.net.SocketInputStream.socketRead

As the default DIH DataSource
connectionTimeout and readTimeout
seem to have no effect, I have started having a look at setting the
following system properties:
- sun.net.client.defaultConnectTimeout
- sun.net.client.defaultReadTimeout

or Oracle own

- oracle.net.READ_TIMEOUT
- oracle.jdbc.ReadTimeout


Please, is there any better way of solving this issue?

Thank you very much.

Arcadius.


DIH Status page:

<str name="command">status</str>

<str name="status">busy</str>

<str name="importResponse">A command is still running...</str>

<lst name="statusMessages">

<str name="Time Elapsed">48:31:22.998</str>

<str name="Total Requests made to DataSource">173708</str>

<str name="Total Rows Fetched">1385636</str>

<str name="Total Documents Skipped">0</str>

<str name="Delta Dump started">2012-11-10 16:35:36</str>

<str name="Identifying Delta">2012-11-10 16:35:36</str>

<str name="Deltas Obtained">2012-11-10 16:38:29</str>

<str name="Building documents">2012-11-10 16:38:29</str>

<str name="Total Changed Documents">1026622</str></lst>






Thread Dump:

> No deadlocks found.
>
> Thread 16873: (state = IN_NATIVE)
>
>  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>
>  - java.net.SocketInputStream.read(byte[], int, int) @bci=84, line=129
> (Compiled frame)
>
>  - oracle.net.ns.Packet.receive() @bci=31, line=240 (Compiled frame)
>
>  - oracle.net.ns.DataPacket.receive() @bci=1, line=92 (Compiled frame)
>
>  - oracle.net.ns.NetInputStream.getNextPacket() @bci=48, line=172
> (Compiled frame)
>
>  - oracle.net.ns.NetInputStream.read(byte[], int, int) @bci=33, line=117
> (Compiled frame)
>
>  - oracle.jdbc.driver.T4CMAREngine.getNBytes(byte[], int, int) @bci=15,
> line=1395 (Compiled frame)
>
>  - oracle.jdbc.driver.T4CMAREngine.unmarshalNBytes(byte[], int, int) @bci=21,
> line=1368 (Compiled frame)
>
>  - oracle.jdbc.driver.T4CMAREngine.unmarshalBuffer(byte[], int, int) @bci=50,
> line=1906 (Compiled frame)
>
>  - oracle.jdbc.driver.T4CMAREngine.unmarshalCLR(byte[], int, int[], int,
> int) @bci=178, line=1672 (Compiled frame)
>
>  - oracle.jdbc.driver.T4CMAREngine.unmarshalCLR(byte[], int, int[], int)
> @bci=7, line=1589 (Compiled frame)
>
>  - oracle.jdbc.driver.T4CMAREngine.unmarshalCLR(byte[], int, int[]) @bci=6,
> line=1582 (Compiled frame)
>
>  - oracle.jdbc.driver.T4CVarcharAccessor.unmarshalOneRow() @bci=214,
> line=206 (Compiled frame)
>
>  - oracle.jdbc.driver.T4CTTIrxd.unmarshal(oracle.jdbc.driver.Accessor[],
> int, int) @bci=167, line=895 (Compiled frame)
>
>  - oracle.jdbc.driver.T4C8Oall.receive() @bci=514, line=713 (Compiled
> frame)
>
>  - oracle.jdbc.driver.T4CStatement.doOall8(boolean, boolean, boolean,
> boolean) @bci=653, line=183 (Compiled frame)
>
>  - oracle.jdbc.driver.T4CStatement.executeForDescribe() @bci=39, line=780
> (Compiled frame)
>
>  - oracle.jdbc.driver.T4CStatement.executeMaybeDescribe() @bci=104,
> line=855 (Interpreted frame)
>
>  - oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout() @bci=139,
> line=1186 (Interpreted frame)
>
>  - oracle.jdbc.driver.OracleStatement.executeInternal(java.lang.String)
> @bci=62, line=1770 (Interpreted frame)
>
>  - oracle.jdbc.driver.OracleStatement.execute(java.lang.String) @bci=13,
> line=1739 (Interpreted frame)
>
>  - oracle.jdbc.driver.OracleStatementWrapper.execute(java.lang.String)
> @bci=5, line=299 (Interpreted frame)
>
>  - org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.
> <init>(org.apache.solr.handler.dataimport.JdbcDataSource,
> java.lang.String) @bci=98, line=246 (Interpreted frame)
>
>  -
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(java.lang.String)
> @bci=6, line=210 (Interpreted frame)
>
>  -
> org.apache.solr.handler.dataimport.JdbcDataSource.getData(java.lang.String)
> @bci=2, line=39 (Interpreted frame)
>
>  -
> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(java.lang.String)
> @bci=19, line=59 (Compiled frame)
>
>  - org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow() @bci=21,
> line=73 (Compiled frame)
>
>  - org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow()
> @bci=18, line=238 (Compiled frame)
>
>  -
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(org.apache.solr.handler.dataimport.VariableResolverImpl,
> org.apache.solr.handler.dataimport.DocBuilder$DocWrapper, java.util.Map,
> org.apache.solr.handler.dataimport.DataConfig$Entity, boolean,
> org.apache.solr.handler.dataimport.ContextImpl) @bci=297, line=596
> (Compiled frame)
>
>  - org.apache.solr.handler.dataimport.DocBuilder.doDelta() @bci=196,
> line=303 (Compiled frame)
>
>  - org.apache.solr.handler.dataimport.DocBuilder.execute() @bci=363,
> line=179 (Interpreted frame)
>
>  -
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(org.apache.solr.handler.dataimport.SolrWriter,
> org.apache.solr.handler.dataimport.DataImporter$RequestParams) @bci=51,
> line=390 (Interpreted frame)
>
>  -
> org.apache.solr.handler.dataimport.DataImporter.runCmd(org.apache.solr.handler.dataimport.DataImporter$RequestParams,
> org.apache.solr.handler.dataimport.SolrWriter) @bci=89, line=429
> (Interpreted frame)
>
>  - org.apache.solr.handler.dataimport.DataImporter$1.run() @bci=12,
> line=408 (Interpreted frame)
>
>
>