How to deal with "java.net.SocketTimeoutException: Read timed out" on commit?

[Geir Gullestad Pettersen <ge...@gmail.com>]

Hi,

We're getting a lot of these timeouts during bulk feeding or a large
document set. We're sending batches of 1000 documents and commiting every 15
minutes or for every 10.000 docs, whichever happens first. We find that the
first few commits (after 10'/20'/30' docs) go through without exceptions
(although the "CommonsHttpSolrServer.soTimeout" is set quite high -
120000ms). The time for a commit to complete seems to increase for every
subsequent commit request, and eventually the timeout value is reached
causing Solrj to throw the java.net.SocketTimeoutException. Increasing the
soTimeout value only delays the problem and it will allow us to send a few
more commits before seeing the exception.

I've noticed that even though we get these exceptions on commit, documents
seem to be searchable after some time. Does that mean that the commit
process in solr continues even though the socket times out?

We're also setting "waitFlush=false" and  "waitSearcher=false" when
committing, which I would expect to enable a non-blocking commit request to
solr, but on n+1 commits this doesn't aleays happen and this when we're
getting the timeouts.

I suspect that solr.commit(false,false) is a non-blocking call only if a all
previous commits are complete. Correct?

I'm aware that commiting often is not adviced as it is a resouce intensive
process for solr, however it is a requirement that documents are searchable
before the entire document set has been added to solr. Our workaround is to
limit the amount of commits by time and document count "triggers" in our
code. We cannot use the autocommit feature in solrconfig.xml because we also
need to be able to issue immediate commits for incremental feeding.

So, my main question is regarding the severity of
"java.net.SocketTimeoutException: Read timed out" on commit. Should I worry
and try to re-issue commit when this happens, or is it safe to ignore it?


Thanks for any input!

Cheers,
Geir

Re: How to deal with "java.net.SocketTimeoutException: Read timed out" on commit?

[lboutros <bo...@gmail.com>]

We had this type of error too.
Now we are using the StreamingUpdateSolrServer with a quite big queue and
2-4 threads depending on data type:

http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.html

And we do not do any intermediate commit. We send only one commit at the end
of  the process and now everything is fine.

Ludovic.

-----
Jouve
France.
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-deal-with-java-net-SocketTimeoutException-Read-timed-out-on-commit-tp3258430p3259890.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to deal with "java.net.SocketTimeoutException: Read timed out" on commit?

[Manish Bafna <ma...@gmail.com>]

We too were getting same issue.
We solved it by ensuring that when commit is in progress, no one access the
index.

Though SOLR's UpdateRequest does it, we will still read timeout issues
because of CommonsHttpSolrServer.

If we have another layer which doesnt send the request itself, then you wont
get this error.

On Tue, Aug 16, 2011 at 5:34 PM, Geir Gullestad Pettersen
<ge...@gmail.com>wrote:

> Hi,
>
> We're getting a lot of these timeouts during bulk feeding or a large
> document set. We're sending batches of 1000 documents and commiting every
> 15
> minutes or for every 10.000 docs, whichever happens first. We find that the
> first few commits (after 10'/20'/30' docs) go through without exceptions
> (although the "CommonsHttpSolrServer.soTimeout" is set quite high -
> 120000ms). The time for a commit to complete seems to increase for every
> subsequent commit request, and eventually the timeout value is reached
> causing Solrj to throw the java.net.SocketTimeoutException. Increasing the
> soTimeout value only delays the problem and it will allow us to send a few
> more commits before seeing the exception.
>
> I've noticed that even though we get these exceptions on commit, documents
> seem to be searchable after some time. Does that mean that the commit
> process in solr continues even though the socket times out?
>
> We're also setting "waitFlush=false" and  "waitSearcher=false" when
> committing, which I would expect to enable a non-blocking commit request to
> solr, but on n+1 commits this doesn't aleays happen and this when we're
> getting the timeouts.
>
> I suspect that solr.commit(false,false) is a non-blocking call only if a
> all
> previous commits are complete. Correct?
>
> I'm aware that commiting often is not adviced as it is a resouce intensive
> process for solr, however it is a requirement that documents are searchable
> before the entire document set has been added to solr. Our workaround is to
> limit the amount of commits by time and document count "triggers" in our
> code. We cannot use the autocommit feature in solrconfig.xml because we
> also
> need to be able to issue immediate commits for incremental feeding.
>
> So, my main question is regarding the severity of
> "java.net.SocketTimeoutException: Read timed out" on commit. Should I worry
> and try to re-issue commit when this happens, or is it safe to ignore it?
>
>
> Thanks for any input!
>
> Cheers,
> Geir
>