ConcurrentUpdateSolrServer hanging

[qungg <qz...@gmail.com>]

Hi,

I'm using concurrentUpdateSolrServer to do my incremental indexing nightly.
I have 50 shards to index into, about 10,000 documents each night. I start
one concurrentUpdateSolrServer on each shards and start to send documents.
The queue size for concurrentUpdateSolrServer is 100, and 4 threads. At the
end of the import, i will send commit using the same
concurrentUpdateSolrServer. The problem is some of the
concurrentUpdateSolrServer is not sending the commit to the shards and the
import task hangs for a couple hours. 

So I looked at the log and find out that the shards received about 1000
document couple hours later following with a commit. Is there anything
methods I can call to flush out documents before I send the commit? Or are
there any existing issue related to concurrentUpdateSolrServer related to
this?

Thanks,
Qun



--
View this message in context: http://lucene.472066.n3.nabble.com/ConcurrentUpdateSolrServer-hanging-tp4073620.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: ConcurrentUpdateSolrServer hanging

[qungg <qz...@gmail.com>]

Hi Michael,

I realized that I might have to use blockUntilFinished before commit, but do
I have to use shutdown as well??

Thanks,
Qun



--
View this message in context: http://lucene.472066.n3.nabble.com/ConcurrentUpdateSolrServer-hanging-tp4073620p4073651.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: ConcurrentUpdateSolrServer hanging

[qungg <qz...@gmail.com>]

Hi,

BlockUntilFinish block indefinitely sometimes. But if I send a commit from
another thread to the instance, the concurrentUpdateServer unblock and send
the rest of the documents and commit. So the squence look like this:

1. adding documents as usual...
2. finish adding documents...
3. block untill finished... block forever (i try to block before commit,
call this commit 1)
4. from other thread, send a commit (lets call this commit 2)
5. magically unblocked... and flushed out the rest of the documents...
6. commit 1...  
7. commit 2 ... 

The order of commit in 6 and 7 is observed in solr log.

Thanks,
Qun




--
View this message in context: http://lucene.472066.n3.nabble.com/ConcurrentUpdateSolrServer-hanging-tp4073620p4074366.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: ConcurrentUpdateSolrServer hanging

[Shawn Heisey <so...@elyograg.org>]

On 6/27/2013 9:32 AM, Michael Della Bitta wrote:
> Are you using blockUntilFinished() and/or shutdown()?
>
> One of the things to note is that a commit is just another "document," so
> writing a commit into the queue of the ConcurrentUpdateSolrServer isn't
> enough to get it flushed out.

ConcurrentUpdateSolrServer contains this little bit of code:

     // this happens for commit...
     if (req.getDocuments() == null || req.getDocuments().isEmpty()) {
       blockUntilFinished();
       return server.request(request);
     }

Unless the comment is incorrect or there's a bug, sending a commit() 
will inherently do the blockUntilFinished().

Thanks,
Shawn

Re: ConcurrentUpdateSolrServer hanging

[Michael Della Bitta <mi...@appinions.com>]

Qun,

Are you using blockUntilFinished() and/or shutdown()?

One of the things to note is that a commit is just another "document," so
writing a commit into the queue of the ConcurrentUpdateSolrServer isn't
enough to get it flushed out.


Michael Della Bitta

Applications Developer

o: +1 646 532 3062  | c: +1 917 477 7906

appinions inc.

“The Science of Influence Marketing”

18 East 41st Street

New York, NY 10017

t: @appinions <https://twitter.com/Appinions> | g+:
plus.google.com/appinions
w: appinions.com <http://www.appinions.com/>


On Thu, Jun 27, 2013 at 10:21 AM, qungg <qz...@gmail.com> wrote:

> Hi,
>
> I'm using concurrentUpdateSolrServer to do my incremental indexing nightly.
> I have 50 shards to index into, about 10,000 documents each night. I start
> one concurrentUpdateSolrServer on each shards and start to send documents.
> The queue size for concurrentUpdateSolrServer is 100, and 4 threads. At the
> end of the import, i will send commit using the same
> concurrentUpdateSolrServer. The problem is some of the
> concurrentUpdateSolrServer is not sending the commit to the shards and the
> import task hangs for a couple hours.
>
> So I looked at the log and find out that the shards received about 1000
> document couple hours later following with a commit. Is there anything
> methods I can call to flush out documents before I send the commit? Or are
> there any existing issue related to concurrentUpdateSolrServer related to
> this?
>
> Thanks,
> Qun
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/ConcurrentUpdateSolrServer-hanging-tp4073620.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>