ConcurrentUpdateSolrserver - Queue size not working

[Learner <bb...@gmail.com>]

I am using ConcurrentUpdateSolrserver to create 4 threads (threadCount=4)
with queueSize of 30000.

Indexing works fine as expected.

My issue is that, I see that the documents are getting adding to server even
before it reaches the queue size.  Am I doing anything wrong? Or is
queuesize not implemented yet?

Also I dont see a very big performance improvements when I increase /
decrease the number of threads. Can someone let me know the best way to
improve indexing performance when using ConcurrentUpdateSolrserver

FYI... I am running this program on 4 core machine.. 

Sample snippet:

		ConcurrentUpdateSolrServer server = new ConcurrentUpdateSolrServer(
				solrServer, 30000, 4);
try {
			while ((line = bReader.readLine()) != null) {
				inputDocument = line.split("\t");
                          Do some processing....
			server.add(doc);

}}




--
View this message in context: http://lucene.472066.n3.nabble.com/ConcurrentUpdateSolrserver-Queue-size-not-working-tp4071408.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: ConcurrentUpdateSolrserver - Queue size not working

[Shawn Heisey <so...@elyograg.org>]

On 6/18/2013 11:06 AM, Learner wrote:
> My issue is that, I see that the documents are getting adding to server even
> before it reaches the queue size.  Am I doing anything wrong? Or is
> queuesize not implemented yet?
>
> Also I dont see a very big performance improvements when I increase /
> decrease the number of threads. Can someone let me know the best way to
> improve indexing performance when using ConcurrentUpdateSolrserver
>
> FYI... I am running this program on 4 core machine..
>
> Sample snippet:
>
> 		ConcurrentUpdateSolrServer server = new ConcurrentUpdateSolrServer(
> 				solrServer, 30000, 4);
> try {
> 			while ((line = bReader.readLine()) != null) {
> 				inputDocument = line.split("\t");
>                            Do some processing....
> 			server.add(doc);
>
> }}

It looks like the Javadocs for this class are misleading.  The queuesize 
value doesn't cause documents to buffer until that many are present, it 
is used to limit the size of the internal queue.  The object will always 
begin indexing immediately.

I notice that you have your add() call within a try block.  You should 
know that CUSS will *never* throw an exception for things like your 
server being down or the update being badly formed.  It always returns 
immediately with a success.  I did put information about this in the 
javadocs for its predecessor StreamingUpdateSolrServer, but it looks 
like we need an update for CUSS.

If you aren't seeing performance increases from increasing threads, then 
it is very likely that Solr is keeping up with your application with no 
problems and that your application, or the place your application gets 
its data, is the bottleneck.

Thanks,
Shawn

RE: ConcurrentUpdateSolrserver - Queue size not working

[James Thomas <JT...@Camstar.com>]

Looks like the javadoc  on this parameter could use a little tweaking.
>From looking at the 4.3 source code (hoping I get this right :-), it appears the ConcurrentUpdateSolrServer will begin sending documents (on a single thread) as soon as the first document is added.
New threads (up to threadCount) are created only when a document is added and the queue is more than half full.
Kind of makes sense... why wait until the queue is full to send documents.  And if one thread can keep up with your ETL (adds), there's really no need to create new threads.

You might want to create your own buffer (e.g. ArrayList) of the SolrInputDocument objects and then use the "add" API that accepts the collection.
Calling "add" after creating 30,000 SolrInputDocument objects seems a bit much.  Something smaller (like 1,000) might work better.  You'll have to experiment to see what works best for your environment.

-- James

-----Original Message-----
From: Learner [mailto:bbarani@gmail.com] 
Sent: Tuesday, June 18, 2013 1:07 PM
To: solr-user@lucene.apache.org
Subject: ConcurrentUpdateSolrserver - Queue size not working

I am using ConcurrentUpdateSolrserver to create 4 threads (threadCount=4) with queueSize of 30000.

Indexing works fine as expected.

My issue is that, I see that the documents are getting adding to server even before it reaches the queue size.  Am I doing anything wrong? Or is queuesize not implemented yet?

Also I dont see a very big performance improvements when I increase / decrease the number of threads. Can someone let me know the best way to improve indexing performance when using ConcurrentUpdateSolrserver

FYI... I am running this program on 4 core machine.. 

Sample snippet:

		ConcurrentUpdateSolrServer server = new ConcurrentUpdateSolrServer(
				solrServer, 30000, 4);
try {
			while ((line = bReader.readLine()) != null) {
				inputDocument = line.split("\t");
                          Do some processing....
			server.add(doc);

}}




--
View this message in context: http://lucene.472066.n3.nabble.com/ConcurrentUpdateSolrserver-Queue-size-not-working-tp4071408.html
Sent from the Solr - User mailing list archive at Nabble.com.