You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Andy <an...@yahoo.com> on 2010/09/22 11:52:36 UTC

is indexing single-threaded?

Does Solr index data in a single thread or can data be indexed concurrently in multiple threads?

Thanks
Andy


      

Re: is indexing single-threaded?

Posted by Dennis Gearon <ge...@sbcglobal.net>.
I was kind of wondering what magic had been done to achieve multiple writing to the index file :-)

BTW, wouldn't it be possible to have seperate segments per thread? Set up the index with a minimum (desired?) segment count, and write each individually?

Is there any organization in the segments? Or can adjacent data be found in different segments?

I seem to remember that the new stuff gets committed to its own segment until some sort of 'consolidate' command takes place.


Dennis Gearon

Signature Warning
----------------
EARTH has a Right To Life,
  otherwise we all die.

Read 'Hot, Flat, and Crowded'
Laugh at http://www.yert.com/film.php


--- On Thu, 9/23/10, Jan Høydahl / Cominvent <ja...@cominvent.com> wrote:

> From: Jan Høydahl / Cominvent <ja...@cominvent.com>
> Subject: Re: is indexing single-threaded?
> To: solr-user@lucene.apache.org
> Date: Thursday, September 23, 2010, 1:42 AM
> SolrJ threads speeds up feeding
> throughput. The building the index is still single threaded
> (per core), isn't it? Don't know about analysis. But you
> cannot have two threads write to the same file...
> 
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> 
> On 23. sep. 2010, at 08.01, Ryan McKinley wrote:
> 
> > Multiple threads work well.
> > 
> > If you are using solrj, check the StreamingSolrServer
> for an
> > implementation that will keep X number of threads
> busy.
> > 
> > Your mileage will very, but in general I find a
> reasonable thread
> > count is ~ (number of cores)+1
> > 
> > 
> > On Wed, Sep 22, 2010 at 5:52 AM, Andy <an...@yahoo.com>
> wrote:
> >> Does Solr index data in a single thread or can
> data be indexed concurrently in multiple threads?
> >> 
> >> Thanks
> >> Andy
> >> 
> >> 
> >> 
> >> 
> 
> 

Re: is indexing single-threaded?

Posted by Jan Høydahl / Cominvent <ja...@cominvent.com>.
SolrJ threads speeds up feeding throughput. The building the index is still single threaded (per core), isn't it? Don't know about analysis. But you cannot have two threads write to the same file...

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

On 23. sep. 2010, at 08.01, Ryan McKinley wrote:

> Multiple threads work well.
> 
> If you are using solrj, check the StreamingSolrServer for an
> implementation that will keep X number of threads busy.
> 
> Your mileage will very, but in general I find a reasonable thread
> count is ~ (number of cores)+1
> 
> 
> On Wed, Sep 22, 2010 at 5:52 AM, Andy <an...@yahoo.com> wrote:
>> Does Solr index data in a single thread or can data be indexed concurrently in multiple threads?
>> 
>> Thanks
>> Andy
>> 
>> 
>> 
>> 


Re: is indexing single-threaded?

Posted by Ryan McKinley <ry...@gmail.com>.
Multiple threads work well.

If you are using solrj, check the StreamingSolrServer for an
implementation that will keep X number of threads busy.

Your mileage will very, but in general I find a reasonable thread
count is ~ (number of cores)+1


On Wed, Sep 22, 2010 at 5:52 AM, Andy <an...@yahoo.com> wrote:
> Does Solr index data in a single thread or can data be indexed concurrently in multiple threads?
>
> Thanks
> Andy
>
>
>
>

Re: is indexing single-threaded?

Posted by Andy <an...@yahoo.com>.
--- On Wed, 9/22/10, Andy <an...@yahoo.com> wrote:

> Does Solr index data in a single
> thread or can data be indexed concurrently in multiple
> threads?
> 

Can anyone help?