Scaling data extractor with Solr

[Steven White <sw...@gmail.com>]

Hi everyone,

I'm up to speed about Solr on how it can be setup to provide high
availability (if one Solr server goes down, the backup one takes over).  My
question is how do I make my custom crawler to play "nice" with Solr in
this environment.

Let us say I setup Solr with 3 servers so that if one fails the other one
takes over.  Let us say I also setup my crawler with 3 servers so if one
goes down the other takes over.  But how should my crawlers work?  Can each
function unaware of each other and send the same data to Solr or must my
crawlers synchronize with each other so only 1 is active sending data to
Solr and the others are on stand-by mode?

I like to hear from others how they solved this problem so I don't end up
re-inventing it.

Thanks.

Steve

Re: Scaling data extractor with Solr

[Erick Erickson <er...@gmail.com>]

You can have as many clients indexing to Solr (either Cloud or
stand-alone) as you want, limited only by the load you put
on Solr. I.e. if your indexing throughput is so great that it makes
querying too slow then you have to scale back...

I know of setups with 100+ separate clients all indexing to Solr
at the same time.

Best,
Erick

On Mon, Oct 3, 2016 at 3:13 PM, Steven White <sw...@gmail.com> wrote:
> Hi everyone,
>
> I'm up to speed about Solr on how it can be setup to provide high
> availability (if one Solr server goes down, the backup one takes over).  My
> question is how do I make my custom crawler to play "nice" with Solr in
> this environment.
>
> Let us say I setup Solr with 3 servers so that if one fails the other one
> takes over.  Let us say I also setup my crawler with 3 servers so if one
> goes down the other takes over.  But how should my crawlers work?  Can each
> function unaware of each other and send the same data to Solr or must my
> crawlers synchronize with each other so only 1 is active sending data to
> Solr and the others are on stand-by mode?
>
> I like to hear from others how they solved this problem so I don't end up
> re-inventing it.
>
> Thanks.
>
> Steve