You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Mark <st...@gmail.com> on 2011/08/26 18:08:56 UTC

DIH importing

We are currently delta-importing using DIH after which all of our 
servers have to download the full index (16G). This obviously puts quite 
a strain on our slaves while they are syncing over the index. Is there 
anyway not to sync over the whole index, but rather just the parts that 
have changed?

We would like to get to the point where are no longer using DIH but 
rather we are constantly sending documents over HTTP to our master in 
realtime. We would then like our slaves to download these changes as 
soon as possible. Is something like this even possible?

Thanks for you help

Re: DIH importing

Posted by Mark <st...@gmail.com>.

Thanks Ill give that a try

On 8/26/11 9:54 AM, simon wrote:
> It sounds as though you are optimizing the index after the delta import. If
> you don't do that, then only new segments will be replicated and syncing
> will be much faster.
>
>
> On Fri, Aug 26, 2011 at 12:08 PM, Mark<st...@gmail.com>  wrote:
>
>> We are currently delta-importing using DIH after which all of our servers
>> have to download the full index (16G). This obviously puts quite a strain on
>> our slaves while they are syncing over the index. Is there anyway not to
>> sync over the whole index, but rather just the parts that have changed?
>>
>> We would like to get to the point where are no longer using DIH but rather
>> we are constantly sending documents over HTTP to our master in realtime. We
>> would then like our slaves to download these changes as soon as possible. Is
>> something like this even possible?
>>
>> Thanks for you help
>>

Re: DIH importing

Posted by simon <mt...@gmail.com>.

It sounds as though you are optimizing the index after the delta import. If
you don't do that, then only new segments will be replicated and syncing
will be much faster.


On Fri, Aug 26, 2011 at 12:08 PM, Mark <st...@gmail.com> wrote:

> We are currently delta-importing using DIH after which all of our servers
> have to download the full index (16G). This obviously puts quite a strain on
> our slaves while they are syncing over the index. Is there anyway not to
> sync over the whole index, but rather just the parts that have changed?
>
> We would like to get to the point where are no longer using DIH but rather
> we are constantly sending documents over HTTP to our master in realtime. We
> would then like our slaves to download these changes as soon as possible. Is
> something like this even possible?
>
> Thanks for you help
>