You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jamie Jackson <ja...@gmail.com> on 2016/11/21 16:54:03 UTC

Scheduling Data Import Handler (DIH) Delta Imports

Hi Folks,

I have DIH cores that are being indexed by my Lucee application. That
works, but I'd like to make some improvements:


   - Make a standalone scheduler that's not part of a larger application.
   (FYI, I want to Dockerize the import-triggering service.)
   - Prevent import requests from stacking up. Some of my cores' delta
   imports run every 15 seconds, and they do so blindly/ignorantly. If there
   is contention, very occasionally, import jobs will run long and stack up,
   so I want to make the scheduler nicer/more intelligent. Maybe the service
   would check the import status to see if there's a job already running
   before requesting a new one.

I can write such a thing myself, but does anybody have a Linux or
cross-platform solution written already?

Thanks,
Jamie

Re: Scheduling Data Import Handler (DIH) Delta Imports

Posted by Walter Underwood <wu...@wunderwood.org>.
First, try to do it with something like Apache Camel. That moves the whole
database import process outside of Solr where it can be more easily controlled.

http://camel.apache.org/ <http://camel.apache.org/>

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Nov 28, 2016, at 12:03 PM, Jamie Jackson <ja...@gmail.com> wrote:
> 
> One last bump before I get crackin'...
> 
> On Mon, Nov 21, 2016 at 11:54 AM, Jamie Jackson <ja...@gmail.com>
> wrote:
> 
>> Hi Folks,
>> 
>> I have DIH cores that are being indexed by my Lucee application. That
>> works, but I'd like to make some improvements:
>> 
>> 
>>   - Make a standalone scheduler that's not part of a larger application.
>>   (FYI, I want to Dockerize the import-triggering service.)
>>   - Prevent import requests from stacking up. Some of my cores' delta
>>   imports run every 15 seconds, and they do so blindly/ignorantly. If there
>>   is contention, very occasionally, import jobs will run long and stack up,
>>   so I want to make the scheduler nicer/more intelligent. Maybe the service
>>   would check the import status to see if there's a job already running
>>   before requesting a new one.
>> 
>> I can write such a thing myself, but does anybody have a Linux or
>> cross-platform solution written already?
>> 
>> Thanks,
>> Jamie
>>

Re: Scheduling Data Import Handler (DIH) Delta Imports

Posted by Jamie Jackson <ja...@gmail.com>.
One last bump before I get crackin'...

On Mon, Nov 21, 2016 at 11:54 AM, Jamie Jackson <ja...@gmail.com>
wrote:

> Hi Folks,
>
> I have DIH cores that are being indexed by my Lucee application. That
> works, but I'd like to make some improvements:
>
>
>    - Make a standalone scheduler that's not part of a larger application.
>    (FYI, I want to Dockerize the import-triggering service.)
>    - Prevent import requests from stacking up. Some of my cores' delta
>    imports run every 15 seconds, and they do so blindly/ignorantly. If there
>    is contention, very occasionally, import jobs will run long and stack up,
>    so I want to make the scheduler nicer/more intelligent. Maybe the service
>    would check the import status to see if there's a job already running
>    before requesting a new one.
>
> I can write such a thing myself, but does anybody have a Linux or
> cross-platform solution written already?
>
> Thanks,
> Jamie
>