Updating Solr index - DIH delta vs. task queues

[Andy <an...@yahoo.com>]

Hi,
I have data stored in a database that is being updated constantly. I need to find a way to update Solr index as data in the database is being updated.
There seems to be 2 main schools of thoughts on this:
1) DIH delta - query the database for all records that have a timestamp later than the last_index_time. Import those records for indexing to Solr
2) Task queue - every time a record is updated in the database, throw a task to a queue to index that record to Solr
Just want to know what are the pros and cons of each approach and what is your experience. For someone starting new, what'd be your recommendation?
ThanksAndy


      

Re:Re: Updating Solr index - DIH delta vs. task queues

[kafka0102 <ka...@163.com>]

Does anyone care about this?
I use task queue for now.
I think DIH delta cannot handle changed data very well. For aim db,it needs not only a last_index_time collum. If a row is deleted, DIH delta cannot know it. So it need another boolean collum marking whether the row is deleted.However,thus has greatly changed original table's schema. By looking at sphinx's practice,you can create a new table including  last_index_time and isdeleted collum to store changed rows,thus DIH delta fetching this table. And I think the new table is like a task queue.





At 2010-11-05 01:52:20，"Ezequiel Calderara" <ez...@gmail.com> wrote:

>I'm in the same scenario, so this answer would be helpful too..
>I'm adding...
>
>3) Web Service - Request a webservice for all the new data that has been
>updated (can this be done?
>On Thu, Nov 4, 2010 at 2:38 PM, Andy <an...@yahoo.com> wrote:
>
>> Hi,
>> I have data stored in a database that is being updated constantly. I need
>> to find a way to update Solr index as data in the database is being updated.
>> There seems to be 2 main schools of thoughts on this:
>> 1) DIH delta - query the database for all records that have a timestamp
>> later than the last_index_time. Import those records for indexing to Solr
>> 2) Task queue - every time a record is updated in the database, throw a
>> task to a queue to index that record to Solr
>> Just want to know what are the pros and cons of each approach and what is
>> your experience. For someone starting new, what'd be your recommendation?
>> ThanksAndy
>>
>>
>>
>
>
>
>
>-- 
>______
>Ezequiel.
>
>Http://www.ironicnet.com

Re: Updating Solr index - DIH delta vs. task queues

[Ezequiel Calderara <ez...@gmail.com>]

I'm in the same scenario, so this answer would be helpful too..
I'm adding...

3) Web Service - Request a webservice for all the new data that has been
updated (can this be done?
On Thu, Nov 4, 2010 at 2:38 PM, Andy <an...@yahoo.com> wrote:

> Hi,
> I have data stored in a database that is being updated constantly. I need
> to find a way to update Solr index as data in the database is being updated.
> There seems to be 2 main schools of thoughts on this:
> 1) DIH delta - query the database for all records that have a timestamp
> later than the last_index_time. Import those records for indexing to Solr
> 2) Task queue - every time a record is updated in the database, throw a
> task to a queue to index that record to Solr
> Just want to know what are the pros and cons of each approach and what is
> your experience. For someone starting new, what'd be your recommendation?
> ThanksAndy
>
>
>




-- 
______
Ezequiel.

Http://www.ironicnet.com