delta import to detect deleted values

[madhav bahuguna <ma...@gmail.com>]

Hi,
How do i make delta import detect deleted values.
I do have a timestamp column to detect changes ,but the requirement is such
that  rows will be deleted from the table.Every time i run delta import the
index still shows the deleted value.
How do i make delta import detect deleted values so that it does not show
deleted values.

-- 
Regards
Madhav Bahuguna

Re: delta import to detect deleted values

[Chris Hostetter <ho...@fucit.org>]

: As Ahmet indicated, you must have a way to detect that deletions have
: happened.  Marking rows as deleted with an active/inactive field is one
: way.  Another way (the way that we use) is to have a delete trigger on
: the table that creates an entry in a delete tracking table.

If you have no control over the source datasource, and no way of knowing 
if/when a document was deleted, your really only have 2 other options:

a) use some external system to track the "diff" of document uniqueKeys and 
send deletes to solr based on that.

b) do a full rebuild everytime

for a lot of datasets, #b can acctually be quite fast and a lot less 
headache then worrying about deletes -- if you keep a timestamp on every 
doc, you can even do a "rolling rebuild" where you re-index all the live 
docs, and then just remove any doc with a timestamp older then the start 
of your re-index to prune the deletes.

-Hoss
http://www.lucidworks.com/

Re: delta import to detect deleted values

[Shawn Heisey <so...@elyograg.org>]

On 7/11/2014 1:04 AM, madhav bahuguna wrote:
> How do i make delta import detect deleted values.
> I do have a timestamp column to detect changes ,but the requirement is such
> that  rows will be deleted from the table.Every time i run delta import the
> index still shows the deleted value.
> How do i make delta import detect deleted values so that it does not show
> deleted values.

As Ahmet indicated, you must have a way to detect that deletions have
happened.  Marking rows as deleted with an active/inactive field is one
way.  Another way (the way that we use) is to have a delete trigger on
the table that creates an entry in a delete tracking table.

Either way, your indexing process will need customization to deal with
this additional information.

Thanks,
Shawn

Re: delta import to detect deleted values

[Ahmet Arslan <io...@yahoo.com.INVALID>]

Hi,

You need a soft deletion column. Don't delete the entry instead mark it as deleted/inactive etc.

Ahmet



On Friday, July 11, 2014 10:12 AM, madhav bahuguna <ma...@gmail.com> wrote:
Hi,
How do i make delta import detect deleted values.
I do have a timestamp column to detect changes ,but the requirement is such
that  rows will be deleted from the table.Every time i run delta import the
index still shows the deleted value.
How do i make delta import detect deleted values so that it does not show
deleted values.

-- 
Regards
Madhav Bahuguna