Understanding delta import

[adeelmahmood <ad...@gmail.com>]

hi there
I am having some trouble understanding delta import and how its different
from full import .. from what I can tell the only difference is that it has
the clean parameter set to false by default .. otherwise as far as setting
up your query to use the data_import_last_index_time .. you can do that even
in full import .. 

so i tried setting up my import process and my query is setup something like
this

SELECT  some fields (all mapped correctly to solr fields)
FROM some tables
WHERE dateModified > data_import_last_index_time

if i use this query and run the delta import .. it doesnt finds anything ..
doesnt updates anything .. if I run full-import with clean=false .. it finds
the modified documents and updates them correctly ..  right after that if I
run the delta import again .. it finds the same documents AGAIN (the ones
which full-import found) and imports them .. but after that if I keep
changing data and keep hitting delta import url .. nothing happens until I
do the same full-import again .. any idea whats going on here

so based on this it seems like full-import is doing what I want to do ..
simple grab some recently modified data and update or add in solr index ..
should I just use the full-import then ?? any advice would be very helpful

thanks
-- 
View this message in context: http://old.nabble.com/Understanding-delta-import-tp27690376p27690376.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Understanding delta import

[adeelmahmood <ad...@gmail.com>]

any ideas ???


adeelmahmood wrote:
> 
> hi there
> I am having some trouble understanding delta import and how its different
> from full import .. from what I can tell the only difference is that it
> has the clean parameter set to false by default .. otherwise as far as
> setting up your query to use the data_import_last_index_time .. you can do
> that even in full import .. 
> 
> so i tried setting up my import process and my query is setup something
> like this
> 
> SELECT  some fields (all mapped correctly to solr fields)
> FROM some tables
> WHERE dateModified > data_import_last_index_time
> 
> if i use this query and run the delta import .. it doesnt finds anything
> .. doesnt updates anything .. if I run full-import with clean=false .. it
> finds the modified documents and updates them correctly ..  right after
> that if I run the delta import again .. it finds the same documents AGAIN
> (the ones which full-import found) and imports them .. but after that if I
> keep changing data and keep hitting delta import url .. nothing happens
> until I do the same full-import again .. any idea whats going on here
> 
> so based on this it seems like full-import is doing what I want to do ..
> simple grab some recently modified data and update or add in solr index ..
> should I just use the full-import then ?? any advice would be very helpful
> 
> thanks
> 

-- 
View this message in context: http://old.nabble.com/Understanding-delta-import-tp27690376p27698417.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Understanding delta import

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Mon, Feb 22, 2010 at 10:22 PM, adeelmahmood <ad...@gmail.com>wrote:

>
> hi there
> I am having some trouble understanding delta import and how its different
> from full import .. from what I can tell the only difference is that it has
> the clean parameter set to false by default .. otherwise as far as setting
> up your query to use the data_import_last_index_time .. you can do that
> even
> in full import ..
>

Yes, you can do a delta import through a full-import. This is documented at
http://wiki.apache.org/solr/DataImportHandlerFaq#My_delta-import_goes_out_of_memory_._Any_workaround_.3F


>
> so i tried setting up my import process and my query is setup something
> like
> this
>
> SELECT  some fields (all mapped correctly to solr fields)
> FROM some tables
> WHERE dateModified > data_import_last_index_time
>
>
Which attribute is this? "query", "deltaQuery", "deltaImportQuery" or
something else?


> if i use this query and run the delta import .. it doesnt finds anything ..
> doesnt updates anything .. if I run full-import with clean=false .. it
> finds
> the modified documents and updates them correctly ..  right after that if I
> run the delta import again .. it finds the same documents AGAIN (the ones
> which full-import found) and imports them .. but after that if I keep
> changing data and keep hitting delta import url .. nothing happens until I
> do the same full-import again .. any idea whats going on here
>

I can't say anything unless I see the relevant portions of your
data-config.xml

-- 
Regards,
Shalin Shekhar Mangar.