DIH: Delta imports don't write last index time to property file

[Jay Hill <ja...@gmail.com>]

We're using the DIH for delta imports, and we are monitoring on the
<handlerName>.properties file with some health check scripts to verify that
deltas are running. However we noticed that, if nothing has changed, no
update is made to the properties file.

I've verified that this is something within the Solr code where it is
deliberately not updating the property file if there were no documents
created or deleted. Here's a comment line from DocBuilder:

      // Do not commit unnecessarily if this is a delta-import and no
documents were created or deleted
In which case a "finish" method doesn't get called, and that is where the
"persist" method is called which writes out the date to the property file.

So it's clearly not a bug per se since this is intended. And I can see a
point to doing it that way. Deltas will still function correctly on
subsequent runs in that anything changed going forward will still be picked
up. However I also see the point where it is misleading to think that this
represents the last run of the delta import, because you can't rely on the
file to know if the delta actually ran.

The question is: Is this the correct approach? It seems to me that the last
index time should always be logged because it clearly marks when the delta
has been run.

I wanted to get some feedback before opening an issue in JIRA. So please
respond with any preferences to this behavior. My vote would be to change
this so the last index time is always recorded.

-Jay

Re: DIH: Delta imports don't write last index time to property file

[Noble Paul നോബിള്‍ नोब्ळ् <no...@corp.aol.com>]

I am fine with both options. I wish to hear from others too.

On Fri, Aug 7, 2009 at 4:50 AM, Jay Hill<ja...@gmail.com> wrote:
> We're using the DIH for delta imports, and we are monitoring on the
> <handlerName>.properties file with some health check scripts to verify that
> deltas are running. However we noticed that, if nothing has changed, no
> update is made to the properties file.
>
> I've verified that this is something within the Solr code where it is
> deliberately not updating the property file if there were no documents
> created or deleted. Here's a comment line from DocBuilder:
>
>      // Do not commit unnecessarily if this is a delta-import and no
> documents were created or deleted
> In which case a "finish" method doesn't get called, and that is where the
> "persist" method is called which writes out the date to the property file.
>
> So it's clearly not a bug per se since this is intended. And I can see a
> point to doing it that way. Deltas will still function correctly on
> subsequent runs in that anything changed going forward will still be picked
> up. However I also see the point where it is misleading to think that this
> represents the last run of the delta import, because you can't rely on the
> file to know if the delta actually ran.
>
> The question is: Is this the correct approach? It seems to me that the last
> index time should always be logged because it clearly marks when the delta
> has been run.
>
> I wanted to get some feedback before opening an issue in JIRA. So please
> respond with any preferences to this behavior. My vote would be to change
> this so the last index time is always recorded.
>
> -Jay
>



-- 
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com