dataimport.properties is not updated on delta-import

[warb <wa...@mail.com>]

Hello!

I am having some difficulties getting dataimport (DIH) to behave correctly
in Solr 1.4.0. Indexing itself works just as it is supposed to with both
full-import and delta-import adding modified or newly created records to the
index. The problem is however that the date and time of the last
delta-import is not updated in the "dataimport.properites" file. The only
time the file gets updated is when performing a full-import. 

Now, this is not a huge problem since delta-import will simply disregard
records already imported (due to the primary key), but it seems wasteful to
fetch records which have already been added on previous runs. Also, as the
database grows the delta-imports will take longer and longer.

Does anyone know of anything I might have overlooked or known bugs?

Thanks in advance!

Johan Andersson
-- 
View this message in context: http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p916753.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: dataimport.properties is not updated on delta-import

[Alexey Serba <as...@gmail.com>]

Please note that Oracle ( or Oracle jdbc driver ) converts column
names to upper case eventhough you state them in lower case. If this
is the case then try to rewrite your query in the following form
select id as "id", name as "name" from table

On Thursday, June 24, 2010, warb <wa...@mail.com> wrote:
>
> Hello again!
>
> Upon further investigation it seems that something is amiss with
> delta-import after all, the delta-import does not actually import anything
> (I thought it did when I ran it previously but I am not sure that was the
> case any longer.) It does complete successfully as seen from the front-end
> (dataimport?command=delta-import). Also in the logs it is stated the the
> import was successful (INFO: Delta Import completed successfully), but there
> are exception pertaining to some documents.
>
> The exception message is that the id field is missing
> (org.apache.solr.common.SolrException: Document [null] missing required
> field: id). Now, I have checked the column names in the table, the
> data-config.xml file and the schema.xml file and they all have the
> column/field names written in lowercase and are even named exactly the same.
>
> Do Solr rollback delta-imports if one or more of the documents failed?
> --
> View this message in context: http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p919609.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: dataimport.properties is not updated on delta-import

[Erick Erickson <er...@gmail.com>]

Is there any chance that the "id" field is, indeed, missing for those
documents?
Does your schema require ID? I've also seen constraints added to a DB
that are not retro-active, so even if there is a constraint requiring ID
it's
still possible that some items in your DB don't have them.

A shot in the dark.
Erick

On Thu, Jun 24, 2010 at 10:21 AM, warb <wa...@mail.com> wrote:

>
> Hello again!
>
> Upon further investigation it seems that something is amiss with
> delta-import after all, the delta-import does not actually import anything
> (I thought it did when I ran it previously but I am not sure that was the
> case any longer.) It does complete successfully as seen from the front-end
> (dataimport?command=delta-import). Also in the logs it is stated the the
> import was successful (INFO: Delta Import completed successfully), but
> there
> are exception pertaining to some documents.
>
> The exception message is that the id field is missing
> (org.apache.solr.common.SolrException: Document [null] missing required
> field: id). Now, I have checked the column names in the table, the
> data-config.xml file and the schema.xml file and they all have the
> column/field names written in lowercase and are even named exactly the
> same.
>
> Do Solr rollback delta-imports if one or more of the documents failed?
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p919609.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: dataimport.properties is not updated on delta-import

[warb <wa...@mail.com>]

I've finally found the problem causing the delta-import to fail and thought I
would post it here for future reference (if someone makes the same mistake I
did).

I had forgot to collect the "id" column in the deltaImportQuery. I should,
of course, have known this from the log entires about documents not being
added because they lacked "id" fields.

My broken query: SELECT column_1, column_2, column_3 FROM table WHERE id =
${dataimporter.delta.id}
My working query: SELECT id, column_1, column_2, column_3 FROM table WHERE
id = ${dataimporter.delta.id}

Thanks all for you help!
-- 
View this message in context: http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p933342.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: dataimport.properties is not updated on delta-import

[warb <wa...@mail.com>]

Hello again!

Upon further investigation it seems that something is amiss with
delta-import after all, the delta-import does not actually import anything
(I thought it did when I ran it previously but I am not sure that was the
case any longer.) It does complete successfully as seen from the front-end
(dataimport?command=delta-import). Also in the logs it is stated the the
import was successful (INFO: Delta Import completed successfully), but there
are exception pertaining to some documents. 

The exception message is that the id field is missing
(org.apache.solr.common.SolrException: Document [null] missing required
field: id). Now, I have checked the column names in the table, the
data-config.xml file and the schema.xml file and they all have the
column/field names written in lowercase and are even named exactly the same.

Do Solr rollback delta-imports if one or more of the documents failed?
-- 
View this message in context: http://lucene.472066.n3.nabble.com/dataimport-properties-is-not-updated-on-delta-import-tp916753p919609.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: dataimport.properties is not updated on delta-import

[Stefan Moises <mo...@shoptimax.de>]

Hi,

what I have experienced is that the primary key seems to be case 
sensitive for the delta queries, at least for some jdcd drivers... see 
http://lucene.472066.n3.nabble.com/Problem-with-DIH-delta-import-on-JDBC-tp763469p765262.html 
... so make sure you specify it with the correct case (e.g. ID instead 
of id) in your db-data-config.xml.

Maybe that's the problem...

Cheers,
Stefan

Am 23.06.2010 15:09, schrieb warb:
> Hello!
>
> I am having some difficulties getting dataimport (DIH) to behave correctly
> in Solr 1.4.0. Indexing itself works just as it is supposed to with both
> full-import and delta-import adding modified or newly created records to the
> index. The problem is however that the date and time of the last
> delta-import is not updated in the "dataimport.properites" file. The only
> time the file gets updated is when performing a full-import.
>
> Now, this is not a huge problem since delta-import will simply disregard
> records already imported (due to the primary key), but it seems wasteful to
> fetch records which have already been added on previous runs. Also, as the
> database grows the delta-imports will take longer and longer.
>
> Does anyone know of anything I might have overlooked or known bugs?
>
> Thanks in advance!
>
> Johan Andersson
>    

-- 
*******************************************
Stefan Moises
Senior Softwareentwickler

shoptimax GmbH
Guntherstraße 45 a
90461 Nürnberg
Amtsgericht Nürnberg HRB 21703
GF Friedrich Schreieck

Tel.: 0911/25566-25
Fax:  0911/25566-29
moises@shoptimax.de
http://www.shoptimax.de
*******************************************