You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@lucene.apache.org by surinekkalapu <su...@yahoo.com> on 2011/09/28 22:47:53 UTC
DIH Delta Import not working
Hi,
We are using sole 3.4.0. Full import was working very fine except it is
taking too long for indexing. As we have time contraint we want to do delta
indexing instead of full import everytime.
Below is the scenario.
1. Did a full import. Indexing successful and able to see the records from
solr admin tool.
2. Update record1 in the database table and did a delta import.
a. Logs shows delta import successfull and index now shows both
record1 old value and record1 new value.
Note: I was expecting only the new value for record1 but for some reason
old values in the index are getting removed.
3. Now updated record2 in the database table and did a delta import again.
a. Logs shows delta import successfull and index now shows
record1 old value (record1 new value is more displayed),
record2 old value and record2 new value.
Note: I was expecting only the new values for record1 and record2.
If anyone encounter similar scenario, please can you help me resolving
this issue? Am i missing something in the configuration?!!
Below is the snippet from database-config.xml showing my queries.
-----------------
<document name="documents">
<entity name="document" pk="ID"
query="SELECT ID, TITLE, rownum + 50000 UKNUMGEN FROM DOCUMENTS WHERE
DOC_TYPE != 'E' AND CURRENT_STATUS = 'In Review'"
deltaImportQuery="SELECT ID, TITLE, rownum + 50000 UKNUMGEN FROM
DOCUMENTS
WHERE DOC_TYPE != 'E' AND CURRENT_STATUS = 'In Review' AND ID =
'${dataimporter.delta.ID}'"
deltaQuery="SELECT ID FROM DOCUMENTS WHERE last_change_date IS NOT
NULL and (last_change_date > to_date('${dataimporter.last_index_time}',
'yyyy-mm-dd hh24:mi:ss'))"
>
<field column="DOC_ID" name="ID"/>
<field column="DOCUMENT_NUMBER" name="DocumentNumber"/>
<field column="TITLE" name="DocumentTitle"/>
<field column="UKNUMGEN" name="UKNUMGEN"/>
</entity>
</document>
------------------------------------
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-Delta-Import-not-working-tp3377360p3377360.html
Sent from the Lucene - General mailing list archive at Nabble.com.
Re: DIH Delta Import not working
Posted by surinekkalapu <su...@yahoo.com>.
Yes, I have unique key configured in the schema.xml. I tried both the ways as
described below. Unfortunately, I could not get the delta indexing worked in
both ways. The corresponding database table name is Documents. I am updating
the document title in this table for given document id (ID). In my first
post you can see that ID is defined as pk (primary key).
Scenario 1.
schemal.xml snippet.
..........
<field name="ID" type="string" indexed="true" stored="true"
multiValued="false"/>
<field name="DocumentNumber" type="string" indexed="true"
stored="true" multiValued="false"/>
<field name="DocumentTitle" type="string" indexed="true"
stored="true" multiValued="false"/>
<field name="UKNUMGEN" type="string" indexed="true" stored="false"
multiValued="false"/>
........
<uniqueKey>UKNUMGEN</uniqueKey>
...........
Scenario 2.
..........
<field name="ID" type="string" indexed="true" stored="true"
multiValued="false"/>
<field name="DocumentNumber" type="string" indexed="true"
stored="true" multiValued="false"/>
<field name="DocumentTitle" type="string" indexed="true"
stored="true" multiValued="false"/>
<field name="UKNUMGEN" type="string" indexed="true" stored="false"
multiValued="false"/>
........
<uniqueKey>ID</uniqueKey>
...........
--
View this message in context: http://lucene.472066.n3.nabble.com/DIH-Delta-Import-not-working-tp3377360p3382861.html
Sent from the Lucene - General mailing list archive at Nabble.com.
Re: DIH Delta Import not working
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Do you have a uniqueKey configured in your schema.xml ? Is the unique key's
value the same in record1 old and new value?
On Thu, Sep 29, 2011 at 2:17 AM, surinekkalapu <su...@yahoo.com>wrote:
> Hi,
> We are using sole 3.4.0. Full import was working very fine except it is
> taking too long for indexing. As we have time contraint we want to do delta
> indexing instead of full import everytime.
> Below is the scenario.
> 1. Did a full import. Indexing successful and able to see the records from
> solr admin tool.
> 2. Update record1 in the database table and did a delta import.
> a. Logs shows delta import successfull and index now shows both
> record1 old value and record1 new value.
> Note: I was expecting only the new value for record1 but for some reason
> old values in the index are getting removed.
> 3. Now updated record2 in the database table and did a delta import again.
> a. Logs shows delta import successfull and index now shows
> record1 old value (record1 new value is more displayed),
> record2 old value and record2 new value.
> Note: I was expecting only the new values for record1 and record2.
>
> If anyone encounter similar scenario, please can you help me resolving
> this issue? Am i missing something in the configuration?!!
>
> Below is the snippet from database-config.xml showing my queries.
> -----------------
> <document name="documents">
> <entity name="document" pk="ID"
> query="SELECT ID, TITLE, rownum + 50000 UKNUMGEN FROM DOCUMENTS WHERE
> DOC_TYPE != 'E' AND CURRENT_STATUS = 'In Review'"
> deltaImportQuery="SELECT ID, TITLE, rownum + 50000 UKNUMGEN FROM
> DOCUMENTS
> WHERE DOC_TYPE != 'E' AND CURRENT_STATUS = 'In Review' AND ID =
> '${dataimporter.delta.ID}'"
> deltaQuery="SELECT ID FROM DOCUMENTS WHERE last_change_date IS NOT
> NULL and (last_change_date > to_date('${dataimporter.last_index_time}',
> 'yyyy-mm-dd hh24:mi:ss'))"
> >
> <field column="DOC_ID" name="ID"/>
> <field column="DOCUMENT_NUMBER" name="DocumentNumber"/>
> <field column="TITLE" name="DocumentTitle"/>
> <field column="UKNUMGEN" name="UKNUMGEN"/>
> </entity>
> </document>
> ------------------------------------
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/DIH-Delta-Import-not-working-tp3377360p3377360.html
> Sent from the Lucene - General mailing list archive at Nabble.com.
>
--
Regards,
Shalin Shekhar Mangar.