Re: Need help with delta import

[thammegowda <tg...@gmail.com>]

I was having similar problem with delta import. 

I am using solr 4.6 and making use of ${dih.last_index_time},
${dih.delta.xxx} shorter variable names.

I think the issue in previously discussed posts in the thread lies in
deltaQuery and deltaImportQuery.

if  deltaQuery="select *rowId* from MyTable" 
then deltaImportQuery="select ...... from MyTable where
myId='${dih.delta.*rowId*}'".

Note that *rowId* variable name should be in same case in both select clause
of /deltaQuery/ and where clause of /deltaImportQuery /.
<br/>


This following import configuration tested in solr 4.6 for delta import.
========================================
<dataConfig>
 <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" 
url="jdbc:mysql://localhost/product"  user="xxxx" password="yyyyy"
batchSize="-1"/>

 <document name="product">
     <entity name="product"
	     query="select prid, catid, srcid, name, uri, brand, year as date from
Product" transformer="DateFormatTransformer"
	     deltaQuery="select prid from Product where year &gt;
'${dih.last_index_time}'; "
	     deltaImportQuery="select prid, catid, srcid, name, uri, brand, year as
date from Product where prid='${dih.delta.prid}'"
	     pk="prid">
	     
        <field column="prid" name="id"/>
        <field column="catid" name="catid"/>
        <field column="srcid" name="srcid"/>
        <field column="name" name="name"/>
        <field column="uri" name="uri"/>
        <field column="brand" name="brand"/>
	<field column="date" name="date" dateformat="yyyy-MM-dd"/>

	<entity name="now" query="select NOW() as now"
transformer="DateFormatTransformer">
	  <field column="now" name="imported_at" dateFormat="yy-MM-dd hh:mm:ss"/>
	</entity>

      </entity>
 </document>




sureshadapa wrote
> I am using below configuration file and The problem is I do not see any
> solr documents committed into Solr Core Selector 'db'
> 
> When i run full-import,Is give me message.
> Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
> Requests: 1, Fetched: 8, Skipped: 0, Processed: 0
> 
> When i run delta-import,It gives me message.
> Requests: 0, Fetched: 0, Skipped: 0, Processed: 0
> 
> solrconfig.xml
> ==========
> <luceneMatchVersion>
> 4.4
> </luceneMatchVersion>
> <requestHandler name="/dataimport"
> class="org.apache.solr.handler.dataimport.DataImportHandler">
>     
> <lst name="defaults">
>     	
> <str name="config">
> db1-data-config.xml
> </str>
>     
> </lst>
>   
> </requestHandler>
> schema.xml
> ========
> <schema name="db" version="1.1">
>    
> <field name="solrp_id" type="sint" indexed="true" stored="true"
> required="true" />
>  
>    
> <field name="solrp_name" type="string" indexed="true" stored="true" />
>    
> <field name="solrp_phone" type="text" indexed="true" stored="true" />
>    
> <field name="solrp_email" type="text" indexed="true" stored="true"/>
>    
> <field name="solrp_smsno" type="text" indexed="true" stored="true"/>
>    
>  

>  
> <uniqueKey>
> solrp_id
> </uniqueKey>
> 
> db1-data-config.xml
> =================
> <dataConfig>
> <dataSource autoCommit="true" batchSize="-1" encoding="UTF-8"
> convertType="true" type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/suresh" user="suresh"
> password="suresh123"/>
>     
> <document>
>         
> <entity name="list" pk="PROVIDERSID" query="select
> providersid,name,phone,email,smsno from providers" 
>         deltaImportQuery="select providersid,name,phone,email,smsno from
> providers where PROVIDERSID==${dih.delta.PROVIDERSID}"
>         deltaquery="select providersid,name,phone,email,smsno from
> providers where modtime=='${dih.last_index_time}'">
>             
> <field column="PROVIDERSID" name="solrp_id" />
>             
> <field column="NAME" name="solrp_name" />
>             
> <field column="PHONE" name="solrp_phone" />
>             
> <field column="EMAIL" name="solrp_email" />
>             
> <field column="SMSNO" name="solrp_smsno" />
>         
> </entity>
>     
> </document>
> </dataConfig>


Changing ${dih.delta.PROVIDERSID} to ${dih.delta.providersid} should work. 






--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4117167.html
Sent from the Solr - User mailing list archive at Nabble.com.