You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by umajava <um...@gmail.com> on 2012/12/07 03:58:58 UTC
Need help with delta import
Hi,
I am trying to do delta import and I am not able to get it to work. How ever
full import does work. Could you please help me figure out what I am
missing?
data-config.xml file
<document name="persons">
<entity name="person" pk="ID" query="select id from uma_test"
deltaImportQuery="select id from uma_test where
ID='${dataimport.delta.id}'"
deltaQuery="select ID from uma_test where upd_ts >
'${dataimport.last_index_time}'">
<field column="ID" name="id" indexed="true" stored="true" />
<field column="email" name="email" indexed="true" stored="true" />
<field column="fname" name="fname" indexed="true" stored="true" />
</entity>
</document>
dataimport.properties file
metadataObject.last_index_time=2012-09-20 11\:12\:47
person.last_index_time=2012-11-18 13\:54\:29
interval=10
port=8080
server=localhost
params=/select?qt\=/dataimport&command\=delta-import&clean\=false&commit\=true
webapp=solr
syncEnabled=1
last_index_time=2012-11-18 13\:54\:29
syncCores=coreHr,coreEn
log output
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
maybeReloadConfiguration
O: Loading DIH Configuration:
C://Software//apache-solr-4.0.0//apache-solr-4.0.0//Uma//db//db-data-config.xml
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
loadDataConfig
O: Data Configuration loaded successfully
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
O: Starting Delta Import
Dec-2012 02:49:24 org.apache.solr.core.SolrCore execute
O: [collection1] webapp=/solr path=/dataimport
params={commit=false&command=delta-import} status=0 QTime=16
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.SimplePropertiesWriter
readIndexerProperties
O: Read dataimport.properties
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder doDelta
O: Starting delta collection.
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Running ModifiedRowKey() for Entity: person
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
O: Creating a connection for entity person with URL:
jdbc:mysql://localhost/test
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
O: Time taken for getConnection(): 125
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed ModifiedRowKey for Entity: person rows obtained : 8
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed DeletedRowKey for Entity: person rows obtained : 0
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed parentDeltaQuery for Entity: person
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder doDelta
O: Delta Import completed successfully
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder execute
O: Time taken = 0:0:0.156
Dec-2012 02:49:24 org.apache.solr.update.processor.LogUpdateProcessor finish
O: [collection1] webapp=/solr path=/dataimport
params={commit=false&command=delta-import} status=0 QTime=16 {} 0
Thanks,
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
Posted by umajava <um...@gmail.com>.
Can anyone please help me with this?
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4025225.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
Posted by umajava <um...@gmail.com>.
Ouput in the browser is
<str name="status">idle</str><str name="importResponse"/>-<lst
name="statusMessages"><str name="Total Requests made to
DataSource">9</str><str name="Total Rows Fetched">8</str><str name="Total
Documents Skipped">0</str><str name="Delta Dump started">2012-12-07
03:15:36</str><str name="Identifying Delta">2012-12-07 03:15:36</str><str
name="Deltas Obtained">2012-12-07 03:15:36</str><str name="Building
documents">2012-12-07 03:15:36</str><str name="Total Changed
Documents">8</str><str name="Total Documents Processed">0</str><str
name="Time taken">0:0:0.32</str></lst>
I have tried all sorts of URL's to invoke the data import
http://localhost:8080/solr/dataimport?command=delta-import
http://localhost:8080/solr/dataimport?command=delta-import&commit=false
http://localhost:8080/solr/dataimport?command=delta-import&commit=true
http://localhost:8080/solr/dataimport?command=delta-import&commit=true&debug=true
http://localhost:8080/solr/dataimport?command=delta-import&commit=true&debug=true&verbose=true
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4025004.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
Shawn,
I think it only is a problem with "dih.delta.xxx" ... the longer version, "dataimport.delta.xxx" doesn't work. This is coded in DocBuilder#doDelta and this line:
vri.addNamespace(ConfigNameConstants.IMPORTER_NS_SHORT + ".delta", map);
There is no additional line for:
vri.addNamespace(ConfigNameConstants.IMPORTER_NS + ".delta", map);
This is how it is in 3.5, 4.0 and Trunk, which are all the versions I have been testing against.
However, at some point someone did depreciate the longer "dataimporter.", so it would be safer to change to "dih." everywhere you use it. I had thought about removing the longer one for 4.0 and then figured most people wouldn't realize it had been depreciated (there's no warning logged, etc) and I didn't want to give people unnecessary problems. I guess there is no stopping it being removed in 5.0 though.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org]
Sent: Friday, December 14, 2012 1:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Need help with delta import
On 12/14/2012 11:39 AM, Dyer, James wrote:
> Try ${dih.delta.ID} instead of ${dataimporter.delta.id}. Also use ${dih.last_index_time} instead of ${dataimporter.last_index_time} . I noticed when updating the test cases that the wiki incorrectly used the longer name but with all the versions I tested this on only the short name works. The wiki has since been changed.
James,
I use DIH for full Solr reindexes. My dih config makes extensive use of
${dataimporter.request.XXX} variables for my own custom parameters. I
am using branch_4x checked out yesterday on my dev machine, and I did a
full reindex on that version, which worked. Three questions: 1) Should
I be using ${dih.request.XXX} instead? 2) Is the longer syntax going
away? 3) What issues and/or docs would be good reading material?
Thanks,
Shawn
RE: Need help with delta import
Posted by Swati Swoboda <ss...@igloosoftware.com>.
I am also confused, as I've been using dataimporter.* and not dih.* and it is working fine.....
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org]
Sent: Friday, December 14, 2012 2:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Need help with delta import
On 12/14/2012 11:39 AM, Dyer, James wrote:
> Try ${dih.delta.ID} instead of ${dataimporter.delta.id}. Also use ${dih.last_index_time} instead of ${dataimporter.last_index_time} . I noticed when updating the test cases that the wiki incorrectly used the longer name but with all the versions I tested this on only the short name works. The wiki has since been changed.
James,
I use DIH for full Solr reindexes. My dih config makes extensive use of ${dataimporter.request.XXX} variables for my own custom parameters. I am using branch_4x checked out yesterday on my dev machine, and I did a full reindex on that version, which worked. Three questions: 1) Should I be using ${dih.request.XXX} instead? 2) Is the longer syntax going away? 3) What issues and/or docs would be good reading material?
Thanks,
Shawn
Re: Need help with delta import
Posted by Shawn Heisey <so...@elyograg.org>.
On 12/14/2012 11:39 AM, Dyer, James wrote:
> Try ${dih.delta.ID} instead of ${dataimporter.delta.id}. Also use ${dih.last_index_time} instead of ${dataimporter.last_index_time} . I noticed when updating the test cases that the wiki incorrectly used the longer name but with all the versions I tested this on only the short name works. The wiki has since been changed.
James,
I use DIH for full Solr reindexes. My dih config makes extensive use of
${dataimporter.request.XXX} variables for my own custom parameters. I
am using branch_4x checked out yesterday on my dev machine, and I did a
full reindex on that version, which worked. Three questions: 1) Should
I be using ${dih.request.XXX} instead? 2) Is the longer syntax going
away? 3) What issues and/or docs would be good reading material?
Thanks,
Shawn
Re: Need help with delta import
Posted by thammegowda <tg...@gmail.com>.
I was having similar problem with delta import.
I am using solr 4.6 and making use of ${dih.last_index_time},
${dih.delta.xxx} shorter variable names.
I think the issue in previously discussed posts in the thread lies in
deltaQuery and deltaImportQuery.
if deltaQuery="select *rowId* from MyTable"
then deltaImportQuery="select ...... from MyTable where
myId='${dih.delta.*rowId*}'".
Note that *rowId* variable name should be in same case in both select clause
of /deltaQuery/ and where clause of /deltaImportQuery /.
<br/>
This following import configuration tested in solr 4.6 for delta import.
========================================
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/product" user="xxxx" password="yyyyy"
batchSize="-1"/>
<document name="product">
<entity name="product"
query="select prid, catid, srcid, name, uri, brand, year as date from
Product" transformer="DateFormatTransformer"
deltaQuery="select prid from Product where year >
'${dih.last_index_time}'; "
deltaImportQuery="select prid, catid, srcid, name, uri, brand, year as
date from Product where prid='${dih.delta.prid}'"
pk="prid">
<field column="prid" name="id"/>
<field column="catid" name="catid"/>
<field column="srcid" name="srcid"/>
<field column="name" name="name"/>
<field column="uri" name="uri"/>
<field column="brand" name="brand"/>
<field column="date" name="date" dateformat="yyyy-MM-dd"/>
<entity name="now" query="select NOW() as now"
transformer="DateFormatTransformer">
<field column="now" name="imported_at" dateFormat="yy-MM-dd hh:mm:ss"/>
</entity>
</entity>
</document>
sureshadapa wrote
> I am using below configuration file and The problem is I do not see any
> solr documents committed into Solr Core Selector 'db'
>
> When i run full-import,Is give me message.
> Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
> Requests: 1, Fetched: 8, Skipped: 0, Processed: 0
>
> When i run delta-import,It gives me message.
> Requests: 0, Fetched: 0, Skipped: 0, Processed: 0
>
> solrconfig.xml
> ==========
> <luceneMatchVersion>
> 4.4
> </luceneMatchVersion>
> <requestHandler name="/dataimport"
> class="org.apache.solr.handler.dataimport.DataImportHandler">
>
> <lst name="defaults">
>
> <str name="config">
> db1-data-config.xml
> </str>
>
> </lst>
>
> </requestHandler>
> schema.xml
> ========
> <schema name="db" version="1.1">
>
> <field name="solrp_id" type="sint" indexed="true" stored="true"
> required="true" />
>
>
> <field name="solrp_name" type="string" indexed="true" stored="true" />
>
> <field name="solrp_phone" type="text" indexed="true" stored="true" />
>
> <field name="solrp_email" type="text" indexed="true" stored="true"/>
>
> <field name="solrp_smsno" type="text" indexed="true" stored="true"/>
>
>
>
> <uniqueKey>
> solrp_id
> </uniqueKey>
>
> db1-data-config.xml
> =================
> <dataConfig>
> <dataSource autoCommit="true" batchSize="-1" encoding="UTF-8"
> convertType="true" type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/suresh" user="suresh"
> password="suresh123"/>
>
> <document>
>
> <entity name="list" pk="PROVIDERSID" query="select
> providersid,name,phone,email,smsno from providers"
> deltaImportQuery="select providersid,name,phone,email,smsno from
> providers where PROVIDERSID==${dih.delta.PROVIDERSID}"
> deltaquery="select providersid,name,phone,email,smsno from
> providers where modtime=='${dih.last_index_time}'">
>
> <field column="PROVIDERSID" name="solrp_id" />
>
> <field column="NAME" name="solrp_name" />
>
> <field column="PHONE" name="solrp_phone" />
>
> <field column="EMAIL" name="solrp_email" />
>
> <field column="SMSNO" name="solrp_smsno" />
>
> </entity>
>
> </document>
> </dataConfig>
Changing ${dih.delta.PROVIDERSID} to ${dih.delta.providersid} should work.
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4117167.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
Posted by sureshadapa <su...@gmail.com>.
I am using below configuration file and The problem is I do not see any solr
documents committed into Solr Core Selector 'db'
When i run full-import,Is give me message.
Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
Requests: 1, Fetched: 8, Skipped: 0, Processed: 0
When i run delta-import,It gives me message.
Requests: 0, Fetched: 0, Skipped: 0, Processed: 0
solrconfig.xml
==========
<luceneMatchVersion>4.4</luceneMatchVersion>
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">db1-data-config.xml</str>
</lst>
</requestHandler>
schema.xml
========
<schema name="db" version="1.1">
<field name="solrp_id" type="sint" indexed="true" stored="true"
required="true" />
<field name="solrp_name" type="string" indexed="true" stored="true" />
<field name="solrp_phone" type="text" indexed="true" stored="true" />
<field name="solrp_email" type="text" indexed="true" stored="true"/>
<field name="solrp_smsno" type="text" indexed="true" stored="true"/>
<uniqueKey>solrp_id</uniqueKey>
db1-data-config.xml
=================
<dataConfig>
<dataSource autoCommit="true" batchSize="-1" encoding="UTF-8"
convertType="true" type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/suresh" user="suresh"
password="suresh123"/>
<document>
<entity name="list" pk="PROVIDERSID" query="select
providersid,name,phone,email,smsno from providers"
deltaImportQuery="select providersid,name,phone,email,smsno from
providers where PROVIDERSID==${dih.delta.PROVIDERSID}"
deltaquery="select providersid,name,phone,email,smsno from providers
where modtime=='${dih.last_index_time}'">
<field column="PROVIDERSID" name="solrp_id" />
<field column="NAME" name="solrp_name" />
<field column="PHONE" name="solrp_phone" />
<field column="EMAIL" name="solrp_email" />
<field column="SMSNO" name="solrp_smsno" />
</entity>
</document>
</dataConfig>
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4090999.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
Posted by umajava <um...@gmail.com>.
Sorry but I gave up on this issue. I could not resolve it.
On Tue, Sep 10, 2013 at 8:24 PM, suren [via Lucene] <
ml-node+s472066n4089093h65@n3.nabble.com> wrote:
> Any update? I am also having the same issue. pls reply.
>
> This XML file does not appear to have any style information associated
> with it. The document tree is shown below.
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">7</int>
> </lst>
> <lst name="initArgs">
> <lst name="defaults">
> <str name="config">db-data-config.xml</str>
> </lst>
> </lst>
> <str name="command">delta-import</str>
> <str name="status">idle</str>
> <str name="importResponse"/>
> <lst name="statusMessages">
> <str name="Total Requests made to DataSource">2</str>
> <str name="Total Rows Fetched">1</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Delta Dump started">2013-09-10 07:46:34</str>
> <str name="Identifying Delta">2013-09-10 07:46:34</str>
> <str name="Deltas Obtained">2013-09-10 07:46:35</str>
> <str name="Building documents">2013-09-10 07:46:35</str>
> <str name="Total Changed Documents">1</str>
> <str name="Total Documents Processed">0</str>
> <str name="Time taken">0:0:1.30</str>
> </lst>
> <str name="WARNING">
> This response format is experimental. It is likely to change in the
> future.
> </str>
> </response>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089093.html
> To unsubscribe from Need help with delta import, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4025003&code=dW1hamF2YUBnbWFpbC5jb218NDAyNTAwM3wxNDc2MDQyMDE2>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089714.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by suren <gu...@yahoo.com>.
Any update? I am also having the same issue. pls reply.
This XML file does not appear to have any style information associated with
it. The document tree is shown below.
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">7</int>
</lst>
<lst name="initArgs">
<lst name="defaults">
<str name="config">db-data-config.xml</str>
</lst>
</lst>
<str name="command">delta-import</str>
<str name="status">idle</str>
<str name="importResponse"/>
<lst name="statusMessages">
<str name="Total Requests made to DataSource">2</str>
<str name="Total Rows Fetched">1</str>
<str name="Total Documents Skipped">0</str>
<str name="Delta Dump started">2013-09-10 07:46:34</str>
<str name="Identifying Delta">2013-09-10 07:46:34</str>
<str name="Deltas Obtained">2013-09-10 07:46:35</str>
<str name="Building documents">2013-09-10 07:46:35</str>
<str name="Total Changed Documents">1</str>
<str name="Total Documents Processed">0</str>
<str name="Time taken">0:0:1.30</str>
</lst>
<str name="WARNING">
This response format is experimental. It is likely to change in the future.
</str>
</response>
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089093.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by umajava <um...@gmail.com>.
Any one please?
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4028915.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by umajava <um...@gmail.com>.
I have changed to use dih.xx but still no luck. Even with dataimport or
dataimporter the query is able to fetch the delta records but they are not
able to commit to solr. Would there be any other reason why this would fail?
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027297.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
Try ${dih.delta.ID} instead of ${dataimporter.delta.id}. Also use ${dih.last_index_time} instead of ${dataimporter.last_index_time} . I noticed when updating the test cases that the wiki incorrectly used the longer name but with all the versions I tested this on only the short name works. The wiki has since been changed.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: umajava [mailto:umajava@gmail.com]
Sent: Friday, December 14, 2012 11:27 AM
To: solr-user@lucene.apache.org
Subject: RE: Need help with delta import
I tried full import and then delta import but still the issue is same.
<entity name="person" pk="ID" query="select id from uma_test"
deltaImportQuery="select id, email, fname from uma_test where
ID='${dataimporter.delta.id}'"
deltaQuery="select ID from uma_test where upd_ts >
'${dataimporter.last_index_time}'">
<field column="ID" name="id" indexed="true" stored="true" />
<field column="email" name="email" indexed="true" stored="true" />
<field column="fname" name="fname" indexed="true" stored="true" />
</entity>
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027066.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by umajava <um...@gmail.com>.
I tried full import and then delta import but still the issue is same.
<entity name="person" pk="ID" query="select id from uma_test"
deltaImportQuery="select id, email, fname from uma_test where
ID='${dataimporter.delta.id}'"
deltaQuery="select ID from uma_test where upd_ts >
'${dataimporter.last_index_time}'">
<field column="ID" name="id" indexed="true" stored="true" />
<field column="email" name="email" indexed="true" stored="true" />
<field column="fname" name="fname" indexed="true" stored="true" />
</entity>
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027066.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by umajava <um...@gmail.com>.
Thanks, but this didn't help either. Documents are not getting commited.
<str name="Total Requests made to DataSource">3</str><str name="Total Rows
Fetched">2</str><str name="Total Documents Skipped">0</str><str name="Delta
Dump started">2012-12-14 16:57:25</str><str name="Identifying
Delta">2012-12-14 16:57:25</str><str name="Deltas Obtained">2012-12-14
16:57:25</str><str name="Building documents">2012-12-14 16:57:25</str><str
name="Total Changed Documents">2</str><str name="Total Documents
Processed">0</str><str name="Time taken">0:0:0.140</str>
Should I do full import again as I have included email and fname in the
query and start again?
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027058.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by Swati Swoboda <ss...@igloosoftware.com>.
If I am not mistaken, it's suppose to be "dataimporter.delta.ID" and "dataimporter.last_index_time" You are using dataimport.delta.ID and dataimport.last_index_time
http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport
-----Original Message-----
From: umajava [mailto:umajava@gmail.com]
Sent: Thursday, December 13, 2012 9:35 PM
To: solr-user@lucene.apache.org
Subject: RE: Need help with delta import
Thanks a lot for your reply.
I have made the changes but it still does not work. I still get the same results. Any other suggestions please?
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4026910.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by umajava <um...@gmail.com>.
Thanks a lot for your reply.
I have made the changes but it still does not work. I still get the same
results. Any other suggestions please?
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4026910.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Need help with delta import
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
Its surprising that your full import is working for you. Both your "query" and your "deltaImportQuery" have:
SELECT ID FROM...
...So both your full-import ("query" attr) and your delta-import ("deltaImportQuery" attr) are only getting the ID field from your db. Shouldn't you be at least be getting "email" and "fname" to index also? So by changing both these queries to something like:
SELECT ID, EMAIL, FNAME FROM...
...You should see these 3 fields come through after your full-import. Then, after changing data in your rbdms and doing a delta you should see the data update.
Besides this, your log looks right:
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed ModifiedRowKey for Entity: person rows obtained : 8
...so it looks like it was going to update 8 rows. But seeing that your "deltaImportQuery" is only pulling back the ID, it couldn't possibly change the values for fields like "email" and "fname".
Make sense?
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: umajava [mailto:umajava@gmail.com]
Sent: Thursday, December 06, 2012 8:59 PM
To: solr-user@lucene.apache.org
Subject: Need help with delta import
Hi,
I am trying to do delta import and I am not able to get it to work. How ever
full import does work. Could you please help me figure out what I am
missing?
data-config.xml file
<document name="persons">
<entity name="person" pk="ID" query="select id from uma_test"
deltaImportQuery="select id from uma_test where
ID='${dataimport.delta.id}'"
deltaQuery="select ID from uma_test where upd_ts >
'${dataimport.last_index_time}'">
<field column="ID" name="id" indexed="true" stored="true" />
<field column="email" name="email" indexed="true" stored="true" />
<field column="fname" name="fname" indexed="true" stored="true" />
</entity>
</document>
dataimport.properties file
metadataObject.last_index_time=2012-09-20 11\:12\:47
person.last_index_time=2012-11-18 13\:54\:29
interval=10
port=8080
server=localhost
params=/select?qt\=/dataimport&command\=delta-import&clean\=false&commit\=true
webapp=solr
syncEnabled=1
last_index_time=2012-11-18 13\:54\:29
syncCores=coreHr,coreEn
log output
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
maybeReloadConfiguration
O: Loading DIH Configuration:
C://Software//apache-solr-4.0.0//apache-solr-4.0.0//Uma//db//db-data-config.xml
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
loadDataConfig
O: Data Configuration loaded successfully
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
O: Starting Delta Import
Dec-2012 02:49:24 org.apache.solr.core.SolrCore execute
O: [collection1] webapp=/solr path=/dataimport
params={commit=false&command=delta-import} status=0 QTime=16
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.SimplePropertiesWriter
readIndexerProperties
O: Read dataimport.properties
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder doDelta
O: Starting delta collection.
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Running ModifiedRowKey() for Entity: person
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
O: Creating a connection for entity person with URL:
jdbc:mysql://localhost/test
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
O: Time taken for getConnection(): 125
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed ModifiedRowKey for Entity: person rows obtained : 8
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed DeletedRowKey for Entity: person rows obtained : 0
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed parentDeltaQuery for Entity: person
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder doDelta
O: Delta Import completed successfully
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder execute
O: Time taken = 0:0:0.156
Dec-2012 02:49:24 org.apache.solr.update.processor.LogUpdateProcessor finish
O: [collection1] webapp=/solr path=/dataimport
params={commit=false&command=delta-import} status=0 QTime=16 {} 0
Thanks,
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003.html
Sent from the Solr - User mailing list archive at Nabble.com.