You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by suren <gu...@yahoo.com> on 2013/09/10 16:54:08 UTC
RE: Need help with delta import
Any update? I am also having the same issue. pls reply.
This XML file does not appear to have any style information associated with
it. The document tree is shown below.
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">7</int>
</lst>
<lst name="initArgs">
<lst name="defaults">
<str name="config">db-data-config.xml</str>
</lst>
</lst>
<str name="command">delta-import</str>
<str name="status">idle</str>
<str name="importResponse"/>
<lst name="statusMessages">
<str name="Total Requests made to DataSource">2</str>
<str name="Total Rows Fetched">1</str>
<str name="Total Documents Skipped">0</str>
<str name="Delta Dump started">2013-09-10 07:46:34</str>
<str name="Identifying Delta">2013-09-10 07:46:34</str>
<str name="Deltas Obtained">2013-09-10 07:46:35</str>
<str name="Building documents">2013-09-10 07:46:35</str>
<str name="Total Changed Documents">1</str>
<str name="Total Documents Processed">0</str>
<str name="Time taken">0:0:1.30</str>
</lst>
<str name="WARNING">
This response format is experimental. It is likely to change in the future.
</str>
</response>
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089093.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
Posted by thammegowda <tg...@gmail.com>.
I was having similar problem with delta import.
I am using solr 4.6 and making use of ${dih.last_index_time},
${dih.delta.xxx} shorter variable names.
I think the issue in previously discussed posts in the thread lies in
deltaQuery and deltaImportQuery.
if deltaQuery="select *rowId* from MyTable"
then deltaImportQuery="select ...... from MyTable where
myId='${dih.delta.*rowId*}'".
Note that *rowId* variable name should be in same case in both select clause
of /deltaQuery/ and where clause of /deltaImportQuery /.
<br/>
This following import configuration tested in solr 4.6 for delta import.
========================================
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/product" user="xxxx" password="yyyyy"
batchSize="-1"/>
<document name="product">
<entity name="product"
query="select prid, catid, srcid, name, uri, brand, year as date from
Product" transformer="DateFormatTransformer"
deltaQuery="select prid from Product where year >
'${dih.last_index_time}'; "
deltaImportQuery="select prid, catid, srcid, name, uri, brand, year as
date from Product where prid='${dih.delta.prid}'"
pk="prid">
<field column="prid" name="id"/>
<field column="catid" name="catid"/>
<field column="srcid" name="srcid"/>
<field column="name" name="name"/>
<field column="uri" name="uri"/>
<field column="brand" name="brand"/>
<field column="date" name="date" dateformat="yyyy-MM-dd"/>
<entity name="now" query="select NOW() as now"
transformer="DateFormatTransformer">
<field column="now" name="imported_at" dateFormat="yy-MM-dd hh:mm:ss"/>
</entity>
</entity>
</document>
sureshadapa wrote
> I am using below configuration file and The problem is I do not see any
> solr documents committed into Solr Core Selector 'db'
>
> When i run full-import,Is give me message.
> Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
> Requests: 1, Fetched: 8, Skipped: 0, Processed: 0
>
> When i run delta-import,It gives me message.
> Requests: 0, Fetched: 0, Skipped: 0, Processed: 0
>
> solrconfig.xml
> ==========
> <luceneMatchVersion>
> 4.4
> </luceneMatchVersion>
> <requestHandler name="/dataimport"
> class="org.apache.solr.handler.dataimport.DataImportHandler">
>
> <lst name="defaults">
>
> <str name="config">
> db1-data-config.xml
> </str>
>
> </lst>
>
> </requestHandler>
> schema.xml
> ========
> <schema name="db" version="1.1">
>
> <field name="solrp_id" type="sint" indexed="true" stored="true"
> required="true" />
>
>
> <field name="solrp_name" type="string" indexed="true" stored="true" />
>
> <field name="solrp_phone" type="text" indexed="true" stored="true" />
>
> <field name="solrp_email" type="text" indexed="true" stored="true"/>
>
> <field name="solrp_smsno" type="text" indexed="true" stored="true"/>
>
>
>
> <uniqueKey>
> solrp_id
> </uniqueKey>
>
> db1-data-config.xml
> =================
> <dataConfig>
> <dataSource autoCommit="true" batchSize="-1" encoding="UTF-8"
> convertType="true" type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/suresh" user="suresh"
> password="suresh123"/>
>
> <document>
>
> <entity name="list" pk="PROVIDERSID" query="select
> providersid,name,phone,email,smsno from providers"
> deltaImportQuery="select providersid,name,phone,email,smsno from
> providers where PROVIDERSID==${dih.delta.PROVIDERSID}"
> deltaquery="select providersid,name,phone,email,smsno from
> providers where modtime=='${dih.last_index_time}'">
>
> <field column="PROVIDERSID" name="solrp_id" />
>
> <field column="NAME" name="solrp_name" />
>
> <field column="PHONE" name="solrp_phone" />
>
> <field column="EMAIL" name="solrp_email" />
>
> <field column="SMSNO" name="solrp_smsno" />
>
> </entity>
>
> </document>
> </dataConfig>
Changing ${dih.delta.PROVIDERSID} to ${dih.delta.providersid} should work.
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4117167.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
Posted by sureshadapa <su...@gmail.com>.
I am using below configuration file and The problem is I do not see any solr
documents committed into Solr Core Selector 'db'
When i run full-import,Is give me message.
Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
Requests: 1, Fetched: 8, Skipped: 0, Processed: 0
When i run delta-import,It gives me message.
Requests: 0, Fetched: 0, Skipped: 0, Processed: 0
solrconfig.xml
==========
<luceneMatchVersion>4.4</luceneMatchVersion>
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">db1-data-config.xml</str>
</lst>
</requestHandler>
schema.xml
========
<schema name="db" version="1.1">
<field name="solrp_id" type="sint" indexed="true" stored="true"
required="true" />
<field name="solrp_name" type="string" indexed="true" stored="true" />
<field name="solrp_phone" type="text" indexed="true" stored="true" />
<field name="solrp_email" type="text" indexed="true" stored="true"/>
<field name="solrp_smsno" type="text" indexed="true" stored="true"/>
<uniqueKey>solrp_id</uniqueKey>
db1-data-config.xml
=================
<dataConfig>
<dataSource autoCommit="true" batchSize="-1" encoding="UTF-8"
convertType="true" type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/suresh" user="suresh"
password="suresh123"/>
<document>
<entity name="list" pk="PROVIDERSID" query="select
providersid,name,phone,email,smsno from providers"
deltaImportQuery="select providersid,name,phone,email,smsno from
providers where PROVIDERSID==${dih.delta.PROVIDERSID}"
deltaquery="select providersid,name,phone,email,smsno from providers
where modtime=='${dih.last_index_time}'">
<field column="PROVIDERSID" name="solrp_id" />
<field column="NAME" name="solrp_name" />
<field column="PHONE" name="solrp_phone" />
<field column="EMAIL" name="solrp_email" />
<field column="SMSNO" name="solrp_smsno" />
</entity>
</document>
</dataConfig>
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4090999.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help with delta import
Posted by umajava <um...@gmail.com>.
Sorry but I gave up on this issue. I could not resolve it.
On Tue, Sep 10, 2013 at 8:24 PM, suren [via Lucene] <
ml-node+s472066n4089093h65@n3.nabble.com> wrote:
> Any update? I am also having the same issue. pls reply.
>
> This XML file does not appear to have any style information associated
> with it. The document tree is shown below.
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">7</int>
> </lst>
> <lst name="initArgs">
> <lst name="defaults">
> <str name="config">db-data-config.xml</str>
> </lst>
> </lst>
> <str name="command">delta-import</str>
> <str name="status">idle</str>
> <str name="importResponse"/>
> <lst name="statusMessages">
> <str name="Total Requests made to DataSource">2</str>
> <str name="Total Rows Fetched">1</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Delta Dump started">2013-09-10 07:46:34</str>
> <str name="Identifying Delta">2013-09-10 07:46:34</str>
> <str name="Deltas Obtained">2013-09-10 07:46:35</str>
> <str name="Building documents">2013-09-10 07:46:35</str>
> <str name="Total Changed Documents">1</str>
> <str name="Total Documents Processed">0</str>
> <str name="Time taken">0:0:1.30</str>
> </lst>
> <str name="WARNING">
> This response format is experimental. It is likely to change in the
> future.
> </str>
> </response>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089093.html
> To unsubscribe from Need help with delta import, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4025003&code=dW1hamF2YUBnbWFpbC5jb218NDAyNTAwM3wxNDc2MDQyMDE2>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089714.html
Sent from the Solr - User mailing list archive at Nabble.com.