You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marc Sturlese <ma...@gmail.com> on 2008/11/14 12:05:29 UTC
troubles with delta import
Hey there, I am using dataimport with full-import successfully but there's no
way do make it work with delta-import. Aparently solr doesn't show any error
but it does not do what it is supose to.
I thing the problme is with dataimport.properties because it is never
updated. I have it placed in the same folder as solrconfig.xml and
schema.xml and the writing permissions are set propertly. What makes me
doubt is that couldn't find anywhere to tell solr the path of this file.
Don't know if solr is suposed to find it automatically.
My data-config.xml looks like this:
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://path_db" user="user" password="pwd"/>
<document name="product">
<entity name="entity_n" pk="id" query="select id,field1 from table
order by id limit 1000" deltaQuery="select id from table where
dt_last_modified > '${dataimporter.last_index_time}' order by id" >
<field name="id" column="id" />
<field name="field1" column="field1" />
</entity>
</document>
</dataConfig>
*I have in the rows of the table a timestamp field called dt_last_modified
Other thing that can't exactly understant is why i have to put the query and
delta-query... why just with deltaquery (with more fields in the select) is
not enough?
After the ejecution everything seems to go ok (even with the debug and
verbose mode) but no docs have changed and dataimport.properties is not
updated...
Any suggestion? Have done many tests but no way...
--
View this message in context: http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: troubles with delta import
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Mark, even if the dataimport.properties is not present, DIH will create it.
Are you seeing any exceptions in the log or in the debug mode?
On Fri, Nov 14, 2008 at 7:03 PM, Marc Sturlese <ma...@gmail.com>wrote:
>
> Hey Shalin,
> I have tryied 2 methods:
> 1-First doing a full-import and after a delta-import
> 2.-Start directly with the delta-import.
>
> In any of both cases the date of the import.properties file is updated. I
> have it placed in the same folder as schema.xml,data-config.xml and
> solrconfig.xml (is where I think is must be placed acording to what I
> understand in the wiki). Is it correct?
> Is the only thing that I think maybe I am missing...
> Thanks in advance
>
>
> Shalin Shekhar Mangar wrote:
> >
> > Hi Marc,
> >
> > Did you do a full-import first? If not, no value for last import time is
> > written and the delta query may fail. We should fix this to use a sane
> > default so that people do not need to full import first.
> >
> > You need to put both because we support both full and delta, both of
> which
> > need different kinds of queries and we cannot decide what you are going
> to
> > use.
> >
> > On Fri, Nov 14, 2008 at 4:35 PM, Marc Sturlese
> > <ma...@gmail.com>wrote:
> >
> >>
> >> Hey there, I am using dataimport with full-import successfully but
> >> there's
> >> no
> >> way do make it work with delta-import. Aparently solr doesn't show any
> >> error
> >> but it does not do what it is supose to.
> >> I thing the problme is with dataimport.properties because it is never
> >> updated. I have it placed in the same folder as solrconfig.xml and
> >> schema.xml and the writing permissions are set propertly. What makes me
> >> doubt is that couldn't find anywhere to tell solr the path of this file.
> >> Don't know if solr is suposed to find it automatically.
> >>
> >> My data-config.xml looks like this:
> >> <dataConfig>
> >> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> >> url="jdbc:mysql://path_db" user="user" password="pwd"/>
> >> <document name="product">
> >> <entity name="entity_n" pk="id" query="select id,field1 from
> table
> >> order by id limit 1000" deltaQuery="select id from table where
> >> dt_last_modified > '${dataimporter.last_index_time}' order by id" >
> >> <field name="id" column="id" />
> >> <field name="field1" column="field1" />
> >> </entity>
> >> </document>
> >> </dataConfig>
> >> *I have in the rows of the table a timestamp field called
> >> dt_last_modified
> >>
> >> Other thing that can't exactly understant is why i have to put the query
> >> and
> >> delta-query... why just with deltaquery (with more fields in the select)
> >> is
> >> not enough?
> >>
> >> After the ejecution everything seems to go ok (even with the debug and
> >> verbose mode) but no docs have changed and dataimport.properties is not
> >> updated...
> >>
> >> Any suggestion? Have done many tests but no way...
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/troubles-with-delta-import-tp20498449p20500510.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: troubles with delta import
Posted by Marc Sturlese <ma...@gmail.com>.
Hey Shalin,
I have tryied 2 methods:
1-First doing a full-import and after a delta-import
2.-Start directly with the delta-import.
In any of both cases the date of the import.properties file is updated. I
have it placed in the same folder as schema.xml,data-config.xml and
solrconfig.xml (is where I think is must be placed acording to what I
understand in the wiki). Is it correct?
Is the only thing that I think maybe I am missing...
Thanks in advance
Shalin Shekhar Mangar wrote:
>
> Hi Marc,
>
> Did you do a full-import first? If not, no value for last import time is
> written and the delta query may fail. We should fix this to use a sane
> default so that people do not need to full import first.
>
> You need to put both because we support both full and delta, both of which
> need different kinds of queries and we cannot decide what you are going to
> use.
>
> On Fri, Nov 14, 2008 at 4:35 PM, Marc Sturlese
> <ma...@gmail.com>wrote:
>
>>
>> Hey there, I am using dataimport with full-import successfully but
>> there's
>> no
>> way do make it work with delta-import. Aparently solr doesn't show any
>> error
>> but it does not do what it is supose to.
>> I thing the problme is with dataimport.properties because it is never
>> updated. I have it placed in the same folder as solrconfig.xml and
>> schema.xml and the writing permissions are set propertly. What makes me
>> doubt is that couldn't find anywhere to tell solr the path of this file.
>> Don't know if solr is suposed to find it automatically.
>>
>> My data-config.xml looks like this:
>> <dataConfig>
>> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
>> url="jdbc:mysql://path_db" user="user" password="pwd"/>
>> <document name="product">
>> <entity name="entity_n" pk="id" query="select id,field1 from table
>> order by id limit 1000" deltaQuery="select id from table where
>> dt_last_modified > '${dataimporter.last_index_time}' order by id" >
>> <field name="id" column="id" />
>> <field name="field1" column="field1" />
>> </entity>
>> </document>
>> </dataConfig>
>> *I have in the rows of the table a timestamp field called
>> dt_last_modified
>>
>> Other thing that can't exactly understant is why i have to put the query
>> and
>> delta-query... why just with deltaquery (with more fields in the select)
>> is
>> not enough?
>>
>> After the ejecution everything seems to go ok (even with the debug and
>> verbose mode) but no docs have changed and dataimport.properties is not
>> updated...
>>
>> Any suggestion? Have done many tests but no way...
>>
>> --
>> View this message in context:
>> http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/troubles-with-delta-import-tp20498449p20500510.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: troubles with delta import
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Hi Marc,
Did you do a full-import first? If not, no value for last import time is
written and the delta query may fail. We should fix this to use a sane
default so that people do not need to full import first.
You need to put both because we support both full and delta, both of which
need different kinds of queries and we cannot decide what you are going to
use.
On Fri, Nov 14, 2008 at 4:35 PM, Marc Sturlese <ma...@gmail.com>wrote:
>
> Hey there, I am using dataimport with full-import successfully but there's
> no
> way do make it work with delta-import. Aparently solr doesn't show any
> error
> but it does not do what it is supose to.
> I thing the problme is with dataimport.properties because it is never
> updated. I have it placed in the same folder as solrconfig.xml and
> schema.xml and the writing permissions are set propertly. What makes me
> doubt is that couldn't find anywhere to tell solr the path of this file.
> Don't know if solr is suposed to find it automatically.
>
> My data-config.xml looks like this:
> <dataConfig>
> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://path_db" user="user" password="pwd"/>
> <document name="product">
> <entity name="entity_n" pk="id" query="select id,field1 from table
> order by id limit 1000" deltaQuery="select id from table where
> dt_last_modified > '${dataimporter.last_index_time}' order by id" >
> <field name="id" column="id" />
> <field name="field1" column="field1" />
> </entity>
> </document>
> </dataConfig>
> *I have in the rows of the table a timestamp field called dt_last_modified
>
> Other thing that can't exactly understant is why i have to put the query
> and
> delta-query... why just with deltaquery (with more fields in the select) is
> not enough?
>
> After the ejecution everything seems to go ok (even with the debug and
> verbose mode) but no docs have changed and dataimport.properties is not
> updated...
>
> Any suggestion? Have done many tests but no way...
>
> --
> View this message in context:
> http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: troubles with delta import
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
On Fri, Nov 14, 2008 at 4:35 PM, Marc Sturlese <ma...@gmail.com> wrote:
>
> Hey there, I am using dataimport with full-import successfully but there's no
> way do make it work with delta-import. Aparently solr doesn't show any error
> but it does not do what it is supose to.
> I thing the problme is with dataimport.properties because it is never
> updated. I have it placed in the same folder as solrconfig.xml and
> schema.xml and the writing permissions are set propertly. What makes me
> doubt is that couldn't find anywhere to tell solr the path of this file.
> Don't know if solr is suposed to find it automatically.
>
> My data-config.xml looks like this:
> <dataConfig>
> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://path_db" user="user" password="pwd"/>
> <document name="product">
> <entity name="entity_n" pk="id" query="select id,field1 from table
> order by id limit 1000" deltaQuery="select id from table where
> dt_last_modified > '${dataimporter.last_index_time}' order by id" >
> <field name="id" column="id" />
> <field name="field1" column="field1" />
> </entity>
> </document>
> </dataConfig>
> *I have in the rows of the table a timestamp field called dt_last_modified
>
> Other thing that can't exactly understant is why i have to put the query and
> delta-query... why just with deltaquery (with more fields in the select) is
> not enough?
deltaQuery is used to identify the changed rows.
after the changed rows are identified , it uses the query, or
deltaImportQuery to do the actual import
>
> After the ejecution everything seems to go ok (even with the debug and
> verbose mode) but no docs have changed and dataimport.properties is not
> updated...
>
> Any suggestion? Have done many tests but no way...
>
> --
> View this message in context: http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: troubles with delta import
Posted by Marc Sturlese <ma...@gmail.com>.
Hey Shalin!
Now at least I am getting some errors in the log file :D... Hope now I will
be able to find the problem.
Thanks for everything!
Shalin Shekhar Mangar wrote:
>
> Ok I found the problem.
>
> In debug mode, DataImportHandler does not commit documents since it is
> meant
> for debugging only. If you want to do a commit, add commit=true as a
> request
> parameter.
>
> On Fri, Nov 14, 2008 at 7:56 PM, Marc Sturlese
> <ma...@gmail.com>wrote:
>
>>
>> Hey,
>> That's the weird thing... in the log everything seems to work fine:
>>
>> Nov 14, 2008 3:12:46 PM
>> org.apache.solr.handler.dataimport.DataImportHandler
>> processConfiguration
>> INFO: Processing configuration from solrconfig.xml:
>>
>> {config=/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml}
>> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
>> loadDataConfig
>> INFO: Data Configuration loaded successfully
>> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
>> doDeltaImport
>> INFO: Starting Delta Import
>> Nov 14, 2008 3:12:46 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>> call
>> INFO: Creating a connection for entity homes_tbl_ads with URL:
>> jdbc:mysql://localhost/path_db
>> Nov 14, 2008 3:12:46 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>> call
>> INFO: Time taken for getConnection(): 11
>> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DocBuilder
>> execute
>> INFO: Time taken = 0:0:0.47
>> Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr_web path=/dataimport
>> params={verbose=true&command=delta-import&debug=on} status=0 QTime=130
>> Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr_web path=/dataimport params={command=show-config}
>> status=0 QTime=0
>>
>> I am calling the dataimport this way:
>> http://...dataimport?command=full-import&debug=on&verbose=true
>> http://...dataimport?command=delta-import&debug=on&verbose=true
>>
>> In delta-import I am getting this aoutput with the verbose debug:
>>
>> ...
>> <str name="command">delta-import</str>
>> <str name="mode">debug</str>
>> <arr name="documents">
>> ...
>> lst name="statusMessages">
>> <str name="Total Requests made to DataSource">1</str>
>> <str name="Total Rows Fetched">10</str>
>> <str name="Total Documents Skipped">0</str>
>> <str name="Full Dump Started">2008-11-14 15:12:46</str>
>> <str name="Time taken ">0:0:0.47</str>
>> </lst>
>>
>> It also shows the changes in the rows in the output of the verbose debug
>> but
>> nothing change in the index when I check it with Luke.
>> I keep thinking that something is wrong coz the import.properties it is
>> not
>> being created... but can't find why :(
>>
>> solrconfig.xml:
>> <requestHandler name="/dataimport"
>> class="org.apache.solr.handler.dataimport.DataImportHandler"
>> default="false">
>> <lst name="defaults">
>> <!-- <str name="update.processor">dedupe</str> -->
>> <str
>>
>> name="config">/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml</str>
>> </lst>
>> </requestHandler>
>>
>> data-config.xml:
>>
>> <dataConfig>
>> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
>> url="jdbc:mysql://localhost/trovit_es" user="root" password=""/>
>> <document name="item">
>> <entity name="tables" pk="id" query="select id,field1 from table
>> by
>> id
>> limit 1000" deltaQuery="select id from table where dt_last_modified >
>> '${dataimporter.last_index_time}' order by id" >
>> <field name="id" column="id" />
>> <field name="field1" column="field1" />
>> </entity>
>> </document>
>> </dataConfig>
>>
>> Thanks a lot
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/troubles-with-delta-import-tp20498449p20501450.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/troubles-with-delta-import-tp20498449p20502269.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: troubles with delta import
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Ok I found the problem.
In debug mode, DataImportHandler does not commit documents since it is meant
for debugging only. If you want to do a commit, add commit=true as a request
parameter.
On Fri, Nov 14, 2008 at 7:56 PM, Marc Sturlese <ma...@gmail.com>wrote:
>
> Hey,
> That's the weird thing... in the log everything seems to work fine:
>
> Nov 14, 2008 3:12:46 PM
> org.apache.solr.handler.dataimport.DataImportHandler
> processConfiguration
> INFO: Processing configuration from solrconfig.xml:
>
> {config=/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml}
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
> loadDataConfig
> INFO: Data Configuration loaded successfully
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
> INFO: Starting Delta Import
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
> call
> INFO: Creating a connection for entity homes_tbl_ads with URL:
> jdbc:mysql://localhost/path_db
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
> call
> INFO: Time taken for getConnection(): 11
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DocBuilder
> execute
> INFO: Time taken = 0:0:0.47
> Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr_web path=/dataimport
> params={verbose=true&command=delta-import&debug=on} status=0 QTime=130
> Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr_web path=/dataimport params={command=show-config}
> status=0 QTime=0
>
> I am calling the dataimport this way:
> http://...dataimport?command=full-import&debug=on&verbose=true
> http://...dataimport?command=delta-import&debug=on&verbose=true
>
> In delta-import I am getting this aoutput with the verbose debug:
>
> ...
> <str name="command">delta-import</str>
> <str name="mode">debug</str>
> <arr name="documents">
> ...
> lst name="statusMessages">
> <str name="Total Requests made to DataSource">1</str>
> <str name="Total Rows Fetched">10</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Full Dump Started">2008-11-14 15:12:46</str>
> <str name="Time taken ">0:0:0.47</str>
> </lst>
>
> It also shows the changes in the rows in the output of the verbose debug
> but
> nothing change in the index when I check it with Luke.
> I keep thinking that something is wrong coz the import.properties it is not
> being created... but can't find why :(
>
> solrconfig.xml:
> <requestHandler name="/dataimport"
> class="org.apache.solr.handler.dataimport.DataImportHandler"
> default="false">
> <lst name="defaults">
> <!-- <str name="update.processor">dedupe</str> -->
> <str
>
> name="config">/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml</str>
> </lst>
> </requestHandler>
>
> data-config.xml:
>
> <dataConfig>
> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost/trovit_es" user="root" password=""/>
> <document name="item">
> <entity name="tables" pk="id" query="select id,field1 from table by
> id
> limit 1000" deltaQuery="select id from table where dt_last_modified >
> '${dataimporter.last_index_time}' order by id" >
> <field name="id" column="id" />
> <field name="field1" column="field1" />
> </entity>
> </document>
> </dataConfig>
>
> Thanks a lot
>
>
>
> --
> View this message in context:
> http://www.nabble.com/troubles-with-delta-import-tp20498449p20501450.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: troubles with delta import
Posted by Marc Sturlese <ma...@gmail.com>.
Hey,
That's the weird thing... in the log everything seems to work fine:
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImportHandler
processConfiguration
INFO: Processing configuration from solrconfig.xml:
{config=/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml}
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
loadDataConfig
INFO: Data Configuration loaded successfully
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
INFO: Starting Delta Import
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity homes_tbl_ads with URL:
jdbc:mysql://localhost/path_db
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 11
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DocBuilder
execute
INFO: Time taken = 0:0:0.47
Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr_web path=/dataimport
params={verbose=true&command=delta-import&debug=on} status=0 QTime=130
Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr_web path=/dataimport params={command=show-config}
status=0 QTime=0
I am calling the dataimport this way:
http://...dataimport?command=full-import&debug=on&verbose=true
http://...dataimport?command=delta-import&debug=on&verbose=true
In delta-import I am getting this aoutput with the verbose debug:
...
<str name="command">delta-import</str>
<str name="mode">debug</str>
<arr name="documents">
...
lst name="statusMessages">
<str name="Total Requests made to DataSource">1</str>
<str name="Total Rows Fetched">10</str>
<str name="Total Documents Skipped">0</str>
<str name="Full Dump Started">2008-11-14 15:12:46</str>
<str name="Time taken ">0:0:0.47</str>
</lst>
It also shows the changes in the rows in the output of the verbose debug but
nothing change in the index when I check it with Luke.
I keep thinking that something is wrong coz the import.properties it is not
being created... but can't find why :(
solrconfig.xml:
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler"
default="false">
<lst name="defaults">
<!-- <str name="update.processor">dedupe</str> -->
<str
name="config">/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml</str>
</lst>
</requestHandler>
data-config.xml:
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/trovit_es" user="root" password=""/>
<document name="item">
<entity name="tables" pk="id" query="select id,field1 from table by id
limit 1000" deltaQuery="select id from table where dt_last_modified >
'${dataimporter.last_index_time}' order by id" >
<field name="id" column="id" />
<field name="field1" column="field1" />
</entity>
</document>
</dataConfig>
Thanks a lot
--
View this message in context: http://www.nabble.com/troubles-with-delta-import-tp20498449p20501450.html
Sent from the Solr - User mailing list archive at Nabble.com.