You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Manupriya <ma...@gmail.com> on 2008/12/29 10:05:01 UTC
Delta DataImport is not picking the modified value in DB
Hi,
I am using Delta DataImport feature to partially refresh the indexes.
Note - full-import is working perfectly fine. But I need to do delta-import
as I do not want to rebuild all the indexes.
My database structure is same as mentioned in the example at
http://wiki.apache.org/solr/DataImportHandler.
http://www.nabble.com/file/p21200498/example-schema.png
---------------------------------------------------------------------------
The dataimport.properties has the value as -
#Mon Dec 29 14:08:12 IST 2008
last_index_time=2008-12-29 14\:08\:12
---------------------------------------------------------------------------
I have changed the last_modified for the 'tem' table as current timestamp.
And when I query the DB, I get the following result -
http://www.nabble.com/file/p21200498/data.jpg
-------------------------------------------------------------------------------------------------
But when I call delta data-import, it returns me the following response -
Server Response:
{responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
response format is experimental. It is likely to change in the future.}
---------------------------------------------------------------------------------------------------
And when I query for data for '*:*', I get the following -
Note - Data returned by query doesnt reflect the correct timestamp from
database.
SolrDocument(1)
Field Name & Value: includes=12
Field Name & Value: features=[demo feature, demo feature 1]
Field Name & Value: price=12.0
Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
Field Name & Value: manu=manu-12
Field Name & Value: sku=1
Field Name & Value: id=1
Field Name & Value: popularity=1
Field Name & Value: weight=12.0
SolrDocument(2)
Field Name & Value: includes=43
Field Name & Value: features=demo feature 2
Field Name & Value: price=10.0
Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
Field Name & Value: manu=Demo - 12
Field Name & Value: sku=2
Field Name & Value: id=2
Field Name & Value: popularity=5
Field Name & Value: weight=12.0
SolrDocument(3)
Field Name & Value: includes=1
Field Name & Value: features=demo -3
Field Name & Value: price=1.0
Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
Field Name & Value: manu=manu - 36
Field Name & Value: sku=3
Field Name & Value: id=3
Field Name & Value: popularity=1
Field Name & Value: weight=1.0
SolrDocument(4)
Field Name & Value: includes=2
Field Name & Value: features=demo - 4
Field Name & Value: price=2.0
Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
Field Name & Value: manu=manu - 46
Field Name & Value: sku=4
Field Name & Value: id=4
Field Name & Value: popularity=2
Field Name & Value: weight=2.0
SolrDocument(5)
Field Name & Value: includes=3
Field Name & Value: features=demo - 5
Field Name & Value: price=3.0
Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
Field Name & Value: manu=manu - 56
Field Name & Value: sku=5
Field Name & Value: id=5
Field Name & Value: popularity=3
Field Name & Value: weight=3.0
SolrDocument(6)
Field Name & Value: includes=4
Field Name & Value: features=demo - 66
Field Name & Value: price=4.0
Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
Field Name & Value: manu=manu - 66
Field Name & Value: sku=6
Field Name & Value: id=6
Field Name & Value: popularity=4
Field Name & Value: weight=4.0
------------------------------------------------------------------------------------------
My Java code that calls delta import is as follows -
public static void main(String[] args) throws IOException,
SolrServerException {
CommonsHttpSolrServer server = new
CommonsHttpSolrServer("http://localhost:8983/solr");
BinaryResponseParser parser = new BinaryResponseParser();
server.setParser(parser);
ModifiableSolrParams params = new ModifiableSolrParams();
params.set("qt", "/dataimport");
params.set("command", "delta-import");
params.set("commit", "true");
params.set("wt","json");
try{
QueryResponse response = server.query(params);
server.commit();
System.out.println("Server Response: " + response);
SolrQuery query = new SolrQuery("*:*");
QueryResponse results = server.query(query);
SolrDocumentList list = results.getResults();
int index = 1;
for (SolrDocument solrDocument : list) {
System.out.println("SolrDocument("+index+")");
Iterator<Entry<String, Object>> iterator = solrDocument.iterator();
while(iterator.hasNext()){
System.out.println("Field Name & Value: "+iterator.next());
}
index++;
}
}catch(Exception ex){
System.out.println("Exception Occured:"+ex);
}
}
---------------------------------------------------------------------------
Does SOLR do some kind of caching? I dont understand as why the updated
values are not queried correctly from DB?
There are no errors in the logs. But I can see an error on the server
console -
SEVERE: Delta Import Failed
java.lang.NullPointerException
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
What can be the reason for this error?
Thanks,
Manu
--
View this message in context: http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Delta DataImport is not picking the modified value in DB
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
This may happen if your deltaQuery has a problem
Try using the deltaImportQuery also.
--
On Mon, Dec 29, 2008 at 2:57 PM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> What does your data-config look like? especially the delta query part.
>
> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya <ma...@gmail.com>wrote:
>
>>
>> Hi,
>>
>> I am using Delta DataImport feature to partially refresh the indexes.
>>
>> Note - full-import is working perfectly fine. But I need to do delta-import
>> as I do not want to rebuild all the indexes.
>>
>> My database structure is same as mentioned in the example at
>> http://wiki.apache.org/solr/DataImportHandler.
>>
>> http://www.nabble.com/file/p21200498/example-schema.png
>> ---------------------------------------------------------------------------
>> The dataimport.properties has the value as -
>>
>> #Mon Dec 29 14:08:12 IST 2008
>> last_index_time=2008-12-29 14\:08\:12
>> ---------------------------------------------------------------------------
>> I have changed the last_modified for the 'tem' table as current timestamp.
>> And when I query the DB, I get the following result -
>>
>> http://www.nabble.com/file/p21200498/data.jpg
>>
>> -------------------------------------------------------------------------------------------------
>>
>> But when I call delta data-import, it returns me the following response -
>>
>> Server Response:
>>
>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>> response format is experimental. It is likely to change in the future.}
>>
>>
>> ---------------------------------------------------------------------------------------------------
>> And when I query for data for '*:*', I get the following -
>> Note - Data returned by query doesnt reflect the correct timestamp from
>> database.
>>
>> SolrDocument(1)
>> Field Name & Value: includes=12
>> Field Name & Value: features=[demo feature, demo feature 1]
>> Field Name & Value: price=12.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=manu-12
>> Field Name & Value: sku=1
>> Field Name & Value: id=1
>> Field Name & Value: popularity=1
>> Field Name & Value: weight=12.0
>> SolrDocument(2)
>> Field Name & Value: includes=43
>> Field Name & Value: features=demo feature 2
>> Field Name & Value: price=10.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=Demo - 12
>> Field Name & Value: sku=2
>> Field Name & Value: id=2
>> Field Name & Value: popularity=5
>> Field Name & Value: weight=12.0
>> SolrDocument(3)
>> Field Name & Value: includes=1
>> Field Name & Value: features=demo -3
>> Field Name & Value: price=1.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=manu - 36
>> Field Name & Value: sku=3
>> Field Name & Value: id=3
>> Field Name & Value: popularity=1
>> Field Name & Value: weight=1.0
>> SolrDocument(4)
>> Field Name & Value: includes=2
>> Field Name & Value: features=demo - 4
>> Field Name & Value: price=2.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 46
>> Field Name & Value: sku=4
>> Field Name & Value: id=4
>> Field Name & Value: popularity=2
>> Field Name & Value: weight=2.0
>> SolrDocument(5)
>> Field Name & Value: includes=3
>> Field Name & Value: features=demo - 5
>> Field Name & Value: price=3.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 56
>> Field Name & Value: sku=5
>> Field Name & Value: id=5
>> Field Name & Value: popularity=3
>> Field Name & Value: weight=3.0
>> SolrDocument(6)
>> Field Name & Value: includes=4
>> Field Name & Value: features=demo - 66
>> Field Name & Value: price=4.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 66
>> Field Name & Value: sku=6
>> Field Name & Value: id=6
>> Field Name & Value: popularity=4
>> Field Name & Value: weight=4.0
>>
>>
>> ------------------------------------------------------------------------------------------
>> My Java code that calls delta import is as follows -
>>
>> public static void main(String[] args) throws IOException,
>> SolrServerException {
>> CommonsHttpSolrServer server = new
>> CommonsHttpSolrServer("http://localhost:8983/solr");
>> BinaryResponseParser parser = new BinaryResponseParser();
>> server.setParser(parser);
>>
>> ModifiableSolrParams params = new ModifiableSolrParams();
>> params.set("qt", "/dataimport");
>> params.set("command", "delta-import");
>> params.set("commit", "true");
>> params.set("wt","json");
>>
>> try{
>>
>> QueryResponse response = server.query(params);
>> server.commit();
>> System.out.println("Server Response: " + response);
>>
>> SolrQuery query = new SolrQuery("*:*");
>> QueryResponse results = server.query(query);
>>
>> SolrDocumentList list = results.getResults();
>>
>> int index = 1;
>> for (SolrDocument solrDocument : list) {
>>
>> System.out.println("SolrDocument("+index+")");
>> Iterator<Entry<String, Object>> iterator = solrDocument.iterator();
>>
>> while(iterator.hasNext()){
>> System.out.println("Field Name & Value: "+iterator.next());
>> }
>>
>> index++;
>> }
>>
>> }catch(Exception ex){
>>
>> System.out.println("Exception Occured:"+ex);
>> }
>>
>> }
>> ---------------------------------------------------------------------------
>>
>> Does SOLR do some kind of caching? I dont understand as why the updated
>> values are not queried correctly from DB?
>>
>> There are no errors in the logs. But I can see an error on the server
>> console -
>>
>> SEVERE: Delta Import Failed
>> java.lang.NullPointerException
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>> at
>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>
>>
>> What can be the reason for this error?
>>
>> Thanks,
>> Manu
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
--
--Noble Paul
Re: Delta DataImport is not picking the modified value in DB
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
You do not have to take up complete trunk. Solr can continue to be 1.3
and DIH can be a nightly . DIH is shipped as a separate jar in the
same .war file
This can minimize your risks
On Tue, Dec 30, 2008 at 4:54 PM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> On Tue, Dec 30, 2008 at 3:33 PM, Manupriya <ma...@gmail.com>wrote:
>
>>
>> Thanks Shalin and Noble for all the help.
>>
>> I looked into the source code of 1.3 for the Null Pointer that I was
>> getting. And indeed, there was a problem. In release 1.3,
>> SqlEntityProcessor
>> doesnt handle 'deltaImportQuery' for delta imports. But this problem is
>> fixed in the nightly build.
>
>
> DataImportHandler in the Solr 1.3 release supports delta imports. In that
> version we used to generate the query ourselves after detecting the changed
> primary key. This was not very flexible and worked only for very simple
> queries. Due to this the deltaImportQuery attribute was introduced (after
> release 1.3). You can use deltaImportQuery to exactly specify what query
> should be executed once the primary key of a changed row is identified. The
> example given in the wiki should work with Solr 1.3 release.
>
> So now I have one question, we were planning to use Delta Import feature of
>> Solr in our production environment. And the latest stable release (1.3)
>> doesnt support all the features for Delta Import. So is it advisable to use
>> one of the nightly builds in production environement? I am new to Solr, so
>> I
>> need expert opinion in decision making.
>
>
> If the problem was related only to case sensitive names and your import
> works without deltaImportQuery then you should use 1.3 otherwise you don't
> have a lot of choice :)
>
> All the normal caveats of trunk code apply to 1.4, however I believe it is
> fairly stable.
>
>
>>
>> --------------------------------------------------------------------------------------------------
>> Apart from this, I realized, Solr is 'case sensitive', while resolving
>> variable values.
>>
>> For example, the following entity definition gives SQl syntax exception
>> because the varaible is 'id' in deltaQuery and 'ID' in deltaImportQuery.
>>
>> <entity name="item" pk="ID" query="select * from item"
>> deltaQuery="select id from item where last_modified >
>> '${dataimporter.last_index_time}'"
>> deltaImportQuery="select * from item where
>> ID=${dataimporter.delta.ID}">
>>
>> ------------------------------------------------------------------------------------------------
>> And the following definition works fine.
>>
>> <entity name="item" pk="ID" query="select * from item"
>> deltaQuery="select ID from item where last_modified >
>> '${dataimporter.last_index_time}'"
>> deltaImportQuery="select * from item where
>> ID=${dataimporter.delta.ID}">
>>
>> It took me sometime to figure this out and also some looking into the code.
>> I wrongly assumed it to behave like normal SQL query.
>>
>> Is this case-sensitivity intentional? Is it documented somewhere that I can
>> refer for constructing queries for Delta/Full Import?
>
>
> There was a bug related to case-sensitivity in 1.3 which has been fixed in
> trunk. Now, unless you specify the 'name' attribute for a field DIH does
> case-insensitive matches. If you are providing a 'name' it must be in the
> same case as the column name returned by SQL.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
--
--Noble Paul
Re: Delta DataImport is not picking the modified value in DB
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, Dec 30, 2008 at 3:33 PM, Manupriya <ma...@gmail.com>wrote:
>
> Thanks Shalin and Noble for all the help.
>
> I looked into the source code of 1.3 for the Null Pointer that I was
> getting. And indeed, there was a problem. In release 1.3,
> SqlEntityProcessor
> doesnt handle 'deltaImportQuery' for delta imports. But this problem is
> fixed in the nightly build.
DataImportHandler in the Solr 1.3 release supports delta imports. In that
version we used to generate the query ourselves after detecting the changed
primary key. This was not very flexible and worked only for very simple
queries. Due to this the deltaImportQuery attribute was introduced (after
release 1.3). You can use deltaImportQuery to exactly specify what query
should be executed once the primary key of a changed row is identified. The
example given in the wiki should work with Solr 1.3 release.
So now I have one question, we were planning to use Delta Import feature of
> Solr in our production environment. And the latest stable release (1.3)
> doesnt support all the features for Delta Import. So is it advisable to use
> one of the nightly builds in production environement? I am new to Solr, so
> I
> need expert opinion in decision making.
If the problem was related only to case sensitive names and your import
works without deltaImportQuery then you should use 1.3 otherwise you don't
have a lot of choice :)
All the normal caveats of trunk code apply to 1.4, however I believe it is
fairly stable.
>
> --------------------------------------------------------------------------------------------------
> Apart from this, I realized, Solr is 'case sensitive', while resolving
> variable values.
>
> For example, the following entity definition gives SQl syntax exception
> because the varaible is 'id' in deltaQuery and 'ID' in deltaImportQuery.
>
> <entity name="item" pk="ID" query="select * from item"
> deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'"
> deltaImportQuery="select * from item where
> ID=${dataimporter.delta.ID}">
>
> ------------------------------------------------------------------------------------------------
> And the following definition works fine.
>
> <entity name="item" pk="ID" query="select * from item"
> deltaQuery="select ID from item where last_modified >
> '${dataimporter.last_index_time}'"
> deltaImportQuery="select * from item where
> ID=${dataimporter.delta.ID}">
>
> It took me sometime to figure this out and also some looking into the code.
> I wrongly assumed it to behave like normal SQL query.
>
> Is this case-sensitivity intentional? Is it documented somewhere that I can
> refer for constructing queries for Delta/Full Import?
There was a bug related to case-sensitivity in 1.3 which has been fixed in
trunk. Now, unless you specify the 'name' attribute for a field DIH does
case-insensitive matches. If you are providing a 'name' it must be in the
same case as the column name returned by SQL.
--
Regards,
Shalin Shekhar Mangar.
Re: Delta DataImport is not picking the modified value in DB
Posted by Manupriya <ma...@gmail.com>.
Thanks Shalin and Noble for all the help.
I looked into the source code of 1.3 for the Null Pointer that I was
getting. And indeed, there was a problem. In release 1.3, SqlEntityProcessor
doesnt handle 'deltaImportQuery' for delta imports. But this problem is
fixed in the nightly build.
So now I have one question, we were planning to use Delta Import feature of
Solr in our production environment. And the latest stable release (1.3)
doesnt support all the features for Delta Import. So is it advisable to use
one of the nightly builds in production environement? I am new to Solr, so I
need expert opinion in decision making.
--------------------------------------------------------------------------------------------------
Apart from this, I realized, Solr is 'case sensitive', while resolving
variable values.
For example, the following entity definition gives SQl syntax exception
because the varaible is 'id' in deltaQuery and 'ID' in deltaImportQuery.
<entity name="item" pk="ID" query="select * from item"
deltaQuery="select id from item where last_modified >
'${dataimporter.last_index_time}'"
deltaImportQuery="select * from item where
ID=${dataimporter.delta.ID}">
------------------------------------------------------------------------------------------------
And the following definition works fine.
<entity name="item" pk="ID" query="select * from item"
deltaQuery="select ID from item where last_modified >
'${dataimporter.last_index_time}'"
deltaImportQuery="select * from item where
ID=${dataimporter.delta.ID}">
It took me sometime to figure this out and also some looking into the code.
I wrongly assumed it to behave like normal SQL query.
Is this case-sensitivity intentional? Is it documented somewhere that I can
refer for constructing queries for Delta/Full Import?
Thanks,
Manu
Noble Paul നോബിള് नोब्ळ् wrote:
>
> On Mon, Dec 29, 2008 at 5:11 PM, Manupriya <ma...@gmail.com>
> wrote:
>>
>> Hi Shalin,
>>
>> I am using apache-solr-1.3.0 in my project. I thought it to be the latest
>> release. Does it not support delta import?
> It should, 1.3 of solr was the first realease of DIH. A lot of
> features got added to DIH since then. The nightly build of DIH is
> compatible with Solr 1.3. BTW 1.3 supports delta import.
>
>>
>> Thanks,
>> Manu
>>
>>
>> Shalin Shekhar Mangar wrote:
>>>
>>> Which version of Solr are you using? The deltaImportQuery feature was
>>> added
>>> recently, you'd need a nightly build for it to work.
>>>
>>> On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
>>> <ma...@gmail.com>wrote:
>>>
>>>>
>>>> Hi Noble,
>>>>
>>>> I tried with the deltaImportQuery as well. But still I am getting the
>>>> same
>>>> exception on the server console -
>>>>
>>>> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
>>>> doDeltaImport
>>>>
>>>> SEVERE: Delta Import Failed
>>>> java.lang.NullPointerException
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>>> at
>>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>>> at
>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>>
>>>> Another thing, I do have proper data in the DB.
>>>>
>>>> Thanks,
>>>> Manu
>>>>
>>>>
>>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>> >
>>>> > The same example with deltaImportQuery would look as follows
>>>> > <dataConfig>
>>>> > <dataSource driver="com.mysql.jdbc.Driver"
>>>> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>>>> />
>>>> > <document name="products">
>>>> > <entity name="item" pk="ID" query="select * from item"
>>>> > deltaQuery="select id from item where last_modified >
>>>> > '${dataimporter.last_index_time}'"
>>>> > deltaImportQuery="select * from item where
>>>> ID=${dataimporter.delta.ID}"
>>>> >>
>>>> > <field column="NAME" name="name" />
>>>> > <field column="NAME" name="nameSort" />
>>>> > <field column="NAME" name="alphaNameSort" />
>>>> >
>>>> > <entity name="feature" pk="ITEM_ID"
>>>> > query="select DESCRIPTION from FEATURE where
>>>> > ITEM_ID='${item.ID}'"
>>>> > deltaQuery="select ITEM_ID from FEATURE where
>>>> > last_modified > '${dataimporter.last_index_time}'"
>>>> > parentDeltaQuery="select ID from item where
>>>> > ID=${feature.ITEM_ID}">
>>>> > <field name="features" column="DESCRIPTION" />
>>>> > </entity>
>>>> >
>>>> > <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>>>> > query="select CATEGORY_ID from item_category where
>>>> > ITEM_ID='${item.ID}'"
>>>> > deltaQuery="select ITEM_ID, CATEGORY_ID from
>>>> > item_category where last_modified >
>>>> '${dataimporter.last_index_time}'"
>>>> > parentDeltaQuery="select ID from item where
>>>> > ID=${item_category.ITEM_ID}">
>>>> > <entity name="category" pk="ID"
>>>> > query="select DESCRIPTION from category where
>>>> ID
>>>> =
>>>> > '${item_category.CATEGORY_ID}'"
>>>> > deltaQuery="select ID from category where
>>>> > last_modified > '${dataimporter.last_index_time}'"
>>>> > parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>>>> from
>>>> > item_category where CATEGORY_ID=${category.ID}">
>>>> > <field column="description" name="cat" />
>>>> > </entity>
>>>> > </entity>
>>>> > </entity>
>>>> > </document>
>>>> > </dataConfig>
>>>> >
>>>> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
>>>> > <no...@gmail.com> wrote:
>>>> >> did you put in the data?
>>>> >>
>>>> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya
>>>> <ma...@gmail.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> Hi Shalin,
>>>> >>>
>>>> >>> I am actually trying out the example provided at
>>>> >>> http://wiki.apache.org/solr/DataImportHandler.
>>>> >>>
>>>> >>> My db-data-config.xml is as following -
>>>> >>>
>>>> >>> <dataConfig>
>>>> >>> <dataSource driver="com.mysql.jdbc.Driver"
>>>> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root"
>>>> password="root"
>>>> />
>>>> >>> <document name="products">
>>>> >>> <entity name="item" pk="ID" query="select * from item"
>>>> >>> deltaQuery="select id from item where last_modified
>>>> >
>>>> >>> '${dataimporter.last_index_time}'">
>>>> >>> <field column="NAME" name="name" />
>>>> >>> <field column="NAME" name="nameSort" />
>>>> >>> <field column="NAME" name="alphaNameSort" />
>>>> >>>
>>>> >>> <entity name="feature" pk="ITEM_ID"
>>>> >>> query="select DESCRIPTION from FEATURE where
>>>> >>> ITEM_ID='${item.ID}'"
>>>> >>> deltaQuery="select ITEM_ID from FEATURE where
>>>> >>> last_modified > '${dataimporter.last_index_time}'"
>>>> >>> parentDeltaQuery="select ID from item where
>>>> >>> ID=${feature.ITEM_ID}">
>>>> >>> <field name="features" column="DESCRIPTION" />
>>>> >>> </entity>
>>>> >>>
>>>> >>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>>>> >>> query="select CATEGORY_ID from item_category
>>>> where
>>>> >>> ITEM_ID='${item.ID}'"
>>>> >>> deltaQuery="select ITEM_ID, CATEGORY_ID from
>>>> >>> item_category where last_modified >
>>>> '${dataimporter.last_index_time}'"
>>>> >>> parentDeltaQuery="select ID from item where
>>>> >>> ID=${item_category.ITEM_ID}">
>>>> >>> <entity name="category" pk="ID"
>>>> >>> query="select DESCRIPTION from category
>>>> where
>>>> ID
>>>> >>> =
>>>> >>> '${item_category.CATEGORY_ID}'"
>>>> >>> deltaQuery="select ID from category where
>>>> >>> last_modified > '${dataimporter.last_index_time}'"
>>>> >>> parentDeltaQuery="select ITEM_ID,
>>>> CATEGORY_ID
>>>> >>> from
>>>> >>> item_category where CATEGORY_ID=${category.ID}">
>>>> >>> <field column="description" name="cat" />
>>>> >>> </entity>
>>>> >>> </entity>
>>>> >>> </entity>
>>>> >>> </document>
>>>> >>> </dataConfig>
>>>> >>>
>>>> ---------------------------------------------------------------------------------------------------
>>>> >>>
>>>> >>> My DB structure is as following -
>>>> >>>
>>>> >>> DROP TABLE IF EXISTS solr_demo_db.item;
>>>> >>> CREATE TABLE `item` (
>>>> >>> `ID` int(2) NOT NULL auto_increment,
>>>> >>> `name` varchar(100) default NULL,
>>>> >>> `manu` varchar(20) default NULL,
>>>> >>> `weight` varchar(20) default NULL,
>>>> >>> `price` varchar(20) default NULL,
>>>> >>> `popularity` varchar(2) default NULL,
>>>> >>> `includes` varchar(10) default NULL,
>>>> >>> `last_modified` datetime default NULL,
>>>> >>> PRIMARY KEY (`ID`)
>>>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>>>> >>>
>>>> >>> DROP TABLE IF EXISTS solr_demo_db.feature;
>>>> >>> CREATE TABLE `feature` (
>>>> >>> `DESCRIPTION` varchar(100) default NULL,
>>>> >>> `ITEM_ID` int(2) NOT NULL,
>>>> >>> `last_modified` datetime default NULL,
>>>> >>> PRIMARY KEY (`ITEM_ID`)
>>>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>>> >>>
>>>> >>> DROP TABLE IF EXISTS solr_demo_db.category;
>>>> >>> CREATE TABLE `category` (
>>>> >>> `ID` int(2) NOT NULL auto_increment,
>>>> >>> `DESCRIPTION` varchar(100) default NULL,
>>>> >>> `last_modified` date default NULL,
>>>> >>> PRIMARY KEY (`ID`)
>>>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>>>> >>>
>>>> >>> DROP TABLE IF EXISTS solr_demo_db.item_category;
>>>> >>> CREATE TABLE `item_category` (
>>>> >>> `ITEM_ID` int(2) NOT NULL default '0',
>>>> >>> `CATEGORY_ID` int(2) NOT NULL default '0',
>>>> >>> `last_modified` date default NULL,
>>>> >>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
>>>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>>> >>>
>>>> >>> Am I doing something wrong here?
>>>> >>>
>>>> >>> Thanks,
>>>> >>> Manu
>>>> >>>
>>>> >>>
>>>> >>> Shalin Shekhar Mangar wrote:
>>>> >>>>
>>>> >>>> What does your data-config look like? especially the delta query
>>>> part.
>>>> >>>>
>>>> >>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>>>> >>>> <ma...@gmail.com>wrote:
>>>> >>>>
>>>> >>>>>
>>>> >>>>> Hi,
>>>> >>>>>
>>>> >>>>> I am using Delta DataImport feature to partially refresh the
>>>> indexes.
>>>> >>>>>
>>>> >>>>> Note - full-import is working perfectly fine. But I need to do
>>>> >>>>> delta-import
>>>> >>>>> as I do not want to rebuild all the indexes.
>>>> >>>>>
>>>> >>>>> My database structure is same as mentioned in the example at
>>>> >>>>> http://wiki.apache.org/solr/DataImportHandler.
>>>> >>>>>
>>>> >>>>> http://www.nabble.com/file/p21200498/example-schema.png
>>>> >>>>>
>>>> ---------------------------------------------------------------------------
>>>> >>>>> The dataimport.properties has the value as -
>>>> >>>>>
>>>> >>>>> #Mon Dec 29 14:08:12 IST 2008
>>>> >>>>> last_index_time=2008-12-29 14\:08\:12
>>>> >>>>>
>>>> ---------------------------------------------------------------------------
>>>> >>>>> I have changed the last_modified for the 'tem' table as current
>>>> >>>>> timestamp.
>>>> >>>>> And when I query the DB, I get the following result -
>>>> >>>>>
>>>> >>>>> http://www.nabble.com/file/p21200498/data.jpg
>>>> >>>>>
>>>> >>>>>
>>>> -------------------------------------------------------------------------------------------------
>>>> >>>>>
>>>> >>>>> But when I call delta data-import, it returns me the following
>>>> >>>>> response -
>>>> >>>>>
>>>> >>>>> Server Response:
>>>> >>>>>
>>>> >>>>>
>>>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>>>> >>>>> response format is experimental. It is likely to change in the
>>>> >>>>> future.}
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> ---------------------------------------------------------------------------------------------------
>>>> >>>>> And when I query for data for '*:*', I get the following -
>>>> >>>>> Note - Data returned by query doesnt reflect the correct
>>>> timestamp
>>>> >>>>> from
>>>> >>>>> database.
>>>> >>>>>
>>>> >>>>> SolrDocument(1)
>>>> >>>>> Field Name & Value: includes=12
>>>> >>>>> Field Name & Value: features=[demo feature, demo feature 1]
>>>> >>>>> Field Name & Value: price=12.0
>>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>> >>>>> Field Name & Value: manu=manu-12
>>>> >>>>> Field Name & Value: sku=1
>>>> >>>>> Field Name & Value: id=1
>>>> >>>>> Field Name & Value: popularity=1
>>>> >>>>> Field Name & Value: weight=12.0
>>>> >>>>> SolrDocument(2)
>>>> >>>>> Field Name & Value: includes=43
>>>> >>>>> Field Name & Value: features=demo feature 2
>>>> >>>>> Field Name & Value: price=10.0
>>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>> >>>>> Field Name & Value: manu=Demo - 12
>>>> >>>>> Field Name & Value: sku=2
>>>> >>>>> Field Name & Value: id=2
>>>> >>>>> Field Name & Value: popularity=5
>>>> >>>>> Field Name & Value: weight=12.0
>>>> >>>>> SolrDocument(3)
>>>> >>>>> Field Name & Value: includes=1
>>>> >>>>> Field Name & Value: features=demo -3
>>>> >>>>> Field Name & Value: price=1.0
>>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>> >>>>> Field Name & Value: manu=manu - 36
>>>> >>>>> Field Name & Value: sku=3
>>>> >>>>> Field Name & Value: id=3
>>>> >>>>> Field Name & Value: popularity=1
>>>> >>>>> Field Name & Value: weight=1.0
>>>> >>>>> SolrDocument(4)
>>>> >>>>> Field Name & Value: includes=2
>>>> >>>>> Field Name & Value: features=demo - 4
>>>> >>>>> Field Name & Value: price=2.0
>>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>> >>>>> Field Name & Value: manu=manu - 46
>>>> >>>>> Field Name & Value: sku=4
>>>> >>>>> Field Name & Value: id=4
>>>> >>>>> Field Name & Value: popularity=2
>>>> >>>>> Field Name & Value: weight=2.0
>>>> >>>>> SolrDocument(5)
>>>> >>>>> Field Name & Value: includes=3
>>>> >>>>> Field Name & Value: features=demo - 5
>>>> >>>>> Field Name & Value: price=3.0
>>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>> >>>>> Field Name & Value: manu=manu - 56
>>>> >>>>> Field Name & Value: sku=5
>>>> >>>>> Field Name & Value: id=5
>>>> >>>>> Field Name & Value: popularity=3
>>>> >>>>> Field Name & Value: weight=3.0
>>>> >>>>> SolrDocument(6)
>>>> >>>>> Field Name & Value: includes=4
>>>> >>>>> Field Name & Value: features=demo - 66
>>>> >>>>> Field Name & Value: price=4.0
>>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>> >>>>> Field Name & Value: manu=manu - 66
>>>> >>>>> Field Name & Value: sku=6
>>>> >>>>> Field Name & Value: id=6
>>>> >>>>> Field Name & Value: popularity=4
>>>> >>>>> Field Name & Value: weight=4.0
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> ------------------------------------------------------------------------------------------
>>>> >>>>> My Java code that calls delta import is as follows -
>>>> >>>>>
>>>> >>>>> public static void main(String[] args) throws IOException,
>>>> >>>>> SolrServerException {
>>>> >>>>> CommonsHttpSolrServer server = new
>>>> >>>>> CommonsHttpSolrServer("http://localhost:8983/solr");
>>>> >>>>> BinaryResponseParser parser = new BinaryResponseParser();
>>>> >>>>> server.setParser(parser);
>>>> >>>>>
>>>> >>>>> ModifiableSolrParams params = new ModifiableSolrParams();
>>>> >>>>> params.set("qt", "/dataimport");
>>>> >>>>> params.set("command", "delta-import");
>>>> >>>>> params.set("commit", "true");
>>>> >>>>> params.set("wt","json");
>>>> >>>>>
>>>> >>>>> try{
>>>> >>>>>
>>>> >>>>> QueryResponse response = server.query(params);
>>>> >>>>> server.commit();
>>>> >>>>> System.out.println("Server Response: " + response);
>>>> >>>>>
>>>> >>>>> SolrQuery query = new SolrQuery("*:*");
>>>> >>>>> QueryResponse results = server.query(query);
>>>> >>>>>
>>>> >>>>> SolrDocumentList list = results.getResults();
>>>> >>>>>
>>>> >>>>> int index = 1;
>>>> >>>>> for (SolrDocument solrDocument : list) {
>>>> >>>>>
>>>> >>>>> System.out.println("SolrDocument("+index+")");
>>>> >>>>> Iterator<Entry<String, Object>> iterator =
>>>> >>>>> solrDocument.iterator();
>>>> >>>>>
>>>> >>>>> while(iterator.hasNext()){
>>>> >>>>> System.out.println("Field Name & Value:
>>>> >>>>> "+iterator.next());
>>>> >>>>> }
>>>> >>>>>
>>>> >>>>> index++;
>>>> >>>>> }
>>>> >>>>>
>>>> >>>>> }catch(Exception ex){
>>>> >>>>>
>>>> >>>>> System.out.println("Exception Occured:"+ex);
>>>> >>>>> }
>>>> >>>>>
>>>> >>>>> }
>>>> >>>>>
>>>> ---------------------------------------------------------------------------
>>>> >>>>>
>>>> >>>>> Does SOLR do some kind of caching? I dont understand as why the
>>>> >>>>> updated
>>>> >>>>> values are not queried correctly from DB?
>>>> >>>>>
>>>> >>>>> There are no errors in the logs. But I can see an error on the
>>>> server
>>>> >>>>> console -
>>>> >>>>>
>>>> >>>>> SEVERE: Delta Import Failed
>>>> >>>>> java.lang.NullPointerException
>>>> >>>>> at
>>>> >>>>>
>>>> >>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>>> >>>>> at
>>>> >>>>>
>>>> >>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>>> >>>>> at
>>>> >>>>>
>>>> >>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>>> >>>>> at
>>>> >>>>>
>>>> >>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>>> >>>>> at
>>>> >>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>>> >>>>> at
>>>> >>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>>> >>>>> at
>>>> >>>>>
>>>> >>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>>> >>>>> at
>>>> >>>>>
>>>> >>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>>> >>>>> at
>>>> >>>>>
>>>> >>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> What can be the reason for this error?
>>>> >>>>>
>>>> >>>>> Thanks,
>>>> >>>>> Manu
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> View this message in context:
>>>> >>>>>
>>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>>>> >>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>> Regards,
>>>> >>>> Shalin Shekhar Mangar.
>>>> >>>>
>>>> >>>>
>>>> >>>
>>>> >>> --
>>>> >>> View this message in context:
>>>> >>>
>>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
>>>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>> >>>
>>>> >>>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> --Noble Paul
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > --Noble Paul
>>>> >
>>>> >
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21201980.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21202107.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21215131.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Delta DataImport is not picking the modified value in DB
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
On Mon, Dec 29, 2008 at 5:11 PM, Manupriya <ma...@gmail.com> wrote:
>
> Hi Shalin,
>
> I am using apache-solr-1.3.0 in my project. I thought it to be the latest
> release. Does it not support delta import?
It should, 1.3 of solr was the first realease of DIH. A lot of
features got added to DIH since then. The nightly build of DIH is
compatible with Solr 1.3. BTW 1.3 supports delta import.
>
> Thanks,
> Manu
>
>
> Shalin Shekhar Mangar wrote:
>>
>> Which version of Solr are you using? The deltaImportQuery feature was
>> added
>> recently, you'd need a nightly build for it to work.
>>
>> On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
>> <ma...@gmail.com>wrote:
>>
>>>
>>> Hi Noble,
>>>
>>> I tried with the deltaImportQuery as well. But still I am getting the
>>> same
>>> exception on the server console -
>>>
>>> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
>>> doDeltaImport
>>>
>>> SEVERE: Delta Import Failed
>>> java.lang.NullPointerException
>>> at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>> at
>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>> at
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>
>>> Another thing, I do have proper data in the DB.
>>>
>>> Thanks,
>>> Manu
>>>
>>>
>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>> >
>>> > The same example with deltaImportQuery would look as follows
>>> > <dataConfig>
>>> > <dataSource driver="com.mysql.jdbc.Driver"
>>> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>>> />
>>> > <document name="products">
>>> > <entity name="item" pk="ID" query="select * from item"
>>> > deltaQuery="select id from item where last_modified >
>>> > '${dataimporter.last_index_time}'"
>>> > deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>>> >>
>>> > <field column="NAME" name="name" />
>>> > <field column="NAME" name="nameSort" />
>>> > <field column="NAME" name="alphaNameSort" />
>>> >
>>> > <entity name="feature" pk="ITEM_ID"
>>> > query="select DESCRIPTION from FEATURE where
>>> > ITEM_ID='${item.ID}'"
>>> > deltaQuery="select ITEM_ID from FEATURE where
>>> > last_modified > '${dataimporter.last_index_time}'"
>>> > parentDeltaQuery="select ID from item where
>>> > ID=${feature.ITEM_ID}">
>>> > <field name="features" column="DESCRIPTION" />
>>> > </entity>
>>> >
>>> > <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>>> > query="select CATEGORY_ID from item_category where
>>> > ITEM_ID='${item.ID}'"
>>> > deltaQuery="select ITEM_ID, CATEGORY_ID from
>>> > item_category where last_modified > '${dataimporter.last_index_time}'"
>>> > parentDeltaQuery="select ID from item where
>>> > ID=${item_category.ITEM_ID}">
>>> > <entity name="category" pk="ID"
>>> > query="select DESCRIPTION from category where ID
>>> =
>>> > '${item_category.CATEGORY_ID}'"
>>> > deltaQuery="select ID from category where
>>> > last_modified > '${dataimporter.last_index_time}'"
>>> > parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>>> from
>>> > item_category where CATEGORY_ID=${category.ID}">
>>> > <field column="description" name="cat" />
>>> > </entity>
>>> > </entity>
>>> > </entity>
>>> > </document>
>>> > </dataConfig>
>>> >
>>> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
>>> > <no...@gmail.com> wrote:
>>> >> did you put in the data?
>>> >>
>>> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <ma...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi Shalin,
>>> >>>
>>> >>> I am actually trying out the example provided at
>>> >>> http://wiki.apache.org/solr/DataImportHandler.
>>> >>>
>>> >>> My db-data-config.xml is as following -
>>> >>>
>>> >>> <dataConfig>
>>> >>> <dataSource driver="com.mysql.jdbc.Driver"
>>> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>>> />
>>> >>> <document name="products">
>>> >>> <entity name="item" pk="ID" query="select * from item"
>>> >>> deltaQuery="select id from item where last_modified >
>>> >>> '${dataimporter.last_index_time}'">
>>> >>> <field column="NAME" name="name" />
>>> >>> <field column="NAME" name="nameSort" />
>>> >>> <field column="NAME" name="alphaNameSort" />
>>> >>>
>>> >>> <entity name="feature" pk="ITEM_ID"
>>> >>> query="select DESCRIPTION from FEATURE where
>>> >>> ITEM_ID='${item.ID}'"
>>> >>> deltaQuery="select ITEM_ID from FEATURE where
>>> >>> last_modified > '${dataimporter.last_index_time}'"
>>> >>> parentDeltaQuery="select ID from item where
>>> >>> ID=${feature.ITEM_ID}">
>>> >>> <field name="features" column="DESCRIPTION" />
>>> >>> </entity>
>>> >>>
>>> >>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>>> >>> query="select CATEGORY_ID from item_category where
>>> >>> ITEM_ID='${item.ID}'"
>>> >>> deltaQuery="select ITEM_ID, CATEGORY_ID from
>>> >>> item_category where last_modified >
>>> '${dataimporter.last_index_time}'"
>>> >>> parentDeltaQuery="select ID from item where
>>> >>> ID=${item_category.ITEM_ID}">
>>> >>> <entity name="category" pk="ID"
>>> >>> query="select DESCRIPTION from category where
>>> ID
>>> >>> =
>>> >>> '${item_category.CATEGORY_ID}'"
>>> >>> deltaQuery="select ID from category where
>>> >>> last_modified > '${dataimporter.last_index_time}'"
>>> >>> parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>>> >>> from
>>> >>> item_category where CATEGORY_ID=${category.ID}">
>>> >>> <field column="description" name="cat" />
>>> >>> </entity>
>>> >>> </entity>
>>> >>> </entity>
>>> >>> </document>
>>> >>> </dataConfig>
>>> >>>
>>> ---------------------------------------------------------------------------------------------------
>>> >>>
>>> >>> My DB structure is as following -
>>> >>>
>>> >>> DROP TABLE IF EXISTS solr_demo_db.item;
>>> >>> CREATE TABLE `item` (
>>> >>> `ID` int(2) NOT NULL auto_increment,
>>> >>> `name` varchar(100) default NULL,
>>> >>> `manu` varchar(20) default NULL,
>>> >>> `weight` varchar(20) default NULL,
>>> >>> `price` varchar(20) default NULL,
>>> >>> `popularity` varchar(2) default NULL,
>>> >>> `includes` varchar(10) default NULL,
>>> >>> `last_modified` datetime default NULL,
>>> >>> PRIMARY KEY (`ID`)
>>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>>> >>>
>>> >>> DROP TABLE IF EXISTS solr_demo_db.feature;
>>> >>> CREATE TABLE `feature` (
>>> >>> `DESCRIPTION` varchar(100) default NULL,
>>> >>> `ITEM_ID` int(2) NOT NULL,
>>> >>> `last_modified` datetime default NULL,
>>> >>> PRIMARY KEY (`ITEM_ID`)
>>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>> >>>
>>> >>> DROP TABLE IF EXISTS solr_demo_db.category;
>>> >>> CREATE TABLE `category` (
>>> >>> `ID` int(2) NOT NULL auto_increment,
>>> >>> `DESCRIPTION` varchar(100) default NULL,
>>> >>> `last_modified` date default NULL,
>>> >>> PRIMARY KEY (`ID`)
>>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>>> >>>
>>> >>> DROP TABLE IF EXISTS solr_demo_db.item_category;
>>> >>> CREATE TABLE `item_category` (
>>> >>> `ITEM_ID` int(2) NOT NULL default '0',
>>> >>> `CATEGORY_ID` int(2) NOT NULL default '0',
>>> >>> `last_modified` date default NULL,
>>> >>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
>>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>> >>>
>>> >>> Am I doing something wrong here?
>>> >>>
>>> >>> Thanks,
>>> >>> Manu
>>> >>>
>>> >>>
>>> >>> Shalin Shekhar Mangar wrote:
>>> >>>>
>>> >>>> What does your data-config look like? especially the delta query
>>> part.
>>> >>>>
>>> >>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>>> >>>> <ma...@gmail.com>wrote:
>>> >>>>
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>>
>>> >>>>> I am using Delta DataImport feature to partially refresh the
>>> indexes.
>>> >>>>>
>>> >>>>> Note - full-import is working perfectly fine. But I need to do
>>> >>>>> delta-import
>>> >>>>> as I do not want to rebuild all the indexes.
>>> >>>>>
>>> >>>>> My database structure is same as mentioned in the example at
>>> >>>>> http://wiki.apache.org/solr/DataImportHandler.
>>> >>>>>
>>> >>>>> http://www.nabble.com/file/p21200498/example-schema.png
>>> >>>>>
>>> ---------------------------------------------------------------------------
>>> >>>>> The dataimport.properties has the value as -
>>> >>>>>
>>> >>>>> #Mon Dec 29 14:08:12 IST 2008
>>> >>>>> last_index_time=2008-12-29 14\:08\:12
>>> >>>>>
>>> ---------------------------------------------------------------------------
>>> >>>>> I have changed the last_modified for the 'tem' table as current
>>> >>>>> timestamp.
>>> >>>>> And when I query the DB, I get the following result -
>>> >>>>>
>>> >>>>> http://www.nabble.com/file/p21200498/data.jpg
>>> >>>>>
>>> >>>>>
>>> -------------------------------------------------------------------------------------------------
>>> >>>>>
>>> >>>>> But when I call delta data-import, it returns me the following
>>> >>>>> response -
>>> >>>>>
>>> >>>>> Server Response:
>>> >>>>>
>>> >>>>>
>>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>>> >>>>> response format is experimental. It is likely to change in the
>>> >>>>> future.}
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------------------------------------
>>> >>>>> And when I query for data for '*:*', I get the following -
>>> >>>>> Note - Data returned by query doesnt reflect the correct timestamp
>>> >>>>> from
>>> >>>>> database.
>>> >>>>>
>>> >>>>> SolrDocument(1)
>>> >>>>> Field Name & Value: includes=12
>>> >>>>> Field Name & Value: features=[demo feature, demo feature 1]
>>> >>>>> Field Name & Value: price=12.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>> >>>>> Field Name & Value: manu=manu-12
>>> >>>>> Field Name & Value: sku=1
>>> >>>>> Field Name & Value: id=1
>>> >>>>> Field Name & Value: popularity=1
>>> >>>>> Field Name & Value: weight=12.0
>>> >>>>> SolrDocument(2)
>>> >>>>> Field Name & Value: includes=43
>>> >>>>> Field Name & Value: features=demo feature 2
>>> >>>>> Field Name & Value: price=10.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>> >>>>> Field Name & Value: manu=Demo - 12
>>> >>>>> Field Name & Value: sku=2
>>> >>>>> Field Name & Value: id=2
>>> >>>>> Field Name & Value: popularity=5
>>> >>>>> Field Name & Value: weight=12.0
>>> >>>>> SolrDocument(3)
>>> >>>>> Field Name & Value: includes=1
>>> >>>>> Field Name & Value: features=demo -3
>>> >>>>> Field Name & Value: price=1.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>> >>>>> Field Name & Value: manu=manu - 36
>>> >>>>> Field Name & Value: sku=3
>>> >>>>> Field Name & Value: id=3
>>> >>>>> Field Name & Value: popularity=1
>>> >>>>> Field Name & Value: weight=1.0
>>> >>>>> SolrDocument(4)
>>> >>>>> Field Name & Value: includes=2
>>> >>>>> Field Name & Value: features=demo - 4
>>> >>>>> Field Name & Value: price=2.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>> >>>>> Field Name & Value: manu=manu - 46
>>> >>>>> Field Name & Value: sku=4
>>> >>>>> Field Name & Value: id=4
>>> >>>>> Field Name & Value: popularity=2
>>> >>>>> Field Name & Value: weight=2.0
>>> >>>>> SolrDocument(5)
>>> >>>>> Field Name & Value: includes=3
>>> >>>>> Field Name & Value: features=demo - 5
>>> >>>>> Field Name & Value: price=3.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>> >>>>> Field Name & Value: manu=manu - 56
>>> >>>>> Field Name & Value: sku=5
>>> >>>>> Field Name & Value: id=5
>>> >>>>> Field Name & Value: popularity=3
>>> >>>>> Field Name & Value: weight=3.0
>>> >>>>> SolrDocument(6)
>>> >>>>> Field Name & Value: includes=4
>>> >>>>> Field Name & Value: features=demo - 66
>>> >>>>> Field Name & Value: price=4.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>> >>>>> Field Name & Value: manu=manu - 66
>>> >>>>> Field Name & Value: sku=6
>>> >>>>> Field Name & Value: id=6
>>> >>>>> Field Name & Value: popularity=4
>>> >>>>> Field Name & Value: weight=4.0
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> ------------------------------------------------------------------------------------------
>>> >>>>> My Java code that calls delta import is as follows -
>>> >>>>>
>>> >>>>> public static void main(String[] args) throws IOException,
>>> >>>>> SolrServerException {
>>> >>>>> CommonsHttpSolrServer server = new
>>> >>>>> CommonsHttpSolrServer("http://localhost:8983/solr");
>>> >>>>> BinaryResponseParser parser = new BinaryResponseParser();
>>> >>>>> server.setParser(parser);
>>> >>>>>
>>> >>>>> ModifiableSolrParams params = new ModifiableSolrParams();
>>> >>>>> params.set("qt", "/dataimport");
>>> >>>>> params.set("command", "delta-import");
>>> >>>>> params.set("commit", "true");
>>> >>>>> params.set("wt","json");
>>> >>>>>
>>> >>>>> try{
>>> >>>>>
>>> >>>>> QueryResponse response = server.query(params);
>>> >>>>> server.commit();
>>> >>>>> System.out.println("Server Response: " + response);
>>> >>>>>
>>> >>>>> SolrQuery query = new SolrQuery("*:*");
>>> >>>>> QueryResponse results = server.query(query);
>>> >>>>>
>>> >>>>> SolrDocumentList list = results.getResults();
>>> >>>>>
>>> >>>>> int index = 1;
>>> >>>>> for (SolrDocument solrDocument : list) {
>>> >>>>>
>>> >>>>> System.out.println("SolrDocument("+index+")");
>>> >>>>> Iterator<Entry<String, Object>> iterator =
>>> >>>>> solrDocument.iterator();
>>> >>>>>
>>> >>>>> while(iterator.hasNext()){
>>> >>>>> System.out.println("Field Name & Value:
>>> >>>>> "+iterator.next());
>>> >>>>> }
>>> >>>>>
>>> >>>>> index++;
>>> >>>>> }
>>> >>>>>
>>> >>>>> }catch(Exception ex){
>>> >>>>>
>>> >>>>> System.out.println("Exception Occured:"+ex);
>>> >>>>> }
>>> >>>>>
>>> >>>>> }
>>> >>>>>
>>> ---------------------------------------------------------------------------
>>> >>>>>
>>> >>>>> Does SOLR do some kind of caching? I dont understand as why the
>>> >>>>> updated
>>> >>>>> values are not queried correctly from DB?
>>> >>>>>
>>> >>>>> There are no errors in the logs. But I can see an error on the
>>> server
>>> >>>>> console -
>>> >>>>>
>>> >>>>> SEVERE: Delta Import Failed
>>> >>>>> java.lang.NullPointerException
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>> >>>>> at
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>> >>>>> at
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>> >>>>>
>>> >>>>>
>>> >>>>> What can be the reason for this error?
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Manu
>>> >>>>>
>>> >>>>> --
>>> >>>>> View this message in context:
>>> >>>>>
>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>>> >>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Regards,
>>> >>>> Shalin Shekhar Mangar.
>>> >>>>
>>> >>>>
>>> >>>
>>> >>> --
>>> >>> View this message in context:
>>> >>>
>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
>>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> --Noble Paul
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > --Noble Paul
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21201980.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21202107.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Delta DataImport is not picking the modified value in DB
Posted by Manupriya <ma...@gmail.com>.
Hi Shalin,
I am using apache-solr-1.3.0 in my project. I thought it to be the latest
release. Does it not support delta import?
Thanks,
Manu
Shalin Shekhar Mangar wrote:
>
> Which version of Solr are you using? The deltaImportQuery feature was
> added
> recently, you'd need a nightly build for it to work.
>
> On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
> <ma...@gmail.com>wrote:
>
>>
>> Hi Noble,
>>
>> I tried with the deltaImportQuery as well. But still I am getting the
>> same
>> exception on the server console -
>>
>> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
>> doDeltaImport
>>
>> SEVERE: Delta Import Failed
>> java.lang.NullPointerException
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>> at
>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>
>> Another thing, I do have proper data in the DB.
>>
>> Thanks,
>> Manu
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>> >
>> > The same example with deltaImportQuery would look as follows
>> > <dataConfig>
>> > <dataSource driver="com.mysql.jdbc.Driver"
>> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>> />
>> > <document name="products">
>> > <entity name="item" pk="ID" query="select * from item"
>> > deltaQuery="select id from item where last_modified >
>> > '${dataimporter.last_index_time}'"
>> > deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>> >>
>> > <field column="NAME" name="name" />
>> > <field column="NAME" name="nameSort" />
>> > <field column="NAME" name="alphaNameSort" />
>> >
>> > <entity name="feature" pk="ITEM_ID"
>> > query="select DESCRIPTION from FEATURE where
>> > ITEM_ID='${item.ID}'"
>> > deltaQuery="select ITEM_ID from FEATURE where
>> > last_modified > '${dataimporter.last_index_time}'"
>> > parentDeltaQuery="select ID from item where
>> > ID=${feature.ITEM_ID}">
>> > <field name="features" column="DESCRIPTION" />
>> > </entity>
>> >
>> > <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>> > query="select CATEGORY_ID from item_category where
>> > ITEM_ID='${item.ID}'"
>> > deltaQuery="select ITEM_ID, CATEGORY_ID from
>> > item_category where last_modified > '${dataimporter.last_index_time}'"
>> > parentDeltaQuery="select ID from item where
>> > ID=${item_category.ITEM_ID}">
>> > <entity name="category" pk="ID"
>> > query="select DESCRIPTION from category where ID
>> =
>> > '${item_category.CATEGORY_ID}'"
>> > deltaQuery="select ID from category where
>> > last_modified > '${dataimporter.last_index_time}'"
>> > parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>> from
>> > item_category where CATEGORY_ID=${category.ID}">
>> > <field column="description" name="cat" />
>> > </entity>
>> > </entity>
>> > </entity>
>> > </document>
>> > </dataConfig>
>> >
>> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
>> > <no...@gmail.com> wrote:
>> >> did you put in the data?
>> >>
>> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <ma...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi Shalin,
>> >>>
>> >>> I am actually trying out the example provided at
>> >>> http://wiki.apache.org/solr/DataImportHandler.
>> >>>
>> >>> My db-data-config.xml is as following -
>> >>>
>> >>> <dataConfig>
>> >>> <dataSource driver="com.mysql.jdbc.Driver"
>> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>> />
>> >>> <document name="products">
>> >>> <entity name="item" pk="ID" query="select * from item"
>> >>> deltaQuery="select id from item where last_modified >
>> >>> '${dataimporter.last_index_time}'">
>> >>> <field column="NAME" name="name" />
>> >>> <field column="NAME" name="nameSort" />
>> >>> <field column="NAME" name="alphaNameSort" />
>> >>>
>> >>> <entity name="feature" pk="ITEM_ID"
>> >>> query="select DESCRIPTION from FEATURE where
>> >>> ITEM_ID='${item.ID}'"
>> >>> deltaQuery="select ITEM_ID from FEATURE where
>> >>> last_modified > '${dataimporter.last_index_time}'"
>> >>> parentDeltaQuery="select ID from item where
>> >>> ID=${feature.ITEM_ID}">
>> >>> <field name="features" column="DESCRIPTION" />
>> >>> </entity>
>> >>>
>> >>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>> >>> query="select CATEGORY_ID from item_category where
>> >>> ITEM_ID='${item.ID}'"
>> >>> deltaQuery="select ITEM_ID, CATEGORY_ID from
>> >>> item_category where last_modified >
>> '${dataimporter.last_index_time}'"
>> >>> parentDeltaQuery="select ID from item where
>> >>> ID=${item_category.ITEM_ID}">
>> >>> <entity name="category" pk="ID"
>> >>> query="select DESCRIPTION from category where
>> ID
>> >>> =
>> >>> '${item_category.CATEGORY_ID}'"
>> >>> deltaQuery="select ID from category where
>> >>> last_modified > '${dataimporter.last_index_time}'"
>> >>> parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>> >>> from
>> >>> item_category where CATEGORY_ID=${category.ID}">
>> >>> <field column="description" name="cat" />
>> >>> </entity>
>> >>> </entity>
>> >>> </entity>
>> >>> </document>
>> >>> </dataConfig>
>> >>>
>> ---------------------------------------------------------------------------------------------------
>> >>>
>> >>> My DB structure is as following -
>> >>>
>> >>> DROP TABLE IF EXISTS solr_demo_db.item;
>> >>> CREATE TABLE `item` (
>> >>> `ID` int(2) NOT NULL auto_increment,
>> >>> `name` varchar(100) default NULL,
>> >>> `manu` varchar(20) default NULL,
>> >>> `weight` varchar(20) default NULL,
>> >>> `price` varchar(20) default NULL,
>> >>> `popularity` varchar(2) default NULL,
>> >>> `includes` varchar(10) default NULL,
>> >>> `last_modified` datetime default NULL,
>> >>> PRIMARY KEY (`ID`)
>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>> >>>
>> >>> DROP TABLE IF EXISTS solr_demo_db.feature;
>> >>> CREATE TABLE `feature` (
>> >>> `DESCRIPTION` varchar(100) default NULL,
>> >>> `ITEM_ID` int(2) NOT NULL,
>> >>> `last_modified` datetime default NULL,
>> >>> PRIMARY KEY (`ITEM_ID`)
>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>> >>>
>> >>> DROP TABLE IF EXISTS solr_demo_db.category;
>> >>> CREATE TABLE `category` (
>> >>> `ID` int(2) NOT NULL auto_increment,
>> >>> `DESCRIPTION` varchar(100) default NULL,
>> >>> `last_modified` date default NULL,
>> >>> PRIMARY KEY (`ID`)
>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>> >>>
>> >>> DROP TABLE IF EXISTS solr_demo_db.item_category;
>> >>> CREATE TABLE `item_category` (
>> >>> `ITEM_ID` int(2) NOT NULL default '0',
>> >>> `CATEGORY_ID` int(2) NOT NULL default '0',
>> >>> `last_modified` date default NULL,
>> >>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>> >>>
>> >>> Am I doing something wrong here?
>> >>>
>> >>> Thanks,
>> >>> Manu
>> >>>
>> >>>
>> >>> Shalin Shekhar Mangar wrote:
>> >>>>
>> >>>> What does your data-config look like? especially the delta query
>> part.
>> >>>>
>> >>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>> >>>> <ma...@gmail.com>wrote:
>> >>>>
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> I am using Delta DataImport feature to partially refresh the
>> indexes.
>> >>>>>
>> >>>>> Note - full-import is working perfectly fine. But I need to do
>> >>>>> delta-import
>> >>>>> as I do not want to rebuild all the indexes.
>> >>>>>
>> >>>>> My database structure is same as mentioned in the example at
>> >>>>> http://wiki.apache.org/solr/DataImportHandler.
>> >>>>>
>> >>>>> http://www.nabble.com/file/p21200498/example-schema.png
>> >>>>>
>> ---------------------------------------------------------------------------
>> >>>>> The dataimport.properties has the value as -
>> >>>>>
>> >>>>> #Mon Dec 29 14:08:12 IST 2008
>> >>>>> last_index_time=2008-12-29 14\:08\:12
>> >>>>>
>> ---------------------------------------------------------------------------
>> >>>>> I have changed the last_modified for the 'tem' table as current
>> >>>>> timestamp.
>> >>>>> And when I query the DB, I get the following result -
>> >>>>>
>> >>>>> http://www.nabble.com/file/p21200498/data.jpg
>> >>>>>
>> >>>>>
>> -------------------------------------------------------------------------------------------------
>> >>>>>
>> >>>>> But when I call delta data-import, it returns me the following
>> >>>>> response -
>> >>>>>
>> >>>>> Server Response:
>> >>>>>
>> >>>>>
>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>> >>>>> response format is experimental. It is likely to change in the
>> >>>>> future.}
>> >>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------------------------------------
>> >>>>> And when I query for data for '*:*', I get the following -
>> >>>>> Note - Data returned by query doesnt reflect the correct timestamp
>> >>>>> from
>> >>>>> database.
>> >>>>>
>> >>>>> SolrDocument(1)
>> >>>>> Field Name & Value: includes=12
>> >>>>> Field Name & Value: features=[demo feature, demo feature 1]
>> >>>>> Field Name & Value: price=12.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> >>>>> Field Name & Value: manu=manu-12
>> >>>>> Field Name & Value: sku=1
>> >>>>> Field Name & Value: id=1
>> >>>>> Field Name & Value: popularity=1
>> >>>>> Field Name & Value: weight=12.0
>> >>>>> SolrDocument(2)
>> >>>>> Field Name & Value: includes=43
>> >>>>> Field Name & Value: features=demo feature 2
>> >>>>> Field Name & Value: price=10.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> >>>>> Field Name & Value: manu=Demo - 12
>> >>>>> Field Name & Value: sku=2
>> >>>>> Field Name & Value: id=2
>> >>>>> Field Name & Value: popularity=5
>> >>>>> Field Name & Value: weight=12.0
>> >>>>> SolrDocument(3)
>> >>>>> Field Name & Value: includes=1
>> >>>>> Field Name & Value: features=demo -3
>> >>>>> Field Name & Value: price=1.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> >>>>> Field Name & Value: manu=manu - 36
>> >>>>> Field Name & Value: sku=3
>> >>>>> Field Name & Value: id=3
>> >>>>> Field Name & Value: popularity=1
>> >>>>> Field Name & Value: weight=1.0
>> >>>>> SolrDocument(4)
>> >>>>> Field Name & Value: includes=2
>> >>>>> Field Name & Value: features=demo - 4
>> >>>>> Field Name & Value: price=2.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> >>>>> Field Name & Value: manu=manu - 46
>> >>>>> Field Name & Value: sku=4
>> >>>>> Field Name & Value: id=4
>> >>>>> Field Name & Value: popularity=2
>> >>>>> Field Name & Value: weight=2.0
>> >>>>> SolrDocument(5)
>> >>>>> Field Name & Value: includes=3
>> >>>>> Field Name & Value: features=demo - 5
>> >>>>> Field Name & Value: price=3.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> >>>>> Field Name & Value: manu=manu - 56
>> >>>>> Field Name & Value: sku=5
>> >>>>> Field Name & Value: id=5
>> >>>>> Field Name & Value: popularity=3
>> >>>>> Field Name & Value: weight=3.0
>> >>>>> SolrDocument(6)
>> >>>>> Field Name & Value: includes=4
>> >>>>> Field Name & Value: features=demo - 66
>> >>>>> Field Name & Value: price=4.0
>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> >>>>> Field Name & Value: manu=manu - 66
>> >>>>> Field Name & Value: sku=6
>> >>>>> Field Name & Value: id=6
>> >>>>> Field Name & Value: popularity=4
>> >>>>> Field Name & Value: weight=4.0
>> >>>>>
>> >>>>>
>> >>>>>
>> ------------------------------------------------------------------------------------------
>> >>>>> My Java code that calls delta import is as follows -
>> >>>>>
>> >>>>> public static void main(String[] args) throws IOException,
>> >>>>> SolrServerException {
>> >>>>> CommonsHttpSolrServer server = new
>> >>>>> CommonsHttpSolrServer("http://localhost:8983/solr");
>> >>>>> BinaryResponseParser parser = new BinaryResponseParser();
>> >>>>> server.setParser(parser);
>> >>>>>
>> >>>>> ModifiableSolrParams params = new ModifiableSolrParams();
>> >>>>> params.set("qt", "/dataimport");
>> >>>>> params.set("command", "delta-import");
>> >>>>> params.set("commit", "true");
>> >>>>> params.set("wt","json");
>> >>>>>
>> >>>>> try{
>> >>>>>
>> >>>>> QueryResponse response = server.query(params);
>> >>>>> server.commit();
>> >>>>> System.out.println("Server Response: " + response);
>> >>>>>
>> >>>>> SolrQuery query = new SolrQuery("*:*");
>> >>>>> QueryResponse results = server.query(query);
>> >>>>>
>> >>>>> SolrDocumentList list = results.getResults();
>> >>>>>
>> >>>>> int index = 1;
>> >>>>> for (SolrDocument solrDocument : list) {
>> >>>>>
>> >>>>> System.out.println("SolrDocument("+index+")");
>> >>>>> Iterator<Entry<String, Object>> iterator =
>> >>>>> solrDocument.iterator();
>> >>>>>
>> >>>>> while(iterator.hasNext()){
>> >>>>> System.out.println("Field Name & Value:
>> >>>>> "+iterator.next());
>> >>>>> }
>> >>>>>
>> >>>>> index++;
>> >>>>> }
>> >>>>>
>> >>>>> }catch(Exception ex){
>> >>>>>
>> >>>>> System.out.println("Exception Occured:"+ex);
>> >>>>> }
>> >>>>>
>> >>>>> }
>> >>>>>
>> ---------------------------------------------------------------------------
>> >>>>>
>> >>>>> Does SOLR do some kind of caching? I dont understand as why the
>> >>>>> updated
>> >>>>> values are not queried correctly from DB?
>> >>>>>
>> >>>>> There are no errors in the logs. But I can see an error on the
>> server
>> >>>>> console -
>> >>>>>
>> >>>>> SEVERE: Delta Import Failed
>> >>>>> java.lang.NullPointerException
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>> >>>>> at
>> >>>>>
>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>> >>>>> at
>> >>>>>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>> >>>>>
>> >>>>>
>> >>>>> What can be the reason for this error?
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Manu
>> >>>>>
>> >>>>> --
>> >>>>> View this message in context:
>> >>>>>
>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>> >>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Regards,
>> >>>> Shalin Shekhar Mangar.
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> --Noble Paul
>> >>
>> >
>> >
>> >
>> > --
>> > --Noble Paul
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21201980.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21202107.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Delta DataImport is not picking the modified value in DB
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Which version of Solr are you using? The deltaImportQuery feature was added
recently, you'd need a nightly build for it to work.
On Mon, Dec 29, 2008 at 4:57 PM, Manupriya <ma...@gmail.com>wrote:
>
> Hi Noble,
>
> I tried with the deltaImportQuery as well. But still I am getting the same
> exception on the server console -
>
> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
>
> SEVERE: Delta Import Failed
> java.lang.NullPointerException
> at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
> at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
> at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
> at
>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
> at
>
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
> at
>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
> at
>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>
> Another thing, I do have proper data in the DB.
>
> Thanks,
> Manu
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
> >
> > The same example with deltaImportQuery would look as follows
> > <dataConfig>
> > <dataSource driver="com.mysql.jdbc.Driver"
> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
> > <document name="products">
> > <entity name="item" pk="ID" query="select * from item"
> > deltaQuery="select id from item where last_modified >
> > '${dataimporter.last_index_time}'"
> > deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
> >>
> > <field column="NAME" name="name" />
> > <field column="NAME" name="nameSort" />
> > <field column="NAME" name="alphaNameSort" />
> >
> > <entity name="feature" pk="ITEM_ID"
> > query="select DESCRIPTION from FEATURE where
> > ITEM_ID='${item.ID}'"
> > deltaQuery="select ITEM_ID from FEATURE where
> > last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ID from item where
> > ID=${feature.ITEM_ID}">
> > <field name="features" column="DESCRIPTION" />
> > </entity>
> >
> > <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
> > query="select CATEGORY_ID from item_category where
> > ITEM_ID='${item.ID}'"
> > deltaQuery="select ITEM_ID, CATEGORY_ID from
> > item_category where last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ID from item where
> > ID=${item_category.ITEM_ID}">
> > <entity name="category" pk="ID"
> > query="select DESCRIPTION from category where ID =
> > '${item_category.CATEGORY_ID}'"
> > deltaQuery="select ID from category where
> > last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> > item_category where CATEGORY_ID=${category.ID}">
> > <field column="description" name="cat" />
> > </entity>
> > </entity>
> > </entity>
> > </document>
> > </dataConfig>
> >
> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
> > <no...@gmail.com> wrote:
> >> did you put in the data?
> >>
> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <ma...@gmail.com>
> >> wrote:
> >>>
> >>> Hi Shalin,
> >>>
> >>> I am actually trying out the example provided at
> >>> http://wiki.apache.org/solr/DataImportHandler.
> >>>
> >>> My db-data-config.xml is as following -
> >>>
> >>> <dataConfig>
> >>> <dataSource driver="com.mysql.jdbc.Driver"
> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
> />
> >>> <document name="products">
> >>> <entity name="item" pk="ID" query="select * from item"
> >>> deltaQuery="select id from item where last_modified >
> >>> '${dataimporter.last_index_time}'">
> >>> <field column="NAME" name="name" />
> >>> <field column="NAME" name="nameSort" />
> >>> <field column="NAME" name="alphaNameSort" />
> >>>
> >>> <entity name="feature" pk="ITEM_ID"
> >>> query="select DESCRIPTION from FEATURE where
> >>> ITEM_ID='${item.ID}'"
> >>> deltaQuery="select ITEM_ID from FEATURE where
> >>> last_modified > '${dataimporter.last_index_time}'"
> >>> parentDeltaQuery="select ID from item where
> >>> ID=${feature.ITEM_ID}">
> >>> <field name="features" column="DESCRIPTION" />
> >>> </entity>
> >>>
> >>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
> >>> query="select CATEGORY_ID from item_category where
> >>> ITEM_ID='${item.ID}'"
> >>> deltaQuery="select ITEM_ID, CATEGORY_ID from
> >>> item_category where last_modified > '${dataimporter.last_index_time}'"
> >>> parentDeltaQuery="select ID from item where
> >>> ID=${item_category.ITEM_ID}">
> >>> <entity name="category" pk="ID"
> >>> query="select DESCRIPTION from category where ID
> >>> =
> >>> '${item_category.CATEGORY_ID}'"
> >>> deltaQuery="select ID from category where
> >>> last_modified > '${dataimporter.last_index_time}'"
> >>> parentDeltaQuery="select ITEM_ID, CATEGORY_ID
> >>> from
> >>> item_category where CATEGORY_ID=${category.ID}">
> >>> <field column="description" name="cat" />
> >>> </entity>
> >>> </entity>
> >>> </entity>
> >>> </document>
> >>> </dataConfig>
> >>>
> ---------------------------------------------------------------------------------------------------
> >>>
> >>> My DB structure is as following -
> >>>
> >>> DROP TABLE IF EXISTS solr_demo_db.item;
> >>> CREATE TABLE `item` (
> >>> `ID` int(2) NOT NULL auto_increment,
> >>> `name` varchar(100) default NULL,
> >>> `manu` varchar(20) default NULL,
> >>> `weight` varchar(20) default NULL,
> >>> `price` varchar(20) default NULL,
> >>> `popularity` varchar(2) default NULL,
> >>> `includes` varchar(10) default NULL,
> >>> `last_modified` datetime default NULL,
> >>> PRIMARY KEY (`ID`)
> >>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
> >>>
> >>> DROP TABLE IF EXISTS solr_demo_db.feature;
> >>> CREATE TABLE `feature` (
> >>> `DESCRIPTION` varchar(100) default NULL,
> >>> `ITEM_ID` int(2) NOT NULL,
> >>> `last_modified` datetime default NULL,
> >>> PRIMARY KEY (`ITEM_ID`)
> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> >>>
> >>> DROP TABLE IF EXISTS solr_demo_db.category;
> >>> CREATE TABLE `category` (
> >>> `ID` int(2) NOT NULL auto_increment,
> >>> `DESCRIPTION` varchar(100) default NULL,
> >>> `last_modified` date default NULL,
> >>> PRIMARY KEY (`ID`)
> >>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
> >>>
> >>> DROP TABLE IF EXISTS solr_demo_db.item_category;
> >>> CREATE TABLE `item_category` (
> >>> `ITEM_ID` int(2) NOT NULL default '0',
> >>> `CATEGORY_ID` int(2) NOT NULL default '0',
> >>> `last_modified` date default NULL,
> >>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> >>>
> >>> Am I doing something wrong here?
> >>>
> >>> Thanks,
> >>> Manu
> >>>
> >>>
> >>> Shalin Shekhar Mangar wrote:
> >>>>
> >>>> What does your data-config look like? especially the delta query part.
> >>>>
> >>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
> >>>> <ma...@gmail.com>wrote:
> >>>>
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I am using Delta DataImport feature to partially refresh the indexes.
> >>>>>
> >>>>> Note - full-import is working perfectly fine. But I need to do
> >>>>> delta-import
> >>>>> as I do not want to rebuild all the indexes.
> >>>>>
> >>>>> My database structure is same as mentioned in the example at
> >>>>> http://wiki.apache.org/solr/DataImportHandler.
> >>>>>
> >>>>> http://www.nabble.com/file/p21200498/example-schema.png
> >>>>>
> ---------------------------------------------------------------------------
> >>>>> The dataimport.properties has the value as -
> >>>>>
> >>>>> #Mon Dec 29 14:08:12 IST 2008
> >>>>> last_index_time=2008-12-29 14\:08\:12
> >>>>>
> ---------------------------------------------------------------------------
> >>>>> I have changed the last_modified for the 'tem' table as current
> >>>>> timestamp.
> >>>>> And when I query the DB, I get the following result -
> >>>>>
> >>>>> http://www.nabble.com/file/p21200498/data.jpg
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------------------------
> >>>>>
> >>>>> But when I call delta data-import, it returns me the following
> >>>>> response -
> >>>>>
> >>>>> Server Response:
> >>>>>
> >>>>>
> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
> >>>>> response format is experimental. It is likely to change in the
> >>>>> future.}
> >>>>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------------------------------------
> >>>>> And when I query for data for '*:*', I get the following -
> >>>>> Note - Data returned by query doesnt reflect the correct timestamp
> >>>>> from
> >>>>> database.
> >>>>>
> >>>>> SolrDocument(1)
> >>>>> Field Name & Value: includes=12
> >>>>> Field Name & Value: features=[demo feature, demo feature 1]
> >>>>> Field Name & Value: price=12.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> >>>>> Field Name & Value: manu=manu-12
> >>>>> Field Name & Value: sku=1
> >>>>> Field Name & Value: id=1
> >>>>> Field Name & Value: popularity=1
> >>>>> Field Name & Value: weight=12.0
> >>>>> SolrDocument(2)
> >>>>> Field Name & Value: includes=43
> >>>>> Field Name & Value: features=demo feature 2
> >>>>> Field Name & Value: price=10.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> >>>>> Field Name & Value: manu=Demo - 12
> >>>>> Field Name & Value: sku=2
> >>>>> Field Name & Value: id=2
> >>>>> Field Name & Value: popularity=5
> >>>>> Field Name & Value: weight=12.0
> >>>>> SolrDocument(3)
> >>>>> Field Name & Value: includes=1
> >>>>> Field Name & Value: features=demo -3
> >>>>> Field Name & Value: price=1.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> >>>>> Field Name & Value: manu=manu - 36
> >>>>> Field Name & Value: sku=3
> >>>>> Field Name & Value: id=3
> >>>>> Field Name & Value: popularity=1
> >>>>> Field Name & Value: weight=1.0
> >>>>> SolrDocument(4)
> >>>>> Field Name & Value: includes=2
> >>>>> Field Name & Value: features=demo - 4
> >>>>> Field Name & Value: price=2.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> >>>>> Field Name & Value: manu=manu - 46
> >>>>> Field Name & Value: sku=4
> >>>>> Field Name & Value: id=4
> >>>>> Field Name & Value: popularity=2
> >>>>> Field Name & Value: weight=2.0
> >>>>> SolrDocument(5)
> >>>>> Field Name & Value: includes=3
> >>>>> Field Name & Value: features=demo - 5
> >>>>> Field Name & Value: price=3.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> >>>>> Field Name & Value: manu=manu - 56
> >>>>> Field Name & Value: sku=5
> >>>>> Field Name & Value: id=5
> >>>>> Field Name & Value: popularity=3
> >>>>> Field Name & Value: weight=3.0
> >>>>> SolrDocument(6)
> >>>>> Field Name & Value: includes=4
> >>>>> Field Name & Value: features=demo - 66
> >>>>> Field Name & Value: price=4.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> >>>>> Field Name & Value: manu=manu - 66
> >>>>> Field Name & Value: sku=6
> >>>>> Field Name & Value: id=6
> >>>>> Field Name & Value: popularity=4
> >>>>> Field Name & Value: weight=4.0
> >>>>>
> >>>>>
> >>>>>
> ------------------------------------------------------------------------------------------
> >>>>> My Java code that calls delta import is as follows -
> >>>>>
> >>>>> public static void main(String[] args) throws IOException,
> >>>>> SolrServerException {
> >>>>> CommonsHttpSolrServer server = new
> >>>>> CommonsHttpSolrServer("http://localhost:8983/solr");
> >>>>> BinaryResponseParser parser = new BinaryResponseParser();
> >>>>> server.setParser(parser);
> >>>>>
> >>>>> ModifiableSolrParams params = new ModifiableSolrParams();
> >>>>> params.set("qt", "/dataimport");
> >>>>> params.set("command", "delta-import");
> >>>>> params.set("commit", "true");
> >>>>> params.set("wt","json");
> >>>>>
> >>>>> try{
> >>>>>
> >>>>> QueryResponse response = server.query(params);
> >>>>> server.commit();
> >>>>> System.out.println("Server Response: " + response);
> >>>>>
> >>>>> SolrQuery query = new SolrQuery("*:*");
> >>>>> QueryResponse results = server.query(query);
> >>>>>
> >>>>> SolrDocumentList list = results.getResults();
> >>>>>
> >>>>> int index = 1;
> >>>>> for (SolrDocument solrDocument : list) {
> >>>>>
> >>>>> System.out.println("SolrDocument("+index+")");
> >>>>> Iterator<Entry<String, Object>> iterator =
> >>>>> solrDocument.iterator();
> >>>>>
> >>>>> while(iterator.hasNext()){
> >>>>> System.out.println("Field Name & Value:
> >>>>> "+iterator.next());
> >>>>> }
> >>>>>
> >>>>> index++;
> >>>>> }
> >>>>>
> >>>>> }catch(Exception ex){
> >>>>>
> >>>>> System.out.println("Exception Occured:"+ex);
> >>>>> }
> >>>>>
> >>>>> }
> >>>>>
> ---------------------------------------------------------------------------
> >>>>>
> >>>>> Does SOLR do some kind of caching? I dont understand as why the
> >>>>> updated
> >>>>> values are not queried correctly from DB?
> >>>>>
> >>>>> There are no errors in the logs. But I can see an error on the server
> >>>>> console -
> >>>>>
> >>>>> SEVERE: Delta Import Failed
> >>>>> java.lang.NullPointerException
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
> >>>>> at
> >>>>>
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
> >>>>> at
> >>>>>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
> >>>>>
> >>>>>
> >>>>> What can be the reason for this error?
> >>>>>
> >>>>> Thanks,
> >>>>> Manu
> >>>>>
> >>>>> --
> >>>>> View this message in context:
> >>>>>
> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
> >>>>> Sent from the Solr - User mailing list archive at Nabble.com.
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Regards,
> >>>> Shalin Shekhar Mangar.
> >>>>
> >>>>
> >>>
> >>> --
> >>> View this message in context:
> >>>
> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
> >>> Sent from the Solr - User mailing list archive at Nabble.com.
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> --Noble Paul
> >>
> >
> >
> >
> > --
> > --Noble Paul
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21201980.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: Delta DataImport is not picking the modified value in DB
Posted by Manupriya <ma...@gmail.com>.
Hi Noble,
I tried with the deltaImportQuery as well. But still I am getting the same
exception on the server console -
Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
SEVERE: Delta Import Failed
java.lang.NullPointerException
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
at
org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
Another thing, I do have proper data in the DB.
Thanks,
Manu
Noble Paul നോബിള് नोब्ळ् wrote:
>
> The same example with deltaImportQuery would look as follows
> <dataConfig>
> <dataSource driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
> <document name="products">
> <entity name="item" pk="ID" query="select * from item"
> deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'"
> deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>>
> <field column="NAME" name="name" />
> <field column="NAME" name="nameSort" />
> <field column="NAME" name="alphaNameSort" />
>
> <entity name="feature" pk="ITEM_ID"
> query="select DESCRIPTION from FEATURE where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID from FEATURE where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${feature.ITEM_ID}">
> <field name="features" column="DESCRIPTION" />
> </entity>
>
> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
> query="select CATEGORY_ID from item_category where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${item_category.ITEM_ID}">
> <entity name="category" pk="ID"
> query="select DESCRIPTION from category where ID =
> '${item_category.CATEGORY_ID}'"
> deltaQuery="select ID from category where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where CATEGORY_ID=${category.ID}">
> <field column="description" name="cat" />
> </entity>
> </entity>
> </entity>
> </document>
> </dataConfig>
>
> On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
> <no...@gmail.com> wrote:
>> did you put in the data?
>>
>> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <ma...@gmail.com>
>> wrote:
>>>
>>> Hi Shalin,
>>>
>>> I am actually trying out the example provided at
>>> http://wiki.apache.org/solr/DataImportHandler.
>>>
>>> My db-data-config.xml is as following -
>>>
>>> <dataConfig>
>>> <dataSource driver="com.mysql.jdbc.Driver"
>>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
>>> <document name="products">
>>> <entity name="item" pk="ID" query="select * from item"
>>> deltaQuery="select id from item where last_modified >
>>> '${dataimporter.last_index_time}'">
>>> <field column="NAME" name="name" />
>>> <field column="NAME" name="nameSort" />
>>> <field column="NAME" name="alphaNameSort" />
>>>
>>> <entity name="feature" pk="ITEM_ID"
>>> query="select DESCRIPTION from FEATURE where
>>> ITEM_ID='${item.ID}'"
>>> deltaQuery="select ITEM_ID from FEATURE where
>>> last_modified > '${dataimporter.last_index_time}'"
>>> parentDeltaQuery="select ID from item where
>>> ID=${feature.ITEM_ID}">
>>> <field name="features" column="DESCRIPTION" />
>>> </entity>
>>>
>>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>>> query="select CATEGORY_ID from item_category where
>>> ITEM_ID='${item.ID}'"
>>> deltaQuery="select ITEM_ID, CATEGORY_ID from
>>> item_category where last_modified > '${dataimporter.last_index_time}'"
>>> parentDeltaQuery="select ID from item where
>>> ID=${item_category.ITEM_ID}">
>>> <entity name="category" pk="ID"
>>> query="select DESCRIPTION from category where ID
>>> =
>>> '${item_category.CATEGORY_ID}'"
>>> deltaQuery="select ID from category where
>>> last_modified > '${dataimporter.last_index_time}'"
>>> parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>>> from
>>> item_category where CATEGORY_ID=${category.ID}">
>>> <field column="description" name="cat" />
>>> </entity>
>>> </entity>
>>> </entity>
>>> </document>
>>> </dataConfig>
>>> ---------------------------------------------------------------------------------------------------
>>>
>>> My DB structure is as following -
>>>
>>> DROP TABLE IF EXISTS solr_demo_db.item;
>>> CREATE TABLE `item` (
>>> `ID` int(2) NOT NULL auto_increment,
>>> `name` varchar(100) default NULL,
>>> `manu` varchar(20) default NULL,
>>> `weight` varchar(20) default NULL,
>>> `price` varchar(20) default NULL,
>>> `popularity` varchar(2) default NULL,
>>> `includes` varchar(10) default NULL,
>>> `last_modified` datetime default NULL,
>>> PRIMARY KEY (`ID`)
>>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>>>
>>> DROP TABLE IF EXISTS solr_demo_db.feature;
>>> CREATE TABLE `feature` (
>>> `DESCRIPTION` varchar(100) default NULL,
>>> `ITEM_ID` int(2) NOT NULL,
>>> `last_modified` datetime default NULL,
>>> PRIMARY KEY (`ITEM_ID`)
>>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>>
>>> DROP TABLE IF EXISTS solr_demo_db.category;
>>> CREATE TABLE `category` (
>>> `ID` int(2) NOT NULL auto_increment,
>>> `DESCRIPTION` varchar(100) default NULL,
>>> `last_modified` date default NULL,
>>> PRIMARY KEY (`ID`)
>>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>>>
>>> DROP TABLE IF EXISTS solr_demo_db.item_category;
>>> CREATE TABLE `item_category` (
>>> `ITEM_ID` int(2) NOT NULL default '0',
>>> `CATEGORY_ID` int(2) NOT NULL default '0',
>>> `last_modified` date default NULL,
>>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
>>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>>
>>> Am I doing something wrong here?
>>>
>>> Thanks,
>>> Manu
>>>
>>>
>>> Shalin Shekhar Mangar wrote:
>>>>
>>>> What does your data-config look like? especially the delta query part.
>>>>
>>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>>>> <ma...@gmail.com>wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am using Delta DataImport feature to partially refresh the indexes.
>>>>>
>>>>> Note - full-import is working perfectly fine. But I need to do
>>>>> delta-import
>>>>> as I do not want to rebuild all the indexes.
>>>>>
>>>>> My database structure is same as mentioned in the example at
>>>>> http://wiki.apache.org/solr/DataImportHandler.
>>>>>
>>>>> http://www.nabble.com/file/p21200498/example-schema.png
>>>>> ---------------------------------------------------------------------------
>>>>> The dataimport.properties has the value as -
>>>>>
>>>>> #Mon Dec 29 14:08:12 IST 2008
>>>>> last_index_time=2008-12-29 14\:08\:12
>>>>> ---------------------------------------------------------------------------
>>>>> I have changed the last_modified for the 'tem' table as current
>>>>> timestamp.
>>>>> And when I query the DB, I get the following result -
>>>>>
>>>>> http://www.nabble.com/file/p21200498/data.jpg
>>>>>
>>>>> -------------------------------------------------------------------------------------------------
>>>>>
>>>>> But when I call delta data-import, it returns me the following
>>>>> response -
>>>>>
>>>>> Server Response:
>>>>>
>>>>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>>>>> response format is experimental. It is likely to change in the
>>>>> future.}
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------------------------------
>>>>> And when I query for data for '*:*', I get the following -
>>>>> Note - Data returned by query doesnt reflect the correct timestamp
>>>>> from
>>>>> database.
>>>>>
>>>>> SolrDocument(1)
>>>>> Field Name & Value: includes=12
>>>>> Field Name & Value: features=[demo feature, demo feature 1]
>>>>> Field Name & Value: price=12.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>>> Field Name & Value: manu=manu-12
>>>>> Field Name & Value: sku=1
>>>>> Field Name & Value: id=1
>>>>> Field Name & Value: popularity=1
>>>>> Field Name & Value: weight=12.0
>>>>> SolrDocument(2)
>>>>> Field Name & Value: includes=43
>>>>> Field Name & Value: features=demo feature 2
>>>>> Field Name & Value: price=10.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>>> Field Name & Value: manu=Demo - 12
>>>>> Field Name & Value: sku=2
>>>>> Field Name & Value: id=2
>>>>> Field Name & Value: popularity=5
>>>>> Field Name & Value: weight=12.0
>>>>> SolrDocument(3)
>>>>> Field Name & Value: includes=1
>>>>> Field Name & Value: features=demo -3
>>>>> Field Name & Value: price=1.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>>> Field Name & Value: manu=manu - 36
>>>>> Field Name & Value: sku=3
>>>>> Field Name & Value: id=3
>>>>> Field Name & Value: popularity=1
>>>>> Field Name & Value: weight=1.0
>>>>> SolrDocument(4)
>>>>> Field Name & Value: includes=2
>>>>> Field Name & Value: features=demo - 4
>>>>> Field Name & Value: price=2.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>>> Field Name & Value: manu=manu - 46
>>>>> Field Name & Value: sku=4
>>>>> Field Name & Value: id=4
>>>>> Field Name & Value: popularity=2
>>>>> Field Name & Value: weight=2.0
>>>>> SolrDocument(5)
>>>>> Field Name & Value: includes=3
>>>>> Field Name & Value: features=demo - 5
>>>>> Field Name & Value: price=3.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>>> Field Name & Value: manu=manu - 56
>>>>> Field Name & Value: sku=5
>>>>> Field Name & Value: id=5
>>>>> Field Name & Value: popularity=3
>>>>> Field Name & Value: weight=3.0
>>>>> SolrDocument(6)
>>>>> Field Name & Value: includes=4
>>>>> Field Name & Value: features=demo - 66
>>>>> Field Name & Value: price=4.0
>>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>>> Field Name & Value: manu=manu - 66
>>>>> Field Name & Value: sku=6
>>>>> Field Name & Value: id=6
>>>>> Field Name & Value: popularity=4
>>>>> Field Name & Value: weight=4.0
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------------------
>>>>> My Java code that calls delta import is as follows -
>>>>>
>>>>> public static void main(String[] args) throws IOException,
>>>>> SolrServerException {
>>>>> CommonsHttpSolrServer server = new
>>>>> CommonsHttpSolrServer("http://localhost:8983/solr");
>>>>> BinaryResponseParser parser = new BinaryResponseParser();
>>>>> server.setParser(parser);
>>>>>
>>>>> ModifiableSolrParams params = new ModifiableSolrParams();
>>>>> params.set("qt", "/dataimport");
>>>>> params.set("command", "delta-import");
>>>>> params.set("commit", "true");
>>>>> params.set("wt","json");
>>>>>
>>>>> try{
>>>>>
>>>>> QueryResponse response = server.query(params);
>>>>> server.commit();
>>>>> System.out.println("Server Response: " + response);
>>>>>
>>>>> SolrQuery query = new SolrQuery("*:*");
>>>>> QueryResponse results = server.query(query);
>>>>>
>>>>> SolrDocumentList list = results.getResults();
>>>>>
>>>>> int index = 1;
>>>>> for (SolrDocument solrDocument : list) {
>>>>>
>>>>> System.out.println("SolrDocument("+index+")");
>>>>> Iterator<Entry<String, Object>> iterator =
>>>>> solrDocument.iterator();
>>>>>
>>>>> while(iterator.hasNext()){
>>>>> System.out.println("Field Name & Value:
>>>>> "+iterator.next());
>>>>> }
>>>>>
>>>>> index++;
>>>>> }
>>>>>
>>>>> }catch(Exception ex){
>>>>>
>>>>> System.out.println("Exception Occured:"+ex);
>>>>> }
>>>>>
>>>>> }
>>>>> ---------------------------------------------------------------------------
>>>>>
>>>>> Does SOLR do some kind of caching? I dont understand as why the
>>>>> updated
>>>>> values are not queried correctly from DB?
>>>>>
>>>>> There are no errors in the logs. But I can see an error on the server
>>>>> console -
>>>>>
>>>>> SEVERE: Delta Import Failed
>>>>> java.lang.NullPointerException
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>>>> at
>>>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>>>> at
>>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>>>> at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>>>
>>>>>
>>>>> What can be the reason for this error?
>>>>>
>>>>> Thanks,
>>>>> Manu
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context: http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21201980.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Delta DataImport is not picking the modified value in DB
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
The same example with deltaImportQuery would look as follows
<dataConfig>
<dataSource driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
<document name="products">
<entity name="item" pk="ID" query="select * from item"
deltaQuery="select id from item where last_modified >
'${dataimporter.last_index_time}'"
deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>
<field column="NAME" name="name" />
<field column="NAME" name="nameSort" />
<field column="NAME" name="alphaNameSort" />
<entity name="feature" pk="ITEM_ID"
query="select DESCRIPTION from FEATURE where
ITEM_ID='${item.ID}'"
deltaQuery="select ITEM_ID from FEATURE where
last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ID from item where
ID=${feature.ITEM_ID}">
<field name="features" column="DESCRIPTION" />
</entity>
<entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
query="select CATEGORY_ID from item_category where
ITEM_ID='${item.ID}'"
deltaQuery="select ITEM_ID, CATEGORY_ID from
item_category where last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ID from item where
ID=${item_category.ITEM_ID}">
<entity name="category" pk="ID"
query="select DESCRIPTION from category where ID =
'${item_category.CATEGORY_ID}'"
deltaQuery="select ID from category where
last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
item_category where CATEGORY_ID=${category.ID}">
<field column="description" name="cat" />
</entity>
</entity>
</entity>
</document>
</dataConfig>
On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
<no...@gmail.com> wrote:
> did you put in the data?
>
> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <ma...@gmail.com> wrote:
>>
>> Hi Shalin,
>>
>> I am actually trying out the example provided at
>> http://wiki.apache.org/solr/DataImportHandler.
>>
>> My db-data-config.xml is as following -
>>
>> <dataConfig>
>> <dataSource driver="com.mysql.jdbc.Driver"
>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
>> <document name="products">
>> <entity name="item" pk="ID" query="select * from item"
>> deltaQuery="select id from item where last_modified >
>> '${dataimporter.last_index_time}'">
>> <field column="NAME" name="name" />
>> <field column="NAME" name="nameSort" />
>> <field column="NAME" name="alphaNameSort" />
>>
>> <entity name="feature" pk="ITEM_ID"
>> query="select DESCRIPTION from FEATURE where
>> ITEM_ID='${item.ID}'"
>> deltaQuery="select ITEM_ID from FEATURE where
>> last_modified > '${dataimporter.last_index_time}'"
>> parentDeltaQuery="select ID from item where
>> ID=${feature.ITEM_ID}">
>> <field name="features" column="DESCRIPTION" />
>> </entity>
>>
>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>> query="select CATEGORY_ID from item_category where
>> ITEM_ID='${item.ID}'"
>> deltaQuery="select ITEM_ID, CATEGORY_ID from
>> item_category where last_modified > '${dataimporter.last_index_time}'"
>> parentDeltaQuery="select ID from item where
>> ID=${item_category.ITEM_ID}">
>> <entity name="category" pk="ID"
>> query="select DESCRIPTION from category where ID =
>> '${item_category.CATEGORY_ID}'"
>> deltaQuery="select ID from category where
>> last_modified > '${dataimporter.last_index_time}'"
>> parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
>> item_category where CATEGORY_ID=${category.ID}">
>> <field column="description" name="cat" />
>> </entity>
>> </entity>
>> </entity>
>> </document>
>> </dataConfig>
>> ---------------------------------------------------------------------------------------------------
>>
>> My DB structure is as following -
>>
>> DROP TABLE IF EXISTS solr_demo_db.item;
>> CREATE TABLE `item` (
>> `ID` int(2) NOT NULL auto_increment,
>> `name` varchar(100) default NULL,
>> `manu` varchar(20) default NULL,
>> `weight` varchar(20) default NULL,
>> `price` varchar(20) default NULL,
>> `popularity` varchar(2) default NULL,
>> `includes` varchar(10) default NULL,
>> `last_modified` datetime default NULL,
>> PRIMARY KEY (`ID`)
>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>>
>> DROP TABLE IF EXISTS solr_demo_db.feature;
>> CREATE TABLE `feature` (
>> `DESCRIPTION` varchar(100) default NULL,
>> `ITEM_ID` int(2) NOT NULL,
>> `last_modified` datetime default NULL,
>> PRIMARY KEY (`ITEM_ID`)
>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>
>> DROP TABLE IF EXISTS solr_demo_db.category;
>> CREATE TABLE `category` (
>> `ID` int(2) NOT NULL auto_increment,
>> `DESCRIPTION` varchar(100) default NULL,
>> `last_modified` date default NULL,
>> PRIMARY KEY (`ID`)
>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>>
>> DROP TABLE IF EXISTS solr_demo_db.item_category;
>> CREATE TABLE `item_category` (
>> `ITEM_ID` int(2) NOT NULL default '0',
>> `CATEGORY_ID` int(2) NOT NULL default '0',
>> `last_modified` date default NULL,
>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>
>> Am I doing something wrong here?
>>
>> Thanks,
>> Manu
>>
>>
>> Shalin Shekhar Mangar wrote:
>>>
>>> What does your data-config look like? especially the delta query part.
>>>
>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>>> <ma...@gmail.com>wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I am using Delta DataImport feature to partially refresh the indexes.
>>>>
>>>> Note - full-import is working perfectly fine. But I need to do
>>>> delta-import
>>>> as I do not want to rebuild all the indexes.
>>>>
>>>> My database structure is same as mentioned in the example at
>>>> http://wiki.apache.org/solr/DataImportHandler.
>>>>
>>>> http://www.nabble.com/file/p21200498/example-schema.png
>>>> ---------------------------------------------------------------------------
>>>> The dataimport.properties has the value as -
>>>>
>>>> #Mon Dec 29 14:08:12 IST 2008
>>>> last_index_time=2008-12-29 14\:08\:12
>>>> ---------------------------------------------------------------------------
>>>> I have changed the last_modified for the 'tem' table as current
>>>> timestamp.
>>>> And when I query the DB, I get the following result -
>>>>
>>>> http://www.nabble.com/file/p21200498/data.jpg
>>>>
>>>> -------------------------------------------------------------------------------------------------
>>>>
>>>> But when I call delta data-import, it returns me the following response -
>>>>
>>>> Server Response:
>>>>
>>>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>>>> response format is experimental. It is likely to change in the future.}
>>>>
>>>>
>>>> ---------------------------------------------------------------------------------------------------
>>>> And when I query for data for '*:*', I get the following -
>>>> Note - Data returned by query doesnt reflect the correct timestamp from
>>>> database.
>>>>
>>>> SolrDocument(1)
>>>> Field Name & Value: includes=12
>>>> Field Name & Value: features=[demo feature, demo feature 1]
>>>> Field Name & Value: price=12.0
>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>> Field Name & Value: manu=manu-12
>>>> Field Name & Value: sku=1
>>>> Field Name & Value: id=1
>>>> Field Name & Value: popularity=1
>>>> Field Name & Value: weight=12.0
>>>> SolrDocument(2)
>>>> Field Name & Value: includes=43
>>>> Field Name & Value: features=demo feature 2
>>>> Field Name & Value: price=10.0
>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>> Field Name & Value: manu=Demo - 12
>>>> Field Name & Value: sku=2
>>>> Field Name & Value: id=2
>>>> Field Name & Value: popularity=5
>>>> Field Name & Value: weight=12.0
>>>> SolrDocument(3)
>>>> Field Name & Value: includes=1
>>>> Field Name & Value: features=demo -3
>>>> Field Name & Value: price=1.0
>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>>> Field Name & Value: manu=manu - 36
>>>> Field Name & Value: sku=3
>>>> Field Name & Value: id=3
>>>> Field Name & Value: popularity=1
>>>> Field Name & Value: weight=1.0
>>>> SolrDocument(4)
>>>> Field Name & Value: includes=2
>>>> Field Name & Value: features=demo - 4
>>>> Field Name & Value: price=2.0
>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>> Field Name & Value: manu=manu - 46
>>>> Field Name & Value: sku=4
>>>> Field Name & Value: id=4
>>>> Field Name & Value: popularity=2
>>>> Field Name & Value: weight=2.0
>>>> SolrDocument(5)
>>>> Field Name & Value: includes=3
>>>> Field Name & Value: features=demo - 5
>>>> Field Name & Value: price=3.0
>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>> Field Name & Value: manu=manu - 56
>>>> Field Name & Value: sku=5
>>>> Field Name & Value: id=5
>>>> Field Name & Value: popularity=3
>>>> Field Name & Value: weight=3.0
>>>> SolrDocument(6)
>>>> Field Name & Value: includes=4
>>>> Field Name & Value: features=demo - 66
>>>> Field Name & Value: price=4.0
>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>>> Field Name & Value: manu=manu - 66
>>>> Field Name & Value: sku=6
>>>> Field Name & Value: id=6
>>>> Field Name & Value: popularity=4
>>>> Field Name & Value: weight=4.0
>>>>
>>>>
>>>> ------------------------------------------------------------------------------------------
>>>> My Java code that calls delta import is as follows -
>>>>
>>>> public static void main(String[] args) throws IOException,
>>>> SolrServerException {
>>>> CommonsHttpSolrServer server = new
>>>> CommonsHttpSolrServer("http://localhost:8983/solr");
>>>> BinaryResponseParser parser = new BinaryResponseParser();
>>>> server.setParser(parser);
>>>>
>>>> ModifiableSolrParams params = new ModifiableSolrParams();
>>>> params.set("qt", "/dataimport");
>>>> params.set("command", "delta-import");
>>>> params.set("commit", "true");
>>>> params.set("wt","json");
>>>>
>>>> try{
>>>>
>>>> QueryResponse response = server.query(params);
>>>> server.commit();
>>>> System.out.println("Server Response: " + response);
>>>>
>>>> SolrQuery query = new SolrQuery("*:*");
>>>> QueryResponse results = server.query(query);
>>>>
>>>> SolrDocumentList list = results.getResults();
>>>>
>>>> int index = 1;
>>>> for (SolrDocument solrDocument : list) {
>>>>
>>>> System.out.println("SolrDocument("+index+")");
>>>> Iterator<Entry<String, Object>> iterator =
>>>> solrDocument.iterator();
>>>>
>>>> while(iterator.hasNext()){
>>>> System.out.println("Field Name & Value:
>>>> "+iterator.next());
>>>> }
>>>>
>>>> index++;
>>>> }
>>>>
>>>> }catch(Exception ex){
>>>>
>>>> System.out.println("Exception Occured:"+ex);
>>>> }
>>>>
>>>> }
>>>> ---------------------------------------------------------------------------
>>>>
>>>> Does SOLR do some kind of caching? I dont understand as why the updated
>>>> values are not queried correctly from DB?
>>>>
>>>> There are no errors in the logs. But I can see an error on the server
>>>> console -
>>>>
>>>> SEVERE: Delta Import Failed
>>>> java.lang.NullPointerException
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>>> at
>>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>>> at
>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>>> at
>>>>
>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>>
>>>>
>>>> What can be the reason for this error?
>>>>
>>>> Thanks,
>>>> Manu
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>>
>>
>> --
>> View this message in context: http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
--
--Noble Paul
Re: Delta DataImport is not picking the modified value in DB
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
did you put in the data?
On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <ma...@gmail.com> wrote:
>
> Hi Shalin,
>
> I am actually trying out the example provided at
> http://wiki.apache.org/solr/DataImportHandler.
>
> My db-data-config.xml is as following -
>
> <dataConfig>
> <dataSource driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
> <document name="products">
> <entity name="item" pk="ID" query="select * from item"
> deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'">
> <field column="NAME" name="name" />
> <field column="NAME" name="nameSort" />
> <field column="NAME" name="alphaNameSort" />
>
> <entity name="feature" pk="ITEM_ID"
> query="select DESCRIPTION from FEATURE where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID from FEATURE where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${feature.ITEM_ID}">
> <field name="features" column="DESCRIPTION" />
> </entity>
>
> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
> query="select CATEGORY_ID from item_category where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${item_category.ITEM_ID}">
> <entity name="category" pk="ID"
> query="select DESCRIPTION from category where ID =
> '${item_category.CATEGORY_ID}'"
> deltaQuery="select ID from category where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where CATEGORY_ID=${category.ID}">
> <field column="description" name="cat" />
> </entity>
> </entity>
> </entity>
> </document>
> </dataConfig>
> ---------------------------------------------------------------------------------------------------
>
> My DB structure is as following -
>
> DROP TABLE IF EXISTS solr_demo_db.item;
> CREATE TABLE `item` (
> `ID` int(2) NOT NULL auto_increment,
> `name` varchar(100) default NULL,
> `manu` varchar(20) default NULL,
> `weight` varchar(20) default NULL,
> `price` varchar(20) default NULL,
> `popularity` varchar(2) default NULL,
> `includes` varchar(10) default NULL,
> `last_modified` datetime default NULL,
> PRIMARY KEY (`ID`)
> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>
> DROP TABLE IF EXISTS solr_demo_db.feature;
> CREATE TABLE `feature` (
> `DESCRIPTION` varchar(100) default NULL,
> `ITEM_ID` int(2) NOT NULL,
> `last_modified` datetime default NULL,
> PRIMARY KEY (`ITEM_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>
> DROP TABLE IF EXISTS solr_demo_db.category;
> CREATE TABLE `category` (
> `ID` int(2) NOT NULL auto_increment,
> `DESCRIPTION` varchar(100) default NULL,
> `last_modified` date default NULL,
> PRIMARY KEY (`ID`)
> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>
> DROP TABLE IF EXISTS solr_demo_db.item_category;
> CREATE TABLE `item_category` (
> `ITEM_ID` int(2) NOT NULL default '0',
> `CATEGORY_ID` int(2) NOT NULL default '0',
> `last_modified` date default NULL,
> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>
> Am I doing something wrong here?
>
> Thanks,
> Manu
>
>
> Shalin Shekhar Mangar wrote:
>>
>> What does your data-config look like? especially the delta query part.
>>
>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>> <ma...@gmail.com>wrote:
>>
>>>
>>> Hi,
>>>
>>> I am using Delta DataImport feature to partially refresh the indexes.
>>>
>>> Note - full-import is working perfectly fine. But I need to do
>>> delta-import
>>> as I do not want to rebuild all the indexes.
>>>
>>> My database structure is same as mentioned in the example at
>>> http://wiki.apache.org/solr/DataImportHandler.
>>>
>>> http://www.nabble.com/file/p21200498/example-schema.png
>>> ---------------------------------------------------------------------------
>>> The dataimport.properties has the value as -
>>>
>>> #Mon Dec 29 14:08:12 IST 2008
>>> last_index_time=2008-12-29 14\:08\:12
>>> ---------------------------------------------------------------------------
>>> I have changed the last_modified for the 'tem' table as current
>>> timestamp.
>>> And when I query the DB, I get the following result -
>>>
>>> http://www.nabble.com/file/p21200498/data.jpg
>>>
>>> -------------------------------------------------------------------------------------------------
>>>
>>> But when I call delta data-import, it returns me the following response -
>>>
>>> Server Response:
>>>
>>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>>> response format is experimental. It is likely to change in the future.}
>>>
>>>
>>> ---------------------------------------------------------------------------------------------------
>>> And when I query for data for '*:*', I get the following -
>>> Note - Data returned by query doesnt reflect the correct timestamp from
>>> database.
>>>
>>> SolrDocument(1)
>>> Field Name & Value: includes=12
>>> Field Name & Value: features=[demo feature, demo feature 1]
>>> Field Name & Value: price=12.0
>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>> Field Name & Value: manu=manu-12
>>> Field Name & Value: sku=1
>>> Field Name & Value: id=1
>>> Field Name & Value: popularity=1
>>> Field Name & Value: weight=12.0
>>> SolrDocument(2)
>>> Field Name & Value: includes=43
>>> Field Name & Value: features=demo feature 2
>>> Field Name & Value: price=10.0
>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>> Field Name & Value: manu=Demo - 12
>>> Field Name & Value: sku=2
>>> Field Name & Value: id=2
>>> Field Name & Value: popularity=5
>>> Field Name & Value: weight=12.0
>>> SolrDocument(3)
>>> Field Name & Value: includes=1
>>> Field Name & Value: features=demo -3
>>> Field Name & Value: price=1.0
>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>> Field Name & Value: manu=manu - 36
>>> Field Name & Value: sku=3
>>> Field Name & Value: id=3
>>> Field Name & Value: popularity=1
>>> Field Name & Value: weight=1.0
>>> SolrDocument(4)
>>> Field Name & Value: includes=2
>>> Field Name & Value: features=demo - 4
>>> Field Name & Value: price=2.0
>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>> Field Name & Value: manu=manu - 46
>>> Field Name & Value: sku=4
>>> Field Name & Value: id=4
>>> Field Name & Value: popularity=2
>>> Field Name & Value: weight=2.0
>>> SolrDocument(5)
>>> Field Name & Value: includes=3
>>> Field Name & Value: features=demo - 5
>>> Field Name & Value: price=3.0
>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>> Field Name & Value: manu=manu - 56
>>> Field Name & Value: sku=5
>>> Field Name & Value: id=5
>>> Field Name & Value: popularity=3
>>> Field Name & Value: weight=3.0
>>> SolrDocument(6)
>>> Field Name & Value: includes=4
>>> Field Name & Value: features=demo - 66
>>> Field Name & Value: price=4.0
>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>> Field Name & Value: manu=manu - 66
>>> Field Name & Value: sku=6
>>> Field Name & Value: id=6
>>> Field Name & Value: popularity=4
>>> Field Name & Value: weight=4.0
>>>
>>>
>>> ------------------------------------------------------------------------------------------
>>> My Java code that calls delta import is as follows -
>>>
>>> public static void main(String[] args) throws IOException,
>>> SolrServerException {
>>> CommonsHttpSolrServer server = new
>>> CommonsHttpSolrServer("http://localhost:8983/solr");
>>> BinaryResponseParser parser = new BinaryResponseParser();
>>> server.setParser(parser);
>>>
>>> ModifiableSolrParams params = new ModifiableSolrParams();
>>> params.set("qt", "/dataimport");
>>> params.set("command", "delta-import");
>>> params.set("commit", "true");
>>> params.set("wt","json");
>>>
>>> try{
>>>
>>> QueryResponse response = server.query(params);
>>> server.commit();
>>> System.out.println("Server Response: " + response);
>>>
>>> SolrQuery query = new SolrQuery("*:*");
>>> QueryResponse results = server.query(query);
>>>
>>> SolrDocumentList list = results.getResults();
>>>
>>> int index = 1;
>>> for (SolrDocument solrDocument : list) {
>>>
>>> System.out.println("SolrDocument("+index+")");
>>> Iterator<Entry<String, Object>> iterator =
>>> solrDocument.iterator();
>>>
>>> while(iterator.hasNext()){
>>> System.out.println("Field Name & Value:
>>> "+iterator.next());
>>> }
>>>
>>> index++;
>>> }
>>>
>>> }catch(Exception ex){
>>>
>>> System.out.println("Exception Occured:"+ex);
>>> }
>>>
>>> }
>>> ---------------------------------------------------------------------------
>>>
>>> Does SOLR do some kind of caching? I dont understand as why the updated
>>> values are not queried correctly from DB?
>>>
>>> There are no errors in the logs. But I can see an error on the server
>>> console -
>>>
>>> SEVERE: Delta Import Failed
>>> java.lang.NullPointerException
>>> at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>> at
>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>> at
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>
>>>
>>> What can be the reason for this error?
>>>
>>> Thanks,
>>> Manu
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Delta DataImport is not picking the modified value in DB
Posted by Manupriya <ma...@gmail.com>.
Hi Shalin,
I am actually trying out the example provided at
http://wiki.apache.org/solr/DataImportHandler.
My db-data-config.xml is as following -
<dataConfig>
<dataSource driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
<document name="products">
<entity name="item" pk="ID" query="select * from item"
deltaQuery="select id from item where last_modified >
'${dataimporter.last_index_time}'">
<field column="NAME" name="name" />
<field column="NAME" name="nameSort" />
<field column="NAME" name="alphaNameSort" />
<entity name="feature" pk="ITEM_ID"
query="select DESCRIPTION from FEATURE where
ITEM_ID='${item.ID}'"
deltaQuery="select ITEM_ID from FEATURE where
last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ID from item where
ID=${feature.ITEM_ID}">
<field name="features" column="DESCRIPTION" />
</entity>
<entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
query="select CATEGORY_ID from item_category where
ITEM_ID='${item.ID}'"
deltaQuery="select ITEM_ID, CATEGORY_ID from
item_category where last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ID from item where
ID=${item_category.ITEM_ID}">
<entity name="category" pk="ID"
query="select DESCRIPTION from category where ID =
'${item_category.CATEGORY_ID}'"
deltaQuery="select ID from category where
last_modified > '${dataimporter.last_index_time}'"
parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
item_category where CATEGORY_ID=${category.ID}">
<field column="description" name="cat" />
</entity>
</entity>
</entity>
</document>
</dataConfig>
---------------------------------------------------------------------------------------------------
My DB structure is as following -
DROP TABLE IF EXISTS solr_demo_db.item;
CREATE TABLE `item` (
`ID` int(2) NOT NULL auto_increment,
`name` varchar(100) default NULL,
`manu` varchar(20) default NULL,
`weight` varchar(20) default NULL,
`price` varchar(20) default NULL,
`popularity` varchar(2) default NULL,
`includes` varchar(10) default NULL,
`last_modified` datetime default NULL,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
DROP TABLE IF EXISTS solr_demo_db.feature;
CREATE TABLE `feature` (
`DESCRIPTION` varchar(100) default NULL,
`ITEM_ID` int(2) NOT NULL,
`last_modified` datetime default NULL,
PRIMARY KEY (`ITEM_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
DROP TABLE IF EXISTS solr_demo_db.category;
CREATE TABLE `category` (
`ID` int(2) NOT NULL auto_increment,
`DESCRIPTION` varchar(100) default NULL,
`last_modified` date default NULL,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
DROP TABLE IF EXISTS solr_demo_db.item_category;
CREATE TABLE `item_category` (
`ITEM_ID` int(2) NOT NULL default '0',
`CATEGORY_ID` int(2) NOT NULL default '0',
`last_modified` date default NULL,
PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Am I doing something wrong here?
Thanks,
Manu
Shalin Shekhar Mangar wrote:
>
> What does your data-config look like? especially the delta query part.
>
> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
> <ma...@gmail.com>wrote:
>
>>
>> Hi,
>>
>> I am using Delta DataImport feature to partially refresh the indexes.
>>
>> Note - full-import is working perfectly fine. But I need to do
>> delta-import
>> as I do not want to rebuild all the indexes.
>>
>> My database structure is same as mentioned in the example at
>> http://wiki.apache.org/solr/DataImportHandler.
>>
>> http://www.nabble.com/file/p21200498/example-schema.png
>> ---------------------------------------------------------------------------
>> The dataimport.properties has the value as -
>>
>> #Mon Dec 29 14:08:12 IST 2008
>> last_index_time=2008-12-29 14\:08\:12
>> ---------------------------------------------------------------------------
>> I have changed the last_modified for the 'tem' table as current
>> timestamp.
>> And when I query the DB, I get the following result -
>>
>> http://www.nabble.com/file/p21200498/data.jpg
>>
>> -------------------------------------------------------------------------------------------------
>>
>> But when I call delta data-import, it returns me the following response -
>>
>> Server Response:
>>
>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>> response format is experimental. It is likely to change in the future.}
>>
>>
>> ---------------------------------------------------------------------------------------------------
>> And when I query for data for '*:*', I get the following -
>> Note - Data returned by query doesnt reflect the correct timestamp from
>> database.
>>
>> SolrDocument(1)
>> Field Name & Value: includes=12
>> Field Name & Value: features=[demo feature, demo feature 1]
>> Field Name & Value: price=12.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=manu-12
>> Field Name & Value: sku=1
>> Field Name & Value: id=1
>> Field Name & Value: popularity=1
>> Field Name & Value: weight=12.0
>> SolrDocument(2)
>> Field Name & Value: includes=43
>> Field Name & Value: features=demo feature 2
>> Field Name & Value: price=10.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=Demo - 12
>> Field Name & Value: sku=2
>> Field Name & Value: id=2
>> Field Name & Value: popularity=5
>> Field Name & Value: weight=12.0
>> SolrDocument(3)
>> Field Name & Value: includes=1
>> Field Name & Value: features=demo -3
>> Field Name & Value: price=1.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>> Field Name & Value: manu=manu - 36
>> Field Name & Value: sku=3
>> Field Name & Value: id=3
>> Field Name & Value: popularity=1
>> Field Name & Value: weight=1.0
>> SolrDocument(4)
>> Field Name & Value: includes=2
>> Field Name & Value: features=demo - 4
>> Field Name & Value: price=2.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 46
>> Field Name & Value: sku=4
>> Field Name & Value: id=4
>> Field Name & Value: popularity=2
>> Field Name & Value: weight=2.0
>> SolrDocument(5)
>> Field Name & Value: includes=3
>> Field Name & Value: features=demo - 5
>> Field Name & Value: price=3.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 56
>> Field Name & Value: sku=5
>> Field Name & Value: id=5
>> Field Name & Value: popularity=3
>> Field Name & Value: weight=3.0
>> SolrDocument(6)
>> Field Name & Value: includes=4
>> Field Name & Value: features=demo - 66
>> Field Name & Value: price=4.0
>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>> Field Name & Value: manu=manu - 66
>> Field Name & Value: sku=6
>> Field Name & Value: id=6
>> Field Name & Value: popularity=4
>> Field Name & Value: weight=4.0
>>
>>
>> ------------------------------------------------------------------------------------------
>> My Java code that calls delta import is as follows -
>>
>> public static void main(String[] args) throws IOException,
>> SolrServerException {
>> CommonsHttpSolrServer server = new
>> CommonsHttpSolrServer("http://localhost:8983/solr");
>> BinaryResponseParser parser = new BinaryResponseParser();
>> server.setParser(parser);
>>
>> ModifiableSolrParams params = new ModifiableSolrParams();
>> params.set("qt", "/dataimport");
>> params.set("command", "delta-import");
>> params.set("commit", "true");
>> params.set("wt","json");
>>
>> try{
>>
>> QueryResponse response = server.query(params);
>> server.commit();
>> System.out.println("Server Response: " + response);
>>
>> SolrQuery query = new SolrQuery("*:*");
>> QueryResponse results = server.query(query);
>>
>> SolrDocumentList list = results.getResults();
>>
>> int index = 1;
>> for (SolrDocument solrDocument : list) {
>>
>> System.out.println("SolrDocument("+index+")");
>> Iterator<Entry<String, Object>> iterator =
>> solrDocument.iterator();
>>
>> while(iterator.hasNext()){
>> System.out.println("Field Name & Value:
>> "+iterator.next());
>> }
>>
>> index++;
>> }
>>
>> }catch(Exception ex){
>>
>> System.out.println("Exception Occured:"+ex);
>> }
>>
>> }
>> ---------------------------------------------------------------------------
>>
>> Does SOLR do some kind of caching? I dont understand as why the updated
>> values are not queried correctly from DB?
>>
>> There are no errors in the logs. But I can see an error on the server
>> console -
>>
>> SEVERE: Delta Import Failed
>> java.lang.NullPointerException
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>> at
>>
>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>> at
>>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>> at
>>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>
>>
>> What can be the reason for this error?
>>
>> Thanks,
>> Manu
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Delta DataImport is not picking the modified value in DB
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
What does your data-config look like? especially the delta query part.
On Mon, Dec 29, 2008 at 2:35 PM, Manupriya <ma...@gmail.com>wrote:
>
> Hi,
>
> I am using Delta DataImport feature to partially refresh the indexes.
>
> Note - full-import is working perfectly fine. But I need to do delta-import
> as I do not want to rebuild all the indexes.
>
> My database structure is same as mentioned in the example at
> http://wiki.apache.org/solr/DataImportHandler.
>
> http://www.nabble.com/file/p21200498/example-schema.png
> ---------------------------------------------------------------------------
> The dataimport.properties has the value as -
>
> #Mon Dec 29 14:08:12 IST 2008
> last_index_time=2008-12-29 14\:08\:12
> ---------------------------------------------------------------------------
> I have changed the last_modified for the 'tem' table as current timestamp.
> And when I query the DB, I get the following result -
>
> http://www.nabble.com/file/p21200498/data.jpg
>
> -------------------------------------------------------------------------------------------------
>
> But when I call delta data-import, it returns me the following response -
>
> Server Response:
>
> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
> response format is experimental. It is likely to change in the future.}
>
>
> ---------------------------------------------------------------------------------------------------
> And when I query for data for '*:*', I get the following -
> Note - Data returned by query doesnt reflect the correct timestamp from
> database.
>
> SolrDocument(1)
> Field Name & Value: includes=12
> Field Name & Value: features=[demo feature, demo feature 1]
> Field Name & Value: price=12.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> Field Name & Value: manu=manu-12
> Field Name & Value: sku=1
> Field Name & Value: id=1
> Field Name & Value: popularity=1
> Field Name & Value: weight=12.0
> SolrDocument(2)
> Field Name & Value: includes=43
> Field Name & Value: features=demo feature 2
> Field Name & Value: price=10.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> Field Name & Value: manu=Demo - 12
> Field Name & Value: sku=2
> Field Name & Value: id=2
> Field Name & Value: popularity=5
> Field Name & Value: weight=12.0
> SolrDocument(3)
> Field Name & Value: includes=1
> Field Name & Value: features=demo -3
> Field Name & Value: price=1.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> Field Name & Value: manu=manu - 36
> Field Name & Value: sku=3
> Field Name & Value: id=3
> Field Name & Value: popularity=1
> Field Name & Value: weight=1.0
> SolrDocument(4)
> Field Name & Value: includes=2
> Field Name & Value: features=demo - 4
> Field Name & Value: price=2.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> Field Name & Value: manu=manu - 46
> Field Name & Value: sku=4
> Field Name & Value: id=4
> Field Name & Value: popularity=2
> Field Name & Value: weight=2.0
> SolrDocument(5)
> Field Name & Value: includes=3
> Field Name & Value: features=demo - 5
> Field Name & Value: price=3.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> Field Name & Value: manu=manu - 56
> Field Name & Value: sku=5
> Field Name & Value: id=5
> Field Name & Value: popularity=3
> Field Name & Value: weight=3.0
> SolrDocument(6)
> Field Name & Value: includes=4
> Field Name & Value: features=demo - 66
> Field Name & Value: price=4.0
> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> Field Name & Value: manu=manu - 66
> Field Name & Value: sku=6
> Field Name & Value: id=6
> Field Name & Value: popularity=4
> Field Name & Value: weight=4.0
>
>
> ------------------------------------------------------------------------------------------
> My Java code that calls delta import is as follows -
>
> public static void main(String[] args) throws IOException,
> SolrServerException {
> CommonsHttpSolrServer server = new
> CommonsHttpSolrServer("http://localhost:8983/solr");
> BinaryResponseParser parser = new BinaryResponseParser();
> server.setParser(parser);
>
> ModifiableSolrParams params = new ModifiableSolrParams();
> params.set("qt", "/dataimport");
> params.set("command", "delta-import");
> params.set("commit", "true");
> params.set("wt","json");
>
> try{
>
> QueryResponse response = server.query(params);
> server.commit();
> System.out.println("Server Response: " + response);
>
> SolrQuery query = new SolrQuery("*:*");
> QueryResponse results = server.query(query);
>
> SolrDocumentList list = results.getResults();
>
> int index = 1;
> for (SolrDocument solrDocument : list) {
>
> System.out.println("SolrDocument("+index+")");
> Iterator<Entry<String, Object>> iterator = solrDocument.iterator();
>
> while(iterator.hasNext()){
> System.out.println("Field Name & Value: "+iterator.next());
> }
>
> index++;
> }
>
> }catch(Exception ex){
>
> System.out.println("Exception Occured:"+ex);
> }
>
> }
> ---------------------------------------------------------------------------
>
> Does SOLR do some kind of caching? I dont understand as why the updated
> values are not queried correctly from DB?
>
> There are no errors in the logs. But I can see an error on the server
> console -
>
> SEVERE: Delta Import Failed
> java.lang.NullPointerException
> at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
> at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
> at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
> at
>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
> at
>
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
> at
>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
> at
>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>
>
> What can be the reason for this error?
>
> Thanks,
> Manu
>
> --
> View this message in context:
> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.