You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Grant Ingersoll <gs...@apache.org> on 2008/10/03 17:50:44 UTC
[DataImportHandler] Changes in child entity and delta-import
First off, starting to play with the DataImportHandler (DIH) more...
very cool.
I have a rather simple case where I am indexing an RSS feed that
contains articles. For one or more articles, I have an entry in a
database that contains the URL of the article and a rating.
My config is appended below in [1]. The DB looks like [2]. The RSS
feed comes from my blog (heh, it's convenient and I control it)
Now, my question. Let's say I have an initial set of ratings for a
feed. I then do a full import of the articles on that feed.
Everything is peachy so far. Then, I get a new rating for an existing
article that I've already indexed, thus the child entity (named
"rating")
has a delta. However, when I run the delta-import, it doesn't pick
up any changes, since, I believe, the parent hasn't changed. Either
that, or I am doing something wrong. It seems like it is akin to the
parentDeltaQuery problem, but, of course, there is no parent query
since there is no parent table, in the DB sense, at least
not how I see it. The relevant logs are in [3].
Is this case handled? If not, Any suggestions for alternatives? Any
help would be appreciated.
Thanks,
Grant
[1]
<dataConfig>
<dataSource name="ratings" driver="org.postgresql.Driver"
url="jdbc:postgresql://localhost:5432/db" user="user" />
<dataSource name="rss" type="HttpDataSource" encoding="UTF-8"/>
<document>
<entity name="solrFeed"
pk="link"
url="http://lucene.grantingersoll.com/category/solr/feed"
processor="XPathEntityProcessor"
forEach="/rss/channel | /rss/channel/item"
dataSource="rss"
transformer="DateFormatTransformer">
<field column="source" xpath="/rss/channel/title"
commonField="true" />
<field column="source-link" xpath="/rss/channel/link"
commonField="true" />
<field column="title" xpath="/rss/channel/item/title" />
<field column="link" xpath="/rss/channel/item/link" />
<field column="description" xpath="/rss/channel/item/description" />
<field column="category" xpath="/rss/channel/item/category" />
<field column="content" xpath="/rss/channel/item/content" />
<!-- 'Sun, 18 May 2008 11:23:11 +0000' -->
<field column="date" xpath="/rss/channel/item/pubDate"
dateTimeFormat="EEE, dd MMM yyyy HH:mm:ss Z" />
<entity name="rating" pk="feed" query="select rating from feeds
where feed = '${solrFeed.link}'"
deltaQuery="select rating from feeds where feed = '$
{solrFeed.link}' AND last_modified > '${dataimporter.last_index_time}'"
dataSource="ratings"
>
<field column="rating" name="rating"/>
</entity>
</entity>
</document>
</dataConfig>
[2]
\d feeds
Table "public.feeds"
Column | Type | Modifiers
---------------+-----------------------------+-----------
feed | character varying(4096) | not null
rating | double precision |
last_modified | timestamp without time zone |
Indexes:
"feeds_pkey" PRIMARY KEY, btree (feed)
[3]
Oct 3, 2008 11:39:09 AM
org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO: Time taken for getConnection(): 8
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed ModifiedRowKey for Entity: rating rows obtained : 0
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Running DeletedRowKey() for Entity: rating
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed DeletedRowKey for Entity: rating rows obtained : 0
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed parentDeltaQuery for Entity: rating
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Running ModifiedRowKey() for Entity: solrFeed
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed ModifiedRowKey for Entity: solrFeed rows obtained : 0
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Running DeletedRowKey() for Entity: solrFeed
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed DeletedRowKey for Entity: solrFeed rows obtained : 0
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed parentDeltaQuery for Entity: solrFeed
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
INFO: Read dataimport.properties
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.SolrWriter
persistStartTime
INFO: Wrote last indexed time to dataimport.properties
Oct 3, 2008 11:39:09 AM org.apache.solr.handler.dataimport.DocBuilder
doDelta
INFO: Delta Import completed successfully
Re: [DataImportHandler] Changes in child entity and delta-import
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 3, 2008, at 2:11 PM, Shalin Shekhar Mangar wrote:
>
>> Essentially, what I am interested in is a join across data
>> sources. I
>> realize that is non-trivial, but boy would it be powerful.
>
>
> Yeah, that will be very powerful but implementing that in a general
> way is a
> design challenge.
Agreed, at least in the generic case.
>>
>
> Ah yes, I missed that. The delta import works by traversing the
> entity tree
> bottom-up. So the delta query for the child most entity is evaluated
> first,
> then the parentDeltaQuery and then we go up to the parent entity.
> Therefore,
> parent variables will not work in a deltaQuery.
So, there probably is a way to hook it in for this case in a non-
generic way. I'll think some more on it.
Re: [DataImportHandler] Changes in child entity and delta-import
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Fri, Oct 3, 2008 at 10:44 PM, Grant Ingersoll <gs...@apache.org>wrote:
>
> What about somehow using the fact that the variable resolver needs to
> resolve solrFeed.link and then go get all entries from Solr to get those
> values, such that the child entity can then be tested?
>
That may be possible. Here is another idea that might work.
Keep the current RSS root entity and DB child relationship and use it for
full-imports. Create another set of entities with DB root entity and RSS
child relationships and use for delta-imports. You can provide an entity
request parameter for an import command. This way both full and delta will
work and you will get all the data.
> Essentially, what I am interested in is a join across data sources. I
> realize that is non-trivial, but boy would it be powerful.
Yeah, that will be very powerful but implementing that in a general way is a
design challenge.
> Yeah, that bothers me, too. My dataimport.properties contains:
> grantingersoll@molly[1041]$ cat dataimport.properties
> #Fri Oct 03 12:09:29 EDT 2008
> last_index_time=2008-10-03 12\:09\:28
>
> And, when querying by hand my DB shows:
>
> select * from feeds where last_modified > '10/03/2008';
>
>
> feed
> | rating | last_modified
>
> ---------------------------------------------------------------------------+--------+---------------------
> http://lucene.grantingersoll.com/2008/06/21/solr-spell-checking-addition/ |
> 4.9 | 2008-10-04 11:04:00
> (1 row)
>
> So, I am reasonably certain there is a change.
>
> I think the reason is, if you notice further down in the log, is that it
> processes the entities separately. In other words, is the DB entity even
> getting resolved in the context of the parent entity? Or, is it not
> resolving the ${solrFeed.link} clause of the delta query?
>
Ah yes, I missed that. The delta import works by traversing the entity tree
bottom-up. So the delta query for the child most entity is evaluated first,
then the parentDeltaQuery and then we go up to the parent entity. Therefore,
parent variables will not work in a deltaQuery.
--
Regards,
Shalin Shekhar Mangar.
Re: [DataImportHandler] Changes in child entity and delta-import
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 3, 2008, at 12:34 PM, Shalin Shekhar Mangar wrote:
> On Fri, Oct 3, 2008 at 9:20 PM, Grant Ingersoll
> <gs...@apache.org> wrote:
>
>>
>> Now, my question. Let's say I have an initial set of ratings for a
>> feed.
>> I then do a full import of the articles on that feed. Everything
>> is peachy
>> so far. Then, I get a new rating for an existing article that I've
>> already
>> indexed, thus the child entity (named "rating")
>> has a delta. However, when I run the delta-import, it doesn't
>> pick up
>> any changes, since, I believe, the parent hasn't changed. Either
>> that, or I
>> am doing something wrong. It seems like it is akin to the
>> parentDeltaQuery
>> problem, but, of course, there is no parent query since there is no
>> parent
>> table, in the DB sense, at least
>> not how I see it. The relevant logs are in [3].
>>
>> Is this case handled? If not, Any suggestions for alternatives?
>> Any help
>> would be appreciated.
>
>
> XPathEntityProcessor does not support delta imports. It might be
> possible to
> enhance it to accept an xpath condition for joining child to parent
> but it
> seems point-less because we'd need to parse the whole XML anyway for
> each
> changed child row (Joel Spolsky's words echo in my mind!). If the
> XML data
> is small, we can also have a cached implementation like the
> CachedSqlEntityProcessor.
What about somehow using the fact that the variable resolver needs to
resolve solrFeed.link and then go get all entries from Solr to get
those values, such that the child entity can then be tested?
>
>
> The easiest workaround here is to reverse the parent-child. Make the
> DB as
> the parent and join on the child which will let you do delta imports,
> however full imports may be expensive. Depending on the size of XML,
> you may
> be better off doing a full import always.
I thought of reversing the parent-child, but I don't see how it works,
since there isn't necessarily a DB entry for every article. How would
you associate the two to make sure you get all articles?
Also, the current approach seems more intuitive, since the RSS feed is
the authoritative content.
Essentially, what I am interested in is a join across data sources. I
realize that is non-trivial, but boy would it be powerful.
>
>
> Another thing I noticed from your logs: the ModifiedRowKey count is
> 0. Are
> you sure the timestamp column is getting updated? IIRC, you need a
> stored
> proc to do this for postgres.
>
> INFO: Completed ModifiedRowKey for Entity: rating rows obtained : 0
Yeah, that bothers me, too. My dataimport.properties contains:
grantingersoll@molly[1041]$ cat dataimport.properties
#Fri Oct 03 12:09:29 EDT 2008
last_index_time=2008-10-03 12\:09\:28
And, when querying by hand my DB shows:
select * from feeds where last_modified >
'10
/
03
/
2008
'; feed
| rating | last_modified
---------------------------------------------------------------------------+
--------+---------------------
http://lucene.grantingersoll.com/2008/06/21/solr-spell-checking-addition/
| 4.9 | 2008-10-04 11:04:00
(1 row)
So, I am reasonably certain there is a change.
I think the reason is, if you notice further down in the log, is that
it processes the entities separately. In other words, is the DB
entity even getting resolved in the context of the parent entity? Or,
is it not resolving the ${solrFeed.link} clause of the delta query?
-Grant
Re: [DataImportHandler] Changes in child entity and delta-import
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Fri, Oct 3, 2008 at 9:20 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> Now, my question. Let's say I have an initial set of ratings for a feed.
> I then do a full import of the articles on that feed. Everything is peachy
> so far. Then, I get a new rating for an existing article that I've already
> indexed, thus the child entity (named "rating")
> has a delta. However, when I run the delta-import, it doesn't pick up
> any changes, since, I believe, the parent hasn't changed. Either that, or I
> am doing something wrong. It seems like it is akin to the parentDeltaQuery
> problem, but, of course, there is no parent query since there is no parent
> table, in the DB sense, at least
> not how I see it. The relevant logs are in [3].
>
> Is this case handled? If not, Any suggestions for alternatives? Any help
> would be appreciated.
XPathEntityProcessor does not support delta imports. It might be possible to
enhance it to accept an xpath condition for joining child to parent but it
seems point-less because we'd need to parse the whole XML anyway for each
changed child row (Joel Spolsky's words echo in my mind!). If the XML data
is small, we can also have a cached implementation like the
CachedSqlEntityProcessor.
The easiest workaround here is to reverse the parent-child. Make the DB as
the parent and join on the child which will let you do delta imports,
however full imports may be expensive. Depending on the size of XML, you may
be better off doing a full import always.
Another thing I noticed from your logs: the ModifiedRowKey count is 0. Are
you sure the timestamp column is getting updated? IIRC, you need a stored
proc to do this for postgres.
INFO: Completed ModifiedRowKey for Entity: rating rows obtained : 0
--
Regards,
Shalin Shekhar Mangar.