Re: LARQ needs to update the Lucene index when a SPARQL Update request is received

[陶信东 <ta...@myhexin.com>]

Hi Paolo, 

 

Thanks for your prompt response. Our RDF data are generated from MYSQL (and
some other resources) and written to TDB. I used Joseki to expose the RDF
data and joseki:initialization to build the LARQ index. Then the user can do
some fuzzy search for the RDF data on our UI.

 

At the beginning we let our operating team make changes to MYSQL and then
re-generate the RDF data every day (using the TDB java API, not the
tdbloader). The updates are not very often and usually very small. Recently
we want to let the users see these changes immediately after they occurs. So
we used SPARQL update service to update our TDB after the data is updated on
MYSQL. However, as the LARQ index is not updated, the users still cannot see
the change immediately.

 

Maybe I should add some UNION sections (using exact match instead of fuzzy
match) to our SPARQL query so that the newly added item can be found (when
fully matched) without the Lucene update. But it would be great help if the
update features can be implemented for LARQ/Joseki/Fuseki.

 

As you preferred, I copied our comments below so that this discussion happen
on jena-users.

 

Thanks

Tao

 

----------------------------------------------------------------------------
-----------

 <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=castagna>
Paolo Castagna added a comment - 12/Mar/12 08:24

Hi Tao, could you share a few more detail on your use cases and application
scenario? I imagine you are using Fuseki (with TDB) and LARQ for free text
searches. Do you submit all your updates via SPARQL Update or there are
other ways your updates come in? Do you use tdbloader for large/incremental
updates? How frequent and how big are your updates? Would re-building the
Lucene index nightly or hourly be a feasible option for you? 
In any case, thanks for posting your comment here. You can either answer my
questions here or on jena-users/jena-dev mailing lists. I prefer this
discussions to happen on jena-users and leave JIRA comments to specific
comments on how to implement/fix the issue. But comments in JIRA are welcome
anyway and they are better than nothing.

 

-----------------------------------------------------------------------

 <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=laotao> laotao
added a comment - 12/Mar/12 07:27

Hi Paolo/Andy, 

I was searching for a solution regarding this. It is a very important
feature for our project. Would you please inform me when this can be
implemented? Or is there any work around that I can try? 

Thanks 
Tao

RE: LARQ needs to update the Lucene index when a SPARQL Update request is received

["Tao (陶信东)" <ta...@myhexin.com>]

Hi Paolo,

Thanks for your help. The reason why I decided to use SPARQL update (and
Joseki) is that our client app is not on the TDB server. And I also noticed
that Joseki cannot be updated automatically when the underling TDB is
changed. I have to restart Joseki to see the changes.

Maybe I should deploy our app and TDB to a same server and use the Jena/TDB
API without Joseki/Fuseki services?

Thanks
Tao

-----Original Message-----
From: Paolo Castagna [mailto:castagna.lists@googlemail.com] 
Sent: Monday, March 12, 2012 7:00 PM
To: jena-users@incubator.apache.org
Cc: 陶信东
Subject: Re: LARQ needs to update the Lucene index when a SPARQL Update
request is received

Hi Tao,
thanks for your reply, my comments (and personal opinions) are in-line.

陶信东 wrote:
> Hi Paolo,
> 
> Thanks for your prompt response. Our RDF data are generated from MYSQL 
> (and some other resources) and written to TDB. I used Joseki to expose 
> the RDF data and joseki:initialization to build the LARQ index. Then 
> the user can do some fuzzy search for the RDF data on our UI.

I suggest you look at Fuseki (which replaces Joseki and it is were the
majority of development effort is probably going to focus).

> At the beginning we let our operating team make changes to MYSQL and 
> then re-generate the RDF data every day (using the TDB java API, not 
> the tdbloader). The updates are not very often and usually very small.
> Recently we want to let the users see these changes immediately after 
> they occurs. So we used SPARQL update service to update our TDB after 
> the data is updated on MYSQL. However, as the LARQ index is not 
> updated, the users still cannot see the change immediately.

What's the reason why you decided to use SPARQL Update instead of Jena APIs?
In your case you have full control on this and you could use the Jena APIs
for your updates, in particular if they are small.
I am asking this, because if you do that you'll have solved your problem of
updating the Lucene index. LARQ does this when you use the Jena APIs.

> Maybe I should add some UNION sections (using exact match instead of 
> fuzzy match) to our SPARQL query so that the newly added item can be 
> found (when fully matched) without the Lucene update. But it would be 
> great help if the update features can be implemented for
LARQ/Joseki/Fuseki.

Here, I cannot possibly comment since I do not know your data (the shape and
how big/small it is) and I do not know your queries (therefore I am not able
to see if they would run sufficiently quickly or not to drive a UI).

> As you preferred, I copied our comments below so that this discussion 
> happen on jena-users.
> 

Thanks a lot for sharing your scenario and user needs.

Paolo

> 
> Thanks
> 
> Tao
> 
>  
> 
> ----------------------------------------------------------------------
> -----------------
> 
> Paolo Castagna
> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=castagna> 
> added a comment - 12/Mar/12 08:24
> 
> Hi Tao, could you share a few more detail on your use cases and 
> application scenario? I imagine you are using Fuseki (with TDB) and 
> LARQ for free text searches. Do you submit all your updates via SPARQL 
> Update or there are other ways your updates come in? Do you use 
> tdbloader for large/incremental updates? How frequent and how big are your
updates?
> Would re-building the Lucene index nightly or hourly be a feasible 
> option for you?
> In any case, thanks for posting your comment here. You can either 
> answer my questions here or on jena-users/jena-dev mailing lists. I 
> prefer this discussions to happen on jena-users and leave JIRA 
> comments to specific comments on how to implement/fix the issue. But 
> comments in JIRA are welcome anyway and they are better than nothing.
> 
>  
> 
> ----------------------------------------------------------------------
> -
> 
> laotao
> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=laotao> 
> added a comment - 12/Mar/12 07:27
> 
> Hi Paolo/Andy,
> 
> I was searching for a solution regarding this. It is a very important 
> feature for our project. Would you please inform me when this can be 
> implemented? Or is there any work around that I can try?
> 
> Thanks
> Tao
> 

Re: LARQ needs to update the Lucene index when a SPARQL Update request is received

[Paolo Castagna <ca...@googlemail.com>]

Hi Tao,
thanks for your reply, my comments (and personal opinions) are in-line.

陶信东 wrote:
> Hi Paolo,
> 
> Thanks for your prompt response. Our RDF data are generated from MYSQL
> (and some other resources) and written to TDB. I used Joseki to expose
> the RDF data and joseki:initialization to build the LARQ index. Then the
> user can do some fuzzy search for the RDF data on our UI.

I suggest you look at Fuseki (which replaces Joseki and it is were the majority of development effort is probably going to focus).

> At the beginning we let our operating team make changes to MYSQL and
> then re-generate the RDF data every day (using the TDB java API, not the
> tdbloader). The updates are not very often and usually very small.
> Recently we want to let the users see these changes immediately after
> they occurs. So we used SPARQL update service to update our TDB after
> the data is updated on MYSQL. However, as the LARQ index is not updated,
> the users still cannot see the change immediately.

What's the reason why you decided to use SPARQL Update instead of Jena APIs?
In your case you have full control on this and you could use the Jena APIs for your updates, in particular if they are small.
I am asking this, because if you do that you'll have solved your problem of updating the Lucene index. LARQ does this when you use the Jena APIs.

> Maybe I should add some UNION sections (using exact match instead of
> fuzzy match) to our SPARQL query so that the newly added item can be
> found (when fully matched) without the Lucene update. But it would be
> great help if the update features can be implemented for LARQ/Joseki/Fuseki.

Here, I cannot possibly comment since I do not know your data (the shape and how big/small it is) and I do not know your queries (therefore I am not able to see if they would run sufficiently quickly
or not to drive a UI).

> As you preferred, I copied our comments below so that this discussion
> happen on jena-users.
> 

Thanks a lot for sharing your scenario and user needs.

Paolo

> 
> Thanks
> 
> Tao
> 
>  
> 
> ---------------------------------------------------------------------------------------
> 
> Paolo Castagna
> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=castagna> added
> a comment - 12/Mar/12 08:24
> 
> Hi Tao, could you share a few more detail on your use cases and
> application scenario? I imagine you are using Fuseki (with TDB) and LARQ
> for free text searches. Do you submit all your updates via SPARQL Update
> or there are other ways your updates come in? Do you use tdbloader for
> large/incremental updates? How frequent and how big are your updates?
> Would re-building the Lucene index nightly or hourly be a feasible
> option for you? 
> In any case, thanks for posting your comment here. You can either answer
> my questions here or on jena-users/jena-dev mailing lists. I prefer this
> discussions to happen on jena-users and leave JIRA comments to specific
> comments on how to implement/fix the issue. But comments in JIRA are
> welcome anyway and they are better than nothing.
> 
>  
> 
> -----------------------------------------------------------------------
> 
> laotao
> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=laotao> added
> a comment - 12/Mar/12 07:27
> 
> Hi Paolo/Andy, 
> 
> I was searching for a solution regarding this. It is a very important
> feature for our project. Would you please inform me when this can be
> implemented? Or is there any work around that I can try? 
> 
> Thanks 
> Tao
>