You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Bart van Leeuwen <ba...@netage.nl> on 2014/11/18 16:21:02 UTC
Fuseki + TDB + Lucene online text index update
Hi,
I've played around with Fuseki 1.1.1
I created a TDB store which I filled with data through the Fuseki control
panel
I followed the instruction for full text queries at:
http://jena.apache.org/documentation/query/text-query.html
Everything is up and running and works like expected.
Now the data in the TDB store will be updated now and then is it possible
to run the index updater while fuseki is still up and running ?
or do I need to bring fuseki down during that process ? ( I know TDB is
not thread safe but afaik the indexer is not touching TDB for writing )
What would be the best practice for online index updating ?
Met Vriendelijke Groet / With Kind Regards
Bart van Leeuwen
##############################################################
# twitter: @semanticfire
# netage.nl
# http://netage.nl
# Enschedepad 76
# 1324 GJ Almere
# The Netherlands
# tel. +31(0)36-5347479
##############################################################
Re: Fuseki + TDB + Lucene online text index update
Posted by Andy Seaborne <an...@apache.org>.
Incorporated
Thanks - Andy
On 26/11/14 16:37, Bart van Leeuwen wrote:
> done
>
> Andy Seaborne <an...@apache.org> wrote on 21-11-2014 12:27:37:
>
>> Please suggest improvements, and contribute new text. It's always trick
>
>> to write documentation when you are on the "inside" of the
>> implementation :-)
>>
>> (You can edit them online with Apache CMS and the project will receive a
>
>> patch for the website).
>>
>>>
>>> As long as I use Fuseki to update , remove and insert data the index
> stays
>>> in sync, thats all I needed.
>>
>> Yes - that's the way to do it. Lucene is runnig same_JVM so you can
>> only mess it up by updating directly while Fuseki is running.
>>
>> It is possible to have more complex setups with Solr as the text index,
>> then update solr or query solr separately from Fuseki/TDB. Solr is an
>> external service in this example.
>>
>> Andy+## Updating the index
+
+If you allow updates to the dataset through Fuseki, the configured
index will
+automatically be updated on every modification.
+This means that you do not have to run the above mentioned
jena.textindexer after updates.
+
>>
>>>
>>> Bart
>>>
>>>> On 18/11/2014 16:21, "Bart van Leeuwen" <ba...@netage.nl>
>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I've played around with Fuseki 1.1.1
>>>>> I created a TDB store which I filled with data through the Fuseki
>>> control
>>>>> panel
>>>>>
>>>>> I followed the instruction for full text queries at:
>>>>> http://jena.apache.org/documentation/query/text-query.html
>>>>>
>>>>> Everything is up and running and works like expected.
>>>>>
>>>>> Now the data in the TDB store will be updated now and then is it
>>> possible
>>>>> to run the index updater while fuseki is still up and running ?
>>>>> or do I need to bring fuseki down during that process ? ( I know TDB
> is
>>>>> not thread safe but afaik the indexer is not touching TDB for
> writing )
>>>>>
>>>>> What would be the best practice for online index updating ?
>>>>>
>>>>> Met Vriendelijke Groet / With Kind Regards
>>>>> Bart van Leeuwen
>>>>>
>>>>> ##############################################################
>>>>> # twitter: @semanticfire
>>>>> # netage.nl
>>>>> # http://netage.nl
>>>>> # Enschedepad 76
>>>>> # 1324 GJ Almere
>>>>> # The Netherlands
>>>>> # tel. +31(0)36-5347479
>>>>> ##############################################################
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Fuseki + TDB + Lucene online text index update
Posted by Bart van Leeuwen <ba...@netage.nl>.
done
Andy Seaborne <an...@apache.org> wrote on 21-11-2014 12:27:37:
> Please suggest improvements, and contribute new text. It's always trick
> to write documentation when you are on the "inside" of the
> implementation :-)
>
> (You can edit them online with Apache CMS and the project will receive a
> patch for the website).
>
> >
> > As long as I use Fuseki to update , remove and insert data the index
stays
> > in sync, thats all I needed.
>
> Yes - that's the way to do it. Lucene is runnig same_JVM so you can
> only mess it up by updating directly while Fuseki is running.
>
> It is possible to have more complex setups with Solr as the text index,
> then update solr or query solr separately from Fuseki/TDB. Solr is an
> external service in this example.
>
> Andy
>
> >
> > Bart
> >
> >> On 18/11/2014 16:21, "Bart van Leeuwen" <ba...@netage.nl>
> > wrote:
> >>
> >>> Hi,
> >>>
> >>> I've played around with Fuseki 1.1.1
> >>> I created a TDB store which I filled with data through the Fuseki
> > control
> >>> panel
> >>>
> >>> I followed the instruction for full text queries at:
> >>> http://jena.apache.org/documentation/query/text-query.html
> >>>
> >>> Everything is up and running and works like expected.
> >>>
> >>> Now the data in the TDB store will be updated now and then is it
> > possible
> >>> to run the index updater while fuseki is still up and running ?
> >>> or do I need to bring fuseki down during that process ? ( I know TDB
is
> >>> not thread safe but afaik the indexer is not touching TDB for
writing )
> >>>
> >>> What would be the best practice for online index updating ?
> >>>
> >>> Met Vriendelijke Groet / With Kind Regards
> >>> Bart van Leeuwen
> >>>
> >>> ##############################################################
> >>> # twitter: @semanticfire
> >>> # netage.nl
> >>> # http://netage.nl
> >>> # Enschedepad 76
> >>> # 1324 GJ Almere
> >>> # The Netherlands
> >>> # tel. +31(0)36-5347479
> >>> ##############################################################
> >>
> >>
> >>
> >>
> >
>
Re: Fuseki + TDB + Lucene online text index update
Posted by Andy Seaborne <an...@apache.org>.
On 18/11/14 16:11, Bart van Leeuwen wrote:
> Rob Vesse <rv...@dotnetrdf.org> wrote on 18-11-2014 17:03:36:
>
>> Secondly it should be possible to configure Fuseki with a TDB dataset
> that
>> is attached to a Text Index such that the text index is automatically
>> updated. See
>>
> https://github.com/apache/jena/blob/master/jena-fuseki/config-tdb-text.ttl
>> for an example assembler file that does just this.
>
> This solves my workflow questions, however I do not really got this
> clearly from the docs.
Please suggest improvements, and contribute new text. It's always trick
to write documentation when you are on the "inside" of the
implementation :-)
(You can edit them online with Apache CMS and the project will receive a
patch for the website).
>
> As long as I use Fuseki to update , remove and insert data the index stays
> in sync, thats all I needed.
Yes - that's the way to do it. Lucene is runnig same_JVM so you can
only mess it up by updating directly while Fuseki is running.
It is possible to have more complex setups with Solr as the text index,
then update solr or query solr separately from Fuseki/TDB. Solr is an
external service in this example.
Andy
>
> Bart
>
>> On 18/11/2014 16:21, "Bart van Leeuwen" <ba...@netage.nl>
> wrote:
>>
>>> Hi,
>>>
>>> I've played around with Fuseki 1.1.1
>>> I created a TDB store which I filled with data through the Fuseki
> control
>>> panel
>>>
>>> I followed the instruction for full text queries at:
>>> http://jena.apache.org/documentation/query/text-query.html
>>>
>>> Everything is up and running and works like expected.
>>>
>>> Now the data in the TDB store will be updated now and then is it
> possible
>>> to run the index updater while fuseki is still up and running ?
>>> or do I need to bring fuseki down during that process ? ( I know TDB is
>>> not thread safe but afaik the indexer is not touching TDB for writing )
>>>
>>> What would be the best practice for online index updating ?
>>>
>>> Met Vriendelijke Groet / With Kind Regards
>>> Bart van Leeuwen
>>>
>>> ##############################################################
>>> # twitter: @semanticfire
>>> # netage.nl
>>> # http://netage.nl
>>> # Enschedepad 76
>>> # 1324 GJ Almere
>>> # The Netherlands
>>> # tel. +31(0)36-5347479
>>> ##############################################################
>>
>>
>>
>>
>
Re: Fuseki + TDB + Lucene online text index update
Posted by Bart van Leeuwen <ba...@netage.nl>.
Rob Vesse <rv...@dotnetrdf.org> wrote on 18-11-2014 17:03:36:
> Secondly it should be possible to configure Fuseki with a TDB dataset
that
> is attached to a Text Index such that the text index is automatically
> updated. See
>
https://github.com/apache/jena/blob/master/jena-fuseki/config-tdb-text.ttl
> for an example assembler file that does just this.
This solves my workflow questions, however I do not really got this
clearly from the docs.
As long as I use Fuseki to update , remove and insert data the index stays
in sync, thats all I needed.
Bart
> On 18/11/2014 16:21, "Bart van Leeuwen" <ba...@netage.nl>
wrote:
>
> >Hi,
> >
> >I've played around with Fuseki 1.1.1
> >I created a TDB store which I filled with data through the Fuseki
control
> >panel
> >
> >I followed the instruction for full text queries at:
> >http://jena.apache.org/documentation/query/text-query.html
> >
> >Everything is up and running and works like expected.
> >
> >Now the data in the TDB store will be updated now and then is it
possible
> >to run the index updater while fuseki is still up and running ?
> >or do I need to bring fuseki down during that process ? ( I know TDB is
> >not thread safe but afaik the indexer is not touching TDB for writing )
> >
> >What would be the best practice for online index updating ?
> >
> >Met Vriendelijke Groet / With Kind Regards
> >Bart van Leeuwen
> >
> >##############################################################
> ># twitter: @semanticfire
> ># netage.nl
> ># http://netage.nl
> ># Enschedepad 76
> ># 1324 GJ Almere
> ># The Netherlands
> ># tel. +31(0)36-5347479
> >##############################################################
>
>
>
>
Re: Fuseki + TDB + Lucene online text index update
Posted by Rob Vesse <rv...@dotnetrdf.org>.
Bart
Firstly TDB is entirely thread safe as it implements serialisable
transactions with a Multiple Reader + Single Writer concurrency model
Secondly it should be possible to configure Fuseki with a TDB dataset that
is attached to a Text Index such that the text index is automatically
updated. See
https://github.com/apache/jena/blob/master/jena-fuseki/config-tdb-text.ttl
for an example assembler file that does just this.
Of course if you do updates by taking your Fuseki instance offline and
then using tdbupdate (or an application that updates the TDB dataset) then
the index would not automatically updated and would need to be rebuilt
separately. A re-index would require read access to the TDB dataset and
write access to the Lucene index so I would guess it could be run while
Fuseki is online though I'm not sure anyone has necessarily ever tried
that.
More details of your workflow would be useful
Thanks,
Rob
On 18/11/2014 16:21, "Bart van Leeuwen" <ba...@netage.nl> wrote:
>Hi,
>
>I've played around with Fuseki 1.1.1
>I created a TDB store which I filled with data through the Fuseki control
>panel
>
>I followed the instruction for full text queries at:
>http://jena.apache.org/documentation/query/text-query.html
>
>Everything is up and running and works like expected.
>
>Now the data in the TDB store will be updated now and then is it possible
>to run the index updater while fuseki is still up and running ?
>or do I need to bring fuseki down during that process ? ( I know TDB is
>not thread safe but afaik the indexer is not touching TDB for writing )
>
>What would be the best practice for online index updating ?
>
>Met Vriendelijke Groet / With Kind Regards
>Bart van Leeuwen
>
>##############################################################
># twitter: @semanticfire
># netage.nl
># http://netage.nl
># Enschedepad 76
># 1324 GJ Almere
># The Netherlands
># tel. +31(0)36-5347479
>##############################################################