Online reloading of Fuseki config file

[Alexander Dutton <al...@oucs.ox.ac.uk>]

Hi all,

I was hoping to extend Fuseki so that I can POST to a servlet that will
cause the config file to be reloaded.

The full requirements would be:

* Servlets are added or removed as necessary
* Sync and forget about any datasets that are no longer referenced.
* Don't sync any datasets that are still in use
* Requests already in progress are allowed to finish

Assuming this isn't stupid, I *think* the steps that would need to take
place are:

1) Determine a new list of services by calling
FusekiConfig.configure(fusekiConfigFile). This will also open any
datasets that weren't previously open.
2) Update the servlet mapping to reflect the new list of services. This
will probably want doing by playing with ServletHandler.setServletMapping()
3) Get hold of the CachingTDBMaker if it's in use, sync any datasets
which are no longer needed, and remove them from the cache.

This probably wants encapsulating in a locked block, with requests that
can't get the lock given a 409 Conflict to say that it's already in process.

At the moment CachedTDBMaker doesn't expose a list of things in its
cache, which would be needed for (3). I also don't know how this might
work with datasets that aren't provided by TDB; what else might I need
to consider?

If this seems like a Good Idea (or at least, not a Bad Idea), I'll
create an issue in JIRA and start sticking some code in GitHub…

All the best,

Alex

Re: Online reloading of Fuseki config file

[Andy Seaborne <an...@apache.org>]

(sorry for the delay)

On 17/01/12 13:39, Alexander Dutton wrote:
> Hi all,
>
> I was hoping to extend Fuseki so that I can POST to a servlet that will
> cause the config file to be reloaded.
>
> The full requirements would be:
>
> * Servlets are added or removed as necessary
> * Sync and forget about any datasets that are no longer referenced.
> * Don't sync any datasets that are still in use
> * Requests already in progress are allowed to finish
>
> Assuming this isn't stupid,

It would be great to have a way of updating the configuration without 
needing to stop the server or loose outstanding requests.

Transaction means that the sync condition is possible no longer an issue.

> I *think* the steps that would need to take
> place are:
>
> 1) Determine a new list of services by calling
> FusekiConfig.configure(fusekiConfigFile). This will also open any
> datasets that weren't previously open.
> 2) Update the servlet mapping to reflect the new list of services. This
> will probably want doing by playing with ServletHandler.setServletMapping()
> 3) Get hold of the CachingTDBMaker if it's in use, sync any datasets
> which are no longer needed, and remove them from the cache.

Yes - needs to be some kind of "diff" between current and new state and 
a set of changes can be applied.  It might work to simply drop the old 
servlet state and put in a completely new setup - if so, that's easier 
but might have a moment of disruption.  Not sure if there is a place to 
put a lock in to synchronize the changes as Jetty does the dispatch 
before anything gets to Fuseki.

A possibly-too-complicated idea is to have a fixed filter or plain 
servlet that intercepts all requests, and can momentarily hold up while 
a switch is happening.

> This probably wants encapsulating in a locked block, with requests that
> can't get the lock given a 409 Conflict to say that it's already in process.

Reading the new config and preparing for the change can be done before 
incoming requests need to be halted - it might be a very quick change 
over and a simple synchronization lock, not bumping request, might work out.

>
> At the moment CachedTDBMaker doesn't expose a list of things in its
> cache, which would be needed for (3). I also don't know how this might
> work with datasets that aren't provided by TDB; what else might I need
> to consider?

CachedTDBMaker is no longer used.  As of a couple of weeks ago, it was 
used even when using transactions but the system ended up with two 
caches and they interacted.

The cache is in StoreConnection, which is the global coordinator for 
transactions.

> If this seems like a Good Idea (or at least, not a Bad Idea), I'll
> create an issue in JIRA and start sticking some code in GitHub…
>
> All the best,
>
> Alex

	Andy