Webdav servlet caching new info

[Andy Scholz <an...@exodelta.com>]

Hi,

I have installed Slide (1.0.14) with Tomcat 4.0 & configured it to use 
mysql (3.23/mm-mysql-2.0.4) with a file system content store. This seems 
to work fine while all access is via webdav (i.e. webdav servlet).

The problem I have is that I am using one of the Slide api sample files 
(version.java) to create some arbitrary folders using the same store 
(same domain.xml file). This works OK, but after the new folders have 
been created, when I go back to my webdav client (or browser) & do a 
refresh, the new folders dont appear unless I stop/restart the Tomcat 
server - after which a PROPFIND or GET returns the new info OK.

It appears the servlet is caching the info, but I cant see how/where. 
Now if  I force a GET explicitly on the new folder URI (which is OK), it 
will then appear in any subsequent responses to PROPFIND or GET requests 
for the parent - seems like this forces a 'cache' refresh.

Is this supposed to work like this? Is this a Slide thing or maybe mysql 
related?

If someone could shed some light on this I would greatly appreciate it.

Thanks in advance,
Andy Scholz

Re: Webdav servlet caching new info

[Andy Scholz <an...@exodelta.com>]

>
>
>Well, we "only" need a fast mechanism to access an equivalent of a timestamp
>for a node. There's a unique (and timestamped) id for each write operation
>made by Slide (that's the transaction xid). By associating the "timestamp"
>with the uri in a special store, we could easily do cache revalidation.
>
Thanks Remy - I see where your coming from here - makes perfect sense.

Question tho: Shouldnt the timestamp be determined by the contentstore 
provider? My principle concern is more about the validity/integrity of 
the data rather than the performance aspects.

Is there any simple way for a client app to to force a refresh of the 
cache (without 'writing' anything)?

Regards,
Andy Scholz

Re: Webdav servlet caching new info

[Remy Maucherat <rm...@home.com>]

> Thank you Remi, especially for your quick response.
>
> You've largely answered my question, but now I have an issue with
> scalability - I have had a very similar problem recently with a project
> I did for a client - first thing they did was install two different web
> servers (i.e. physical machines) using the same repository. Although I
> was only caching schema/security info rather than content - they
> immediately queried why changes made to one were not reflected in the
> other (until a restart). I added an option for an administrator to
> reload the cache (via a web page), which was fine for them, because
> changes were rare. Obviously this techinique is of little use when we're
> dealing with user content.
>
> This is something of a show-stopper for me. You mention you have lots of
> ideas - any more info? Can I help?

Well, we "only" need a fast mechanism to access an equivalent of a timestamp
for a node. There's a unique (and timestamped) id for each write operation
made by Slide (that's the transaction xid). By associating the "timestamp"
with the uri in a special store, we could easily do cache revalidation.
Overall, there are quite a few behind-the-scene changes that could improve
scalability, but it will take some time.

At the moment, the focus is on building additional APIs and interfaces on
top of Slide (support for more WebDAV protocols, additional features, HTML
interface, ...).

Remy

Re: Webdav servlet caching new info

[Andy Scholz <an...@exodelta.com>]

Thank you Remi, especially for your quick response.

You've largely answered my question, but now I have an issue with 
scalability - I have had a very similar problem recently with a project 
I did for a client - first thing they did was install two different web 
servers (i.e. physical machines) using the same repository. Although I 
was only caching schema/security info rather than content - they 
immediately queried why changes made to one were not reflected in the 
other (until a restart). I added an option for an administrator to 
reload the cache (via a web page), which was fine for them, because 
changes were rare. Obviously this techinique is of little use when we're 
dealing with user content.

This is something of a show-stopper for me. You mention you have lots of 
ideas - any more info? Can I help?

Thanks & Regards,
Andy Scholz



Remy Maucherat wrote:

>>The problem I have is that I am using one of the Slide api sample files
>>(version.java) to create some arbitrary folders using the same store
>>(same domain.xml file). This works OK, but after the new folders have
>>been created, when I go back to my webdav client (or browser) & do a
>>refresh, the new folders dont appear unless I stop/restart the Tomcat
>>server - after which a PROPFIND or GET returns the new info OK.
>>
>If the two applications (here, your application, and the webdav servlet) are
>not in the same VM, then it's (unfortunately) normal with the current
>architecture.
>
>The cache is located in the layer just above mySQL, but never attempts to
>revalidate its entries, unless it has a good reason to do so (for example,
>if you modify something through the API). This was the only easy way to get
>decent performance, since PROPFINDs are extremely hard on the backend (you
>can check the number of operations required when doing one on a large
>resource set). I have a lot of ideas to help solve these problems, but this
>will take some time.
>
>The easiest way to avoid the problem would be to repackage a bit the
>application. Either:
>- write a custom loader for Tomcat and your application to be able to run
>them in the same VM
>- use a controller servlet to run your application
>
>Remy
>

Re: Webdav servlet caching new info

[Remy Maucherat <rm...@home.com>]

> Hi,
>
> I have installed Slide (1.0.14) with Tomcat 4.0 & configured it to use
> mysql (3.23/mm-mysql-2.0.4) with a file system content store. This seems
> to work fine while all access is via webdav (i.e. webdav servlet).
>
> The problem I have is that I am using one of the Slide api sample files
> (version.java) to create some arbitrary folders using the same store
> (same domain.xml file). This works OK, but after the new folders have
> been created, when I go back to my webdav client (or browser) & do a
> refresh, the new folders dont appear unless I stop/restart the Tomcat
> server - after which a PROPFIND or GET returns the new info OK.
>
> It appears the servlet is caching the info, but I cant see how/where.
> Now if  I force a GET explicitly on the new folder URI (which is OK), it
> will then appear in any subsequent responses to PROPFIND or GET requests
> for the parent - seems like this forces a 'cache' refresh.
>
> Is this supposed to work like this? Is this a Slide thing or maybe mysql
> related?

If the two applications (here, your application, and the webdav servlet) are
not in the same VM, then it's (unfortunately) normal with the current
architecture.

The cache is located in the layer just above mySQL, but never attempts to
revalidate its entries, unless it has a good reason to do so (for example,
if you modify something through the API). This was the only easy way to get
decent performance, since PROPFINDs are extremely hard on the backend (you
can check the number of operations required when doing one on a large
resource set). I have a lot of ideas to help solve these problems, but this
will take some time.

The easiest way to avoid the problem would be to repackage a bit the
application. Either:
- write a custom loader for Tomcat and your application to be able to run
them in the same VM
- use a controller servlet to run your application

Remy