You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jcs-users@jakarta.apache.org by Silvano <me...@ipsoft.it> on 2009/06/17 10:23:52 UTC
Indexed Disk Cache safe copying of files
Since we have a large cache that needs significant time to be populated,
we need to be able to copy the files of the regions configured as
Indexed Disk Cache between different installations.
We believe this can be done safely if we turn off the jvm, but we would
need to copy safely the files while the jvm is running. However it seems
that the key files are written only when you turn off the jvm (at least
the size changes when we turn off the jvm, so we suspect that they are
not synchronized to disk in normal operations).
We can make sure at application level that nobody writes in the cache
during the file copy process, but is this sufficient?
Can anybody recommend a solution to this issue?
Best regards,
Silvano.
---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-users-help@jakarta.apache.org
Re: Indexed Disk Cache safe copying of files
Posted by Silvano <me...@ipsoft.it>.
Hi Aron, Thanks a lot for the interest shown in supporting the issue.
This approach seems to be the most applicable to our case. If you think
you can work on it, please let we know and we can cooperate and help
with testing or supporting implementation or whatever needed.
- createSnapshot would be first thing we need, at first glance having
some 'location' argument to control directory where to write the
consistent copy
- loadFromSnapshot might be a plus - however if a consistent snapshot
for a region can be moved at ease using Unix commands, a restore from a
snapshot could be done by moving phisically the files to the desired
location.
- timer ... again, this might be handled at the client, but of course
would be a plus to configure as a timer
We might need to have control over backing up different regions either
individually, or several together e.g. in a short sequence so the backup
is of diffferent regions is roughly at the same time). We say 'roughly'
since the cache does not vary very often and/ or we could limit writes
to the cache at the application level for some time if needed to help
consistency of the snapshot.
Robustness of the snapshot creation would be the most important requirement.
in general, we should consider that the files can be large enough (we
have so far 1.5 GB of cached data - reason for caching a lot is that the
backend is a CMS - rather slow to generate web pages).
Silvano.
Aaron Smuts wrote:
> The keys are not written until shutdown. I could create a method that could write them to disk and make a copy of the datafile. It could be called "createSnapShot" You could call it programatically. There are details to think through. Should we try to load from a snapshot on startup? How many snapshots should we store? Could we create these via a timer? I guess this could all be configurable.
>
> Aaron
>
>
>
> ----- Original Message ----
> From: Silvano <me...@ipsoft.it>
> To: jcs-users@jakarta.apache.org
> Sent: Wednesday, June 17, 2009 4:23:52 AM
> Subject: Indexed Disk Cache safe copying of files
>
> Since we have a large cache that needs significant time to be populated, we need to be able to copy the files of the regions configured as Indexed Disk Cache between different installations.
> We believe this can be done safely if we turn off the jvm, but we would need to copy safely the files while the jvm is running. However it seems that the key files are written only when you turn off the jvm (at least the size changes when we turn off the jvm, so we suspect that they are not synchronized to disk in normal operations).
>
> We can make sure at application level that nobody writes in the cache during the file copy process, but is this sufficient?
>
> Can anybody recommend a solution to this issue?
>
> Best regards,
> Silvano.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jcs-users-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jcs-users-help@jakarta.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-users-help@jakarta.apache.org
Re: Indexed Disk Cache safe copying of files
Posted by Aaron Smuts <as...@yahoo.com>.
The keys are not written until shutdown. I could create a method that could write them to disk and make a copy of the datafile. It could be called "createSnapShot" You could call it programatically. There are details to think through. Should we try to load from a snapshot on startup? How many snapshots should we store? Could we create these via a timer? I guess this could all be configurable.
Aaron
----- Original Message ----
From: Silvano <me...@ipsoft.it>
To: jcs-users@jakarta.apache.org
Sent: Wednesday, June 17, 2009 4:23:52 AM
Subject: Indexed Disk Cache safe copying of files
Since we have a large cache that needs significant time to be populated, we need to be able to copy the files of the regions configured as Indexed Disk Cache between different installations.
We believe this can be done safely if we turn off the jvm, but we would need to copy safely the files while the jvm is running. However it seems that the key files are written only when you turn off the jvm (at least the size changes when we turn off the jvm, so we suspect that they are not synchronized to disk in normal operations).
We can make sure at application level that nobody writes in the cache during the file copy process, but is this sufficient?
Can anybody recommend a solution to this issue?
Best regards,
Silvano.
---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-users-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-users-help@jakarta.apache.org
Re: Indexed Disk Cache safe copying of files
Posted by Tore Halset <ha...@pvv.ntnu.no>.
On Jun 17, 2009, at 22:50, Aaron Smuts wrote:
> We need to think through the way updates are done. If these will be
> shared, we'll have to have some kind of lock, perhaps another file
> with the same name with a .lock suffix. . . . . .
As you see from the patch, I create a new tmp file and rename it after
write. Several writers can write in parallel, and the last will win.
File tmp = File.createTempFile("cache", null, dir);
...
tmp.renameTo(file);
Regards,
- Tore.
---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-users-help@jakarta.apache.org
Re: Indexed Disk Cache safe copying of files
Posted by Aaron Smuts <as...@yahoo.com>.
Thanks. I commented on JIRA.
I'll have to reimplement the general idea without any JDK 1.4 dependencies.
We need to think through the way updates are done. If these will be shared, we'll have to have some kind of lock, perhaps another file with the same name with a .lock suffix. . . . . .
Cheers,
Aaron
----- Original Message ----
From: Tore Halset <ha...@pvv.ntnu.no>
To: JCS Users List <jc...@jakarta.apache.org>
Sent: Wednesday, June 17, 2009 4:35:11 PM
Subject: Re: Indexed Disk Cache safe copying of files
Hello.
I do this in my jcs cache. The items in the cache is 0-30MB each. The total volume is ~20GB. Instead of the standard indexed disc cache, I use a file based cache. The key is converted to a filename and each item is stored as a separate file. Several jvms can share the same cache and the cache can be rsynced between nodes while running the application.
This setup might be slow for caches with lots of small objects, but it works very well with my setup.
The source code is attached to this issue:
https://issues.apache.org/jira/browse/JCS-58
To be able to hold as many objects as possible in ram without getting OutOfMemoryException, I use this cache with a soft reference cache.
https://issues.apache.org/jira/browse/JCS-54
Regards,
- Tore.
On Jun 17, 2009, at 10:23, Silvano wrote:
> Since we have a large cache that needs significant time to be populated, we need to be able to copy the files of the regions configured as Indexed Disk Cache between different installations.
> We believe this can be done safely if we turn off the jvm, but we would need to copy safely the files while the jvm is running. However it seems that the key files are written only when you turn off the jvm (at least the size changes when we turn off the jvm, so we suspect that they are not synchronized to disk in normal operations).
>
> We can make sure at application level that nobody writes in the cache during the file copy process, but is this sufficient?
>
> Can anybody recommend a solution to this issue?
>
> Best regards,
> Silvano.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jcs-users-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-users-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-users-help@jakarta.apache.org
Re: Indexed Disk Cache safe copying of files
Posted by Tore Halset <ha...@pvv.ntnu.no>.
Hello.
I do this in my jcs cache. The items in the cache is 0-30MB each. The
total volume is ~20GB. Instead of the standard indexed disc cache, I
use a file based cache. The key is converted to a filename and each
item is stored as a separate file. Several jvms can share the same
cache and the cache can be rsynced between nodes while running the
application.
This setup might be slow for caches with lots of small objects, but it
works very well with my setup.
The source code is attached to this issue:
https://issues.apache.org/jira/browse/JCS-58
To be able to hold as many objects as possible in ram without getting
OutOfMemoryException, I use this cache with a soft reference cache.
https://issues.apache.org/jira/browse/JCS-54
Regards,
- Tore.
On Jun 17, 2009, at 10:23, Silvano wrote:
> Since we have a large cache that needs significant time to be
> populated, we need to be able to copy the files of the regions
> configured as Indexed Disk Cache between different installations.
> We believe this can be done safely if we turn off the jvm, but we
> would need to copy safely the files while the jvm is running.
> However it seems that the key files are written only when you turn
> off the jvm (at least the size changes when we turn off the jvm, so
> we suspect that they are not synchronized to disk in normal
> operations).
>
> We can make sure at application level that nobody writes in the
> cache during the file copy process, but is this sufficient?
>
> Can anybody recommend a solution to this issue?
>
> Best regards,
> Silvano.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jcs-users-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-users-help@jakarta.apache.org