You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cayenne.apache.org by Marcin Skladaniec <ma...@ish.com.au> on 2008/05/08 01:25:25 UTC
relationship query and cache refreshing in 3tier cayenne
Hi
In our application we are using LOCAL_CACHE and cache keys to refresh
it, but only for a special context called 'shared' . We have
overridden the CayenneContext commitChanges with following to ensure
cache invalidation on every commit :
if (!isSharedContext()) {
try {
List<Class> commitedClasses = new Vector<Class>();
for (Object o : uncommittedObjects())
if (!commitedClasses.contains(o.getClass()) && o instanceof
PersistentObject)
commitedClasses.add(o.getClass());
super.commitChanges();
for (Class<? extends PersistentObject> c : commitedClasses) {
performGenericQuery(new RefreshQuery(new String[]
{ PersistentObject.defaultCacheKeyForEntity(c) }));
}
} catch (CayenneRuntimeException e) {
runtimeExceptionThrown(e, false);
}
} else {
logger.error("Attempt to save shared context", new
IllegalStateException("Shared context is read-only"));
}
whenever a select query is executed we are setting the cache policy
and keys
if (query instanceof SelectQuery) {
SelectQuery sq = ((SelectQuery) query);
if (isSharedContext) {
// if the query is on the shared context then use cache
// sq.setCachePolicy(QueryMetadata.LOCAL_CACHE);
// if the root class of the query is kind of PersistentObject
then use the cache keys
if (sq.getRoot() instanceof Class &&
PersistentObject.class.isAssignableFrom((Class<? extends
PersistentObject>) (sq.getRoot()))) {
Class<? extends PersistentObject> c = (Class<? extends
PersistentObject>) sq.getRoot();
List<String> currentCacheGroups = new Vector<String>();
if (sq.getCacheGroups() != null)
currentCacheGroups = Arrays.asList(sq.getCacheGroups());
String key = PersistentObject.defaultCacheKeyForEntity(c);
if (!currentCacheGroups.contains(key)) {
currentCacheGroups.add(key);
sq.setCacheGroups(currentCacheGroups.toArray(new String[] {}));
}
}
}
}
This works nicely, but there is one problem: the relationship query
does not return refreshed values.
An example. there are two views: One is a list of artists with
painting counts, the counts are calculated using relationship
(anArtist.getPaintings().size()). The second is a simple list of
paintings.
1) client app 1: artists list shows:
Monet - 5
Malevich - 4
vanGogh - 3
painting list contains 12 lines
client app 2 adds a new painting for vanGogh, it lists now
Monet - 5
Malevich - 4
vanGogh - 4
painting list contains 13 lines
client app 1 lists the artists again:
Monet - 5
Malevich - 4
vanGogh - 3 <- incorrect
painting list contains 13 records, which is correct
The odd thing is also that for a given client application both lists
(artist and painting) are using the same context, so if the record is
there, why it does not show up when accessed via relationship ?
Is there something I'm doing wrong ? Is there a way to force the
RelationshipQuery to refresh ?
We are using Cayenne build from sources about a month ago (svn 642725).
Marcin
Re: server-to-client event push - Re: relationship query and cache refreshing in 3tier cayenne
Posted by Andrus Adamchik <an...@objectstyle.org>.
On May 15, 2008, at 5:26 AM, Aristedes Maniatis wrote:
>
> On 15/05/2008, at 7:43 AM, Andrus Adamchik wrote:
>
>> 1. Client sends a "give me events" request in the beginning of the
>> session (separate from data requests), and reads everything that
>> comes back.
>
>
> Three thoughts:
>
> 1. Is your idea to have one queue per client, or a single queue with
> some sort of sequence numbering so a client can grab events later
> than a particular timestamp/sequence?
As the events will be dispatched immediately (there will be no client
polling), the only queue involved is the local EventManager queue.
There is a single one per DataDomain and it is shared by all server
contexts.
> 2. Even without continuations the basic idea could still work if the
> client received these events attached to query results. So every
> time a client performs a query (or executes a special polling
> request) these events are received. Not quite so real-time, but
> possibly useful in many circumstances.
Yeah - that's another possible mechanism. The mechanics of it will be
different. Unlike continuations approach, this will require a "delayed
client queue".
> 3. Are the events here just refresh query cache events, or are we
> talking about 'invalidate record id 1234 from entity Artist'
> messages? Possibly that will generate too much traffic since the
> server will not know which clients have those entities already
> cached and will have to propagate all invalidate messages to all
> clients.
Both types of events... (I guess each type can be suppressed or
enabled via configuration). I forgot to mention one thing though - we
already have per-object events (since Cayenne 1.1), but there is no
query cache events (any such events are specific to an implementation
of the QueryCache, and are not a part of Cayenne). So that's something
that we need to design.
> I'm still a little hazy about whether the server has a copy of the
> same contexts as the client does, so perhaps it does know which
> records the client is holding
Yes - the server contains "peer" contexts of all client contexts,
stored in HttpSession. Their state is almost the same as client's, the
only difference being objects loaded during callbacks execution.
Andrus
Re: server-to-client event push - Re: relationship query and cache refreshing in 3tier cayenne
Posted by Aristedes Maniatis <ar...@ish.com.au>.
On 15/05/2008, at 7:43 AM, Andrus Adamchik wrote:
> 1. Client sends a "give me events" request in the beginning of the
> session (separate from data requests), and reads everything that
> comes back.
Three thoughts:
1. Is your idea to have one queue per client, or a single queue with
some sort of sequence numbering so a client can grab events later than
a particular timestamp/sequence?
2. Even without continuations the basic idea could still work if the
client received these events attached to query results. So every time
a client performs a query (or executes a special polling request)
these events are received. Not quite so real-time, but possibly useful
in many circumstances.
3. Are the events here just refresh query cache events, or are we
talking about 'invalidate record id 1234 from entity Artist' messages?
Possibly that will generate too much traffic since the server will not
know which clients have those entities already cached and will have to
propagate all invalidate messages to all clients. I'm still a little
hazy about whether the server has a copy of the same contexts as the
client does, so perhaps it does know which records the client is
holding, but I'm not sure how SHARED_CACHE impacts that.
Ari
-------------------------->
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001 fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: server-to-client event push - Re: relationship query and cache refreshing in 3tier cayenne
Posted by Andrus Adamchik <an...@objectstyle.org>.
I just took a glance at the early draft of JSR-315 (and looks like
Jetty 7 early releases already have this API built in).
I *hope* we can emulate a persistent connection with that, using a
protocol similar to this:
1. Client sends a "give me events" request in the beginning of the
session (separate from data requests), and reads everything that comes
back.
2. Server suspends the request thread, and keeps it suspended until
the client quits.
3. When an event arrives on the server, it is written to response and
response.flushBuffer() is called (I guess we'll need to resume the
request to do that and then suspend it again ... I hope multi-suspend
is supported ? Or can we flush without resuming?)
Of course all that is pluggable via an EventBridge implementation, so
given the time and motivation we can download Jetty 7 and start
playing with it right away.
Andrus
On May 14, 2008, at 4:42 PM, Tore Halset wrote:
> Hello.
>
> On 14. mai. 2008, at 00.46, Andrus Adamchik wrote:
>
>> (2) Setup server-to-client event push. I did that in the past with
>> a always-on XMPP connection in parallel with the main HTTPS data
>> channel (I know it worked for object updates... it needs to be
>> extended to cache groups invalidation). For a long time I wanted to
>> investigate Jetty Continuations (and more generally, queuing events
>> on the server for each live client, and letting client frequently
>> poll for events). This would allow to reuse the main ROP
>> connection, but this needs to be developed yet.
>
> It looks like the thing Jetty are doing with Continuations will be
> standardized in Servlet 3.0
>
> http://blogs.webtide.com:80/gregw/2008/04/28/1209355449829.html
>
> Regards,
> - Tore.
>
server-to-client event push - Re: relationship query and cache refreshing in 3tier cayenne
Posted by Tore Halset <ha...@pvv.ntnu.no>.
Hello.
On 14. mai. 2008, at 00.46, Andrus Adamchik wrote:
> (2) Setup server-to-client event push. I did that in the past with a
> always-on XMPP connection in parallel with the main HTTPS data
> channel (I know it worked for object updates... it needs to be
> extended to cache groups invalidation). For a long time I wanted to
> investigate Jetty Continuations (and more generally, queuing events
> on the server for each live client, and letting client frequently
> poll for events). This would allow to reuse the main ROP connection,
> but this needs to be developed yet.
It looks like the thing Jetty are doing with Continuations will be
standardized in Servlet 3.0
http://blogs.webtide.com:80/gregw/2008/04/28/1209355449829.html
Regards,
- Tore.
Re: UML class diagram from cayenne maps
Posted by Kevin Menard <km...@servprise.com>.
There's nothing that I know of. It'd be awesome if the modeler could show
ER and class diagrams, but without a champion, those are really unlikely to
happen anytime soon.
Otherwise, your best bet is to use a UML tool that scans Java files. There
are plenty of them out there.
--
Kevin
On 5/14/08 8:38 AM, "Hans Poo" <ha...@welinux.cl> wrote:
> Hi,
>
> I would like to know if there is a way to see a UML class diagram from cayenne
> mappings.
>
> Until now, the best shoot i can take is with a database reverse engineer, but
> is very incomplete,
> starting with the differences on names ...
>
> In the list there are some old postings taking about some poseidon plug-in,
> but they are old...
>
> Thanks
> Hans
>
UML class diagram from cayenne maps
Posted by Hans Poo <ha...@welinux.cl>.
Hi,
I would like to know if there is a way to see a UML class diagram from cayenne mappings.
Until now, the best shoot i can take is with a database reverse engineer, but is very incomplete,
starting with the differences on names ...
In the list there are some old postings taking about some poseidon plug-in, but they are old...
Thanks
Hans
Re: relationship query and cache refreshing in 3tier cayenne
Posted by Andrus Adamchik <an...@objectstyle.org>.
On May 14, 2008, at 4:06 AM, Marcin Skladaniec wrote:
> [1] - re-doing selectquery in different client seems to work! Andrus
> this opposes to what you have said about the RefreshQuery propagation!
Weird... This means that caching is not working for some reason.
Andrus
Re: relationship query and cache refreshing in 3tier cayenne
Posted by Marcin Skladaniec <ma...@ish.com.au>.
Hello
We have started to write the caching documentation, and we will post
it to the list for verification this week. Before that I would like to
clarify my previous email.
From what I can see the LOCAL_CACHE refresh does propagate between
clients. This is what I do in pseudo code:
first initial check on client 1:
performQuery(Painting).size() -> 10 //fetching the entire list of
Paintings using SelectQuery
getArtistWithName("Monet").getPaintings() -> 2 //fetching the list of
paintings using RelationshipQuery
now the same on client 2:
performQuery(Painting).size() -> 10
getArtistWithName("Monet").getPaintings() -> 2
//create a new painting:
getArtistWithName("Monet").addPainting(newPainting());
commitChanges();
//client 2 has all the data updated, as expected:
performQuery(Painting).size() -> 11
getArtistWithName("Monet").getPaintings() -> 3
//back to client 1:
performQuery(Painting).size() -> 11 [1]
getArtistWithName("Monet").getPaintings() -> 2 [2]
[1] - re-doing selectquery in different client seems to work! Andrus
this opposes to what you have said about the RefreshQuery propagation!
[2] - even so, the relationship is not ...
All select queries are executed with paging and LOCAL_CACHE. Every
entity uses a separate cache key.
Best regards
Marcin
On 14/05/2008, at 8:46 AM, Andrus Adamchik wrote:
> From what I see the problem is that RefreshQuery is not propagated
> to remote clients. LOCAL_CACHE is local the tier where the query
> originated. With some fixes in M4, this statement is true for both
> nested contexts and ROP... Since by default there are no events
> pushed from the server to the client and no peer-to-peer client
> communication, client 1 uses its local cache and knows nothing about
> the RefreshQuery executed by client 2.
>
> There are a few solutions to that:
>
> (1) set timemout on the query cache by using something like an
> OSCache cache provider. This will give you a delayed refresh. This
> may or may not work for your app, but this can be used with the
> current Cayenne. For many apps that would be a reasonable
> compromise, with cache ensuring that queries don't happen too often..
>
> (2) Setup server-to-client event push. I did that in the past with a
> always-on XMPP connection in parallel with the main HTTPS data
> channel (I know it worked for object updates... it needs to be
> extended to cache groups invalidation). For a long time I wanted to
> investigate Jetty Continuations (and more generally, queuing events
> on the server for each live client, and letting client frequently
> poll for events). This would allow to reuse the main ROP connection,
> but this needs to be developed yet.
>
> Andrus
>
>
> On May 7, 2008, at 7:25 PM, Marcin Skladaniec wrote:
>> Hi
>>
>> In our application we are using LOCAL_CACHE and cache keys to
>> refresh it, but only for a special context called 'shared' . We
>> have overridden the CayenneContext commitChanges with following to
>> ensure cache invalidation on every commit :
>>
>> if (!isSharedContext()) {
>> try {
>> List<Class> commitedClasses = new Vector<Class>();
>> for (Object o : uncommittedObjects())
>> if (!commitedClasses.contains(o.getClass()) && o instanceof
>> PersistentObject)
>> commitedClasses.add(o.getClass());
>>
>> super.commitChanges();
>>
>> for (Class<? extends PersistentObject> c : commitedClasses) {
>> performGenericQuery(new RefreshQuery(new String[]
>> { PersistentObject.defaultCacheKeyForEntity(c) }));
>> }
>>
>> } catch (CayenneRuntimeException e) {
>> runtimeExceptionThrown(e, false);
>> }
>> } else {
>> logger.error("Attempt to save shared context", new
>> IllegalStateException("Shared context is read-only"));
>> }
>>
>> whenever a select query is executed we are setting the cache policy
>> and keys
>>
>> if (query instanceof SelectQuery) {
>> SelectQuery sq = ((SelectQuery) query);
>> if (isSharedContext) {
>> // if the query is on the shared context then use cache
>> // sq.setCachePolicy(QueryMetadata.LOCAL_CACHE);
>>
>> // if the root class of the query is kind of PersistentObject
>> then use the cache keys
>> if (sq.getRoot() instanceof Class &&
>> PersistentObject.class.isAssignableFrom((Class<? extends
>> PersistentObject>) (sq.getRoot()))) {
>> Class<? extends PersistentObject> c = (Class<? extends
>> PersistentObject>) sq.getRoot();
>> List<String> currentCacheGroups = new Vector<String>();
>> if (sq.getCacheGroups() != null)
>> currentCacheGroups = Arrays.asList(sq.getCacheGroups());
>>
>> String key = PersistentObject.defaultCacheKeyForEntity(c);
>> if (!currentCacheGroups.contains(key)) {
>> currentCacheGroups.add(key);
>> sq.setCacheGroups(currentCacheGroups.toArray(new String[]
>> {}));
>> }
>> }
>>
>> }
>> }
>>
>>
>> This works nicely, but there is one problem: the relationship query
>> does not return refreshed values.
>>
>> An example. there are two views: One is a list of artists with
>> painting counts, the counts are calculated using relationship
>> (anArtist.getPaintings().size()). The second is a simple list of
>> paintings.
>> 1) client app 1: artists list shows:
>> Monet - 5
>> Malevich - 4
>> vanGogh - 3
>> painting list contains 12 lines
>> client app 2 adds a new painting for vanGogh, it lists now
>> Monet - 5
>> Malevich - 4
>> vanGogh - 4
>> painting list contains 13 lines
>> client app 1 lists the artists again:
>> Monet - 5
>> Malevich - 4
>> vanGogh - 3 <- incorrect
>> painting list contains 13 records, which is correct
>>
>> The odd thing is also that for a given client application both
>> lists (artist and painting) are using the same context, so if the
>> record is there, why it does not show up when accessed via
>> relationship ?
>>
>> Is there something I'm doing wrong ? Is there a way to force the
>> RelationshipQuery to refresh ?
>> We are using Cayenne build from sources about a month ago (svn
>> 642725).
>> Marcin
>>
>>
>>
>
Re: relationship query and cache refreshing in 3tier cayenne
Posted by Andrus Adamchik <an...@objectstyle.org>.
From what I see the problem is that RefreshQuery is not propagated to
remote clients. LOCAL_CACHE is local the tier where the query
originated. With some fixes in M4, this statement is true for both
nested contexts and ROP... Since by default there are no events pushed
from the server to the client and no peer-to-peer client
communication, client 1 uses its local cache and knows nothing about
the RefreshQuery executed by client 2.
There are a few solutions to that:
(1) set timemout on the query cache by using something like an OSCache
cache provider. This will give you a delayed refresh. This may or may
not work for your app, but this can be used with the current Cayenne.
For many apps that would be a reasonable compromise, with cache
ensuring that queries don't happen too often..
(2) Setup server-to-client event push. I did that in the past with a
always-on XMPP connection in parallel with the main HTTPS data channel
(I know it worked for object updates... it needs to be extended to
cache groups invalidation). For a long time I wanted to investigate
Jetty Continuations (and more generally, queuing events on the server
for each live client, and letting client frequently poll for events).
This would allow to reuse the main ROP connection, but this needs to
be developed yet.
Andrus
On May 7, 2008, at 7:25 PM, Marcin Skladaniec wrote:
> Hi
>
> In our application we are using LOCAL_CACHE and cache keys to
> refresh it, but only for a special context called 'shared' . We have
> overridden the CayenneContext commitChanges with following to ensure
> cache invalidation on every commit :
>
> if (!isSharedContext()) {
> try {
> List<Class> commitedClasses = new Vector<Class>();
> for (Object o : uncommittedObjects())
> if (!commitedClasses.contains(o.getClass()) && o instanceof
> PersistentObject)
> commitedClasses.add(o.getClass());
>
> super.commitChanges();
>
> for (Class<? extends PersistentObject> c : commitedClasses) {
> performGenericQuery(new RefreshQuery(new String[]
> { PersistentObject.defaultCacheKeyForEntity(c) }));
> }
>
> } catch (CayenneRuntimeException e) {
> runtimeExceptionThrown(e, false);
> }
> } else {
> logger.error("Attempt to save shared context", new
> IllegalStateException("Shared context is read-only"));
> }
>
> whenever a select query is executed we are setting the cache policy
> and keys
>
> if (query instanceof SelectQuery) {
> SelectQuery sq = ((SelectQuery) query);
> if (isSharedContext) {
> // if the query is on the shared context then use cache
> // sq.setCachePolicy(QueryMetadata.LOCAL_CACHE);
>
> // if the root class of the query is kind of PersistentObject
> then use the cache keys
> if (sq.getRoot() instanceof Class &&
> PersistentObject.class.isAssignableFrom((Class<? extends
> PersistentObject>) (sq.getRoot()))) {
> Class<? extends PersistentObject> c = (Class<? extends
> PersistentObject>) sq.getRoot();
> List<String> currentCacheGroups = new Vector<String>();
> if (sq.getCacheGroups() != null)
> currentCacheGroups = Arrays.asList(sq.getCacheGroups());
>
> String key = PersistentObject.defaultCacheKeyForEntity(c);
> if (!currentCacheGroups.contains(key)) {
> currentCacheGroups.add(key);
> sq.setCacheGroups(currentCacheGroups.toArray(new String[] {}));
> }
> }
>
> }
> }
>
>
> This works nicely, but there is one problem: the relationship query
> does not return refreshed values.
>
> An example. there are two views: One is a list of artists with
> painting counts, the counts are calculated using relationship
> (anArtist.getPaintings().size()). The second is a simple list of
> paintings.
> 1) client app 1: artists list shows:
> Monet - 5
> Malevich - 4
> vanGogh - 3
> painting list contains 12 lines
> client app 2 adds a new painting for vanGogh, it lists now
> Monet - 5
> Malevich - 4
> vanGogh - 4
> painting list contains 13 lines
> client app 1 lists the artists again:
> Monet - 5
> Malevich - 4
> vanGogh - 3 <- incorrect
> painting list contains 13 records, which is correct
>
> The odd thing is also that for a given client application both lists
> (artist and painting) are using the same context, so if the record
> is there, why it does not show up when accessed via relationship ?
>
> Is there something I'm doing wrong ? Is there a way to force the
> RelationshipQuery to refresh ?
> We are using Cayenne build from sources about a month ago (svn
> 642725).
> Marcin
>
>
>