Depending on TransportListener for cache consistency

[jlindwall <jl...@yahoo.com>]

We use a JMS topic to coordinate distributed caches.  We have a topic called
"caching" using non-persistent messages.  The subscribers are (of course)
non-durable.

A message in the topic means "clear your cache for the specified datatype
(and id)".  Multiple webapps subscribe to this topic (and also publish). 
When a webapp makes a change to a piece of data, it publishes a message to
the topic to tell the other webapps to clear their cache for that particular
piece of data.  In this way we enforce cache consistency.

We are using a "Shared filesystem Master/Slave" using kahadb.  If the master
fails, then the clients are notified via the ActiveMQ TransportListener
mechanism.  When the connection is determined to be lost (via the
TransportListener), each client clears ALL ENTRIES in it's cache.  When the
connection is reestablished, the cache will be refilled naturally.  This
keeps the cache consistent in the face of a broker failure.

We contemplated using persistent messaging and durable subscribers for this
scheme, to make sure no message were lost. I am thinking that we do not need
to pay the price for persistence and instead use the above scheme based on
TransportListener.

Can you see any faults in this approach?

Thanks!



--
View this message in context: http://activemq.2283324.n4.nabble.com/Depending-on-TransportListener-for-cache-consistency-tp4700535.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Depending on TransportListener for cache consistency

[jlindwall <jl...@yahoo.com>]

We will not be using a network of brokers, due to the possibility of
persistent messages getting undelivered after a failover (not sure if
"orphaned" is a better term?) due to this issue:

https://issues.apache.org/jira/browse/AMQ-5897

We are using a shared filesystem master/slave setup.  We are willing to live
with a brief window of staleness inherent in this solution, as you mentioned
below. We could enable synchronous sends on our messages, which I think
would eliminate one message loss case you described. However that starts me
feeling like I'm becoming a dutchboy :)  There are surely other
possibilities of message loss when persistence is disabled.

I appreciate your suggestions to consider distributed caching solutions.





--
View this message in context: http://activemq.2283324.n4.nabble.com/Depending-on-TransportListener-for-cache-consistency-tp4700535p4701192.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Depending on TransportListener for cache consistency

[artnaseef <ar...@artnaseef.com>]

There are other concerns as well.  For example, when using a network of
brokers, topic messages can be lost if a network connection between brokers
goes down, even just for an instant, and there's no way for the applications
to detect this loss (unless your messages have something that can be used to
detect loss).

Thinking about this problem leads me to thinking of hazelcast and others,
which then makes me wonder why you're implementing a distributed cache as
there are awesome toolsets for that very use-cache.  Ehcache is definitely
one.  I believe hazelcast is another.  Something to consider.

The biggest concern that comes to mind using JMS for this purpose is the
time it takes for messages to distribute to all the consumers which raises
the question, "how bad is it if one of the cache copies is out of date for
any length of time?"



--
View this message in context: http://activemq.2283324.n4.nabble.com/Depending-on-TransportListener-for-cache-consistency-tp4700535p4701174.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Depending on TransportListener for cache consistency

[jlindwall <jl...@yahoo.com>]

You raise some valid-sounding concerns: specifically the information that
there can be message loss for non-durable subscriptions even in the absence
of a connection failure or broker failure.

Our cached items remain cached for extended periods, so we'd in trouble if
we miss a cache message for any reason.

I am now thinking we need to use durable subscribers for these cache
messages, unless there is some other activemq magic that I'm overlooking.

Thanks!





--
View this message in context: http://activemq.2283324.n4.nabble.com/Depending-on-TransportListener-for-cache-consistency-tp4700535p4701167.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Depending on TransportListener for cache consistency

[Christopher Shannon <ch...@gmail.com>]

I don't know the exactly details of your use case but this seems reasonable
to me as long as you account for the fact that messages could still be lost
or not received in a timely matter even if there isn't a broker restart or
lost connection.  An example where a message could be lost is if the topic
is backed up and the client is doing an asynchronous send. Another example
would be a back log on the topic that causes producer flow control to kick
so that messages could be received slowly or could expire if there a TTL
set. One thing you could do to account for these kinds of things is to just
set a max time on elements in your cache before they are evicted
automatically to make sure nothing is stale in case the flow breaks down.

On Thu, Aug 6, 2015 at 4:37 PM, jlindwall <jl...@yahoo.com> wrote:

> We use a JMS topic to coordinate distributed caches.  We have a topic
> called
> "caching" using non-persistent messages.  The subscribers are (of course)
> non-durable.
>
> A message in the topic means "clear your cache for the specified datatype
> (and id)".  Multiple webapps subscribe to this topic (and also publish).
> When a webapp makes a change to a piece of data, it publishes a message to
> the topic to tell the other webapps to clear their cache for that
> particular
> piece of data.  In this way we enforce cache consistency.
>
> We are using a "Shared filesystem Master/Slave" using kahadb.  If the
> master
> fails, then the clients are notified via the ActiveMQ TransportListener
> mechanism.  When the connection is determined to be lost (via the
> TransportListener), each client clears ALL ENTRIES in it's cache.  When the
> connection is reestablished, the cache will be refilled naturally.  This
> keeps the cache consistent in the face of a broker failure.
>
> We contemplated using persistent messaging and durable subscribers for this
> scheme, to make sure no message were lost. I am thinking that we do not
> need
> to pay the price for persistence and instead use the above scheme based on
> TransportListener.
>
> Can you see any faults in this approach?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Depending-on-TransportListener-for-cache-consistency-tp4700535.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>