Re: using memcached as a StateDB.. getting there

[Thanos Chatziathanassiou <tc...@arx.gr>]

(ignoring the fact that no-one seems interested)

I came accross Cache::Memcached::Tie which pretty much does most of the 
work, however I'm having some issues - actually design decisions:
- I suppose StateDir should equal ``namespace'' in memcached parlance. 
Do we want to make this transparent to the end user or add a 
configuration option ?
- memcached has no concept of Lock(), UnLock() and some stuff I haven't 
figured out yet. Add no-ops for these to Cache::Memcached::Tie or is 
there some more elegant way to bypass them inside Apache::ASP ?
- It seems there's no obvious way to enumerate keys in memcached 
(FIRSTKEY  NEXTKEY). Perhaps keep a separate index of keys inside 
memcached and add methods that use those (?)
- memcached needs a few extra configuration options. Most important is 
obviously ``servers'', but also ``compress_threshold'' and ``debug''


---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

Re: using memcached as a StateDB.. getting there

[Thanos Chatziathanassiou <tc...@arx.gr>]

O/H Gregory S. Youngblood έγραψε:
> I, for one, am interested, though I haven't been able to do much or get
> involved.
>
> "Namespace" - I suggest making this an option that is user tunable, but pick
> a good default value that hopefully won't need to be tweaked all the time. 
>   
It would vaguely resemble ``StateDir''. Each separate StateDir signifies 
a unique application.
Thus far, using StateDir in shmfs/tmpfs I'd use
/dev/shm/apache/website1 and /dev/shm/apache/website2 for two separate 
applications. Something similar would apply to using a single memcached 
for multiple applications, no ?
Of course, the default StateDir ``.state'' would mix things up a bit.
> "Lock/Unlock" - Each individual operation of memcached is atomic.. if two
> processes attempt to write to one location at the same time, they will be
> serialized and one will not corrupt the other. The problem is you don't
> necessarily know which one will actually win. This means you can't guarantee
> state if you don't do locking to ensure the right one wins. 
Well, the original point of Lock and UnLock was to avoid corrupting the 
on-disk files, but your point is valid..
> Consider an
> incrementing counter. Assuming you do a get to read the value, increment it
> by one, and then do a set to save it; each individual operation is safe
> (get, set), but not the combination (get+set). If two hit at the same time,
> the number would increment by 1 and not by 2 (one for each). I'd worry that
> making lock and unlock a no-op would create new problems, especially as site
> volume increases and chances of simultaneous updates increase.
>   
Each user gets his own key (session-id in Apache::ASP) to write, apart 
from the generic ``application''.
The case you're describing isn't handled in Apache::ASP already and it's 
probably because the developer (end user in our case) should concern 
himself with that.
But we can work on that one too if you feel so inclined.
> "No list of keys" - Usually I know what keys I'm stuffing into memcacheb and
> I don't need to walk through the keys, or if I do I have a place outside of
> memcache that has the keys to lookup. This one could get tricky. How would
> you keep the list of keys in a second entry intact, especially if two
> processes wanted to add a key at the same time? 
>   
According to perltie, tying hashes pretty much expects ``FIRSTKEY'' and 
``NEXTKEY'' to work. But this is an inadequacy of Cache::Memcached::Tie 
which we may or may not address.
I was (indirectly) asking Josh if these are actually needed for 
Apache::ASP Sessions to work, seeing that they're defined in 
Apache::ASP::Session but never actually called - not from within 
Apache::ASP at least. But someone is bound to have used / wants to use 
keys(%$Session) already..
> Other thoughts/suggestions:
>
> Having not looked at your code or design this may not be applicable, but
> consider making this generic, something that memcache or another cache
> engine could be plugged into. If you're interested, and my time permits, I'd
> be interested in working on part of this with you.
>   
The thing is, Apache::ASP::State is too tightly bound to on-disk dbm 
files and would require a major rewrite to facilitate other storage 
engines. I suspect that's what held Josh from implementing 
Apache::Session storage. Come to think of it, there already is an 
Apache::Session::Memcached thing out there, so perhaps we should focus 
on making Apache::ASP::State work with an Apache::Session and friends 
back-end instead of hacking around.
I'd feel more comfortable if we had help from the original author of 
Apache::ASP for this (or at least his blessing ;)
Let's think this through the weekend and decide on Monday.

Best Regards,
Thanos Chatziathanassiou



---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org

RE: using memcached as a StateDB.. getting there

["Gregory S. Youngblood" <gr...@tcscs.com>]

I, for one, am interested, though I haven't been able to do much or get
involved.

"Namespace" - I suggest making this an option that is user tunable, but pick
a good default value that hopefully won't need to be tweaked all the time. 

"Lock/Unlock" - Each individual operation of memcached is atomic.. if two
processes attempt to write to one location at the same time, they will be
serialized and one will not corrupt the other. The problem is you don't
necessarily know which one will actually win. This means you can't guarantee
state if you don't do locking to ensure the right one wins. Consider an
incrementing counter. Assuming you do a get to read the value, increment it
by one, and then do a set to save it; each individual operation is safe
(get, set), but not the combination (get+set). If two hit at the same time,
the number would increment by 1 and not by 2 (one for each). I'd worry that
making lock and unlock a no-op would create new problems, especially as site
volume increases and chances of simultaneous updates increase.

"No list of keys" - Usually I know what keys I'm stuffing into memcacheb and
I don't need to walk through the keys, or if I do I have a place outside of
memcache that has the keys to lookup. This one could get tricky. How would
you keep the list of keys in a second entry intact, especially if two
processes wanted to add a key at the same time? 

Other thoughts/suggestions:

Having not looked at your code or design this may not be applicable, but
consider making this generic, something that memcache or another cache
engine could be plugged into. If you're interested, and my time permits, I'd
be interested in working on part of this with you.

Greg

> -----Original Message-----
> From: Thanos Chatziathanassiou [mailto:tchatzi@arx.gr]
> Sent: Thursday, March 26, 2009 5:33 AM
> Cc: 'asp@perl.apache.org'
> Subject: Re: using memcached as a StateDB.. getting there
> 
> (ignoring the fact that no-one seems interested)
> 
> I came accross Cache::Memcached::Tie which pretty much does most of the
> work, however I'm having some issues - actually design decisions:
> - I suppose StateDir should equal ``namespace'' in memcached parlance.
> Do we want to make this transparent to the end user or add a
> configuration option ?
> - memcached has no concept of Lock(), UnLock() and some stuff I haven't
> figured out yet. Add no-ops for these to Cache::Memcached::Tie or is
> there some more elegant way to bypass them inside Apache::ASP ?
> - It seems there's no obvious way to enumerate keys in memcached
> (FIRSTKEY  NEXTKEY). Perhaps keep a separate index of keys inside
> memcached and add methods that use those (?)
> - memcached needs a few extra configuration options. Most important is
> obviously ``servers'', but also ``compress_threshold'' and ``debug''
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
> For additional commands, e-mail: asp-help@perl.apache.org
> 
> 
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.0.238 / Virus Database: 270.11.26/2020 - Release Date:
> 03/25/09 07:16:00


---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org