You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2013/03/21 01:39:20 UTC
[OT] Sharing lots of little pieces of data across a cluster
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
I have an in-process "service" that stores valid nonces on a server
for a particular set of client operations. The nonces are created
once, then expire after a certain amount of time. They never change.
I'd like to make this in-process service into an out-of-process
service that can be accessed by any node in my cluster, basically
acting like a communal hash map.
Memcached is the perfect application for this kind of thing, right? It
is fast and simple, plus supports key expiration out of the box.
Doing a bit of reading (I've never actually used memcached before), it
seems like memcached is better-suited as a /cache/ -- that is,
something that sits between a slow data source and clients. They
suggest that you /not/ configure "failover" but instead allow a dying
node in your memcached cluster to simply die and consider the data
lost: go back to the canonical data source and re-fetch the data. In
my case, I have no (other) canonical data source: I just want to use
memcached.
(Note that if the whole service were to fall-over and I had to restart
the nonce-storage cluster and start with a completely empty
"database", it wouldn't be the end of the world. There would be a lot
of grumbling, because everyone would have to request new nonces and
re-start any transactions that were using them.)
Also, the memcached servers don't really know about each other, right?
So, it's not really a big, shared hashtable. Instead, it's like a
bunch of separate hash tables and the client knows which server ought
to have the data when it requests it based upon the key.
Am I barking up the wrong tree by looking at memcached? Is there
something else that would be better for me? It's a simple enough set
of requirements that writing it myself could be done easily. Then
again, it's a simple enough set of requirements that someone /must/
have done this before me.
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJRSla3AAoJEBzwKT+lPKRYGpsP+wW0kKEjm5k0p2cFrRDF9KAr
y5SWi3GWCPSNRbOJkR487CRfFN9cUCuyiq1+wuQrtbhG2osdeLUIb4dS7NOCTRh8
dknyZvGQw9BrBZXvUeVQnMsLrD02YE6qgEp8hAdnLZBoKLb8EOA1FACs2qdWaBW6
8XLxI6nw5yT/y6glP35syq/MfgjFsXdn8+2Wlu5KQdc6YciUMrG/L7ifB4Huxz+S
NFqVCsXJeVQU6MGpL1Bucn135WE3dHrZWJlnnP38iq2cATzo+0SM6Yq4ul2APjye
EoN252a5WXddEhzMyjRKC8U89XE8ELF44WiP9NN3niEyyHh035+iK3dawhpN40qi
XUw87TbnbL/4cAk8wu2d+gD3BHAFl9SrmkAcJ8lPKpn+ExSzFcgXwldc/TJah+yh
M8FOGNwF4FfOeVCkAEMEltsk9YJev8IQNPSAYKtI0tzzWtckmq0ujHQpBWd/2knw
YtSQALT6cjx0oyJnRTG02Jx1I6OsOQgVaHcarb8ejAAqkK3I0pge1tNroMU2QaHD
gNY5wl33msB5WpHRYlsmg0y+6bshjrOrJkpPoYpxuVgGZj5zrPls5WhuFk1zgZ6Z
e7j1BMMF/xVykNKS/bO1T6hqGsjCFSXCfT7WURbkkQyQ85Yiyvcmcfrt2tnVFX0x
dtB4zpCWED0XLw1CbG88
=eJjT
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Sharing lots of little pieces of data across a cluster
Posted by "Howard W. Smith, Jr." <sm...@gmail.com>.
On Thu, Mar 21, 2013 at 8:54 AM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> > For more complex data structures, and more ability in queries, i
> > would suggest http://www.mongodb.org/
>
> Ooh! Web-scale!
>
> > They have json-like storage, expiry is also supported, scaling
> > horizontally with sharding, support replicas for availability, and
> > although it's always in persistent mode, the active workset would
> > be in the mapped-memory for fast access.
>
> One of the major draws of Memecached is its near-zero configuration:
> just launch it and you've got an instance. I want to avoid a
> full-blown "database" if possible.
>
>
+1 (on all the above)
Re: [OT] Sharing lots of little pieces of data across a cluster
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Albert,
On 3/20/13 9:22 PM, Albert Kam wrote:
> I am not sure this might suit you, but if i am currently thinking
> of http://redis.io/ Key and value storage (binary value is
> supported), expiry is supported, support scaling horizontally, can
> be set to be non-persistent (only in memory, which is fast)
Okay, I'll check that out.
> For more complex data structures, and more ability in queries, i
> would suggest http://www.mongodb.org/
Ooh! Web-scale!
> They have json-like storage, expiry is also supported, scaling
> horizontally with sharding, support replicas for availability, and
> although it's always in persistent mode, the active workset would
> be in the mapped-memory for fast access.
One of the major draws of Memecached is its near-zero configuration:
just launch it and you've got an instance. I want to avoid a
full-blown "database" if possible.
Thanks for the thoughts: I'll give them (both) a good look.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJRSwL6AAoJEBzwKT+lPKRYabsP/issbYZC8p1vxb0Uq8AoXz4R
mAdCbnOwU17TKUG4B6LrFHGYX1/fjVPwyEjWpmCbwUB5K4+9XIjX0LgzuTEgQx1m
mCjMSReB0iQQ53sAv1feyuOGIQ4e1dTsY0vDnIBKnt7CwyiN/EyCg/mAqHc8Tump
KUetnNWrJfn23mVQUfZGplLinzbbaRtdx1+tv4np+srezUtH/wwEgfPJEgwVWtoK
2w8a/jtrAaNmDFdvkSJ1mPVHei2IVru+ITyU+pbSX3IogHCqg8mnpvENqF+NT079
sVuKPT1oMMFgE3XQDNKySgrNDso8P+E4SowpSURDogeW6VciGuupkVw8OYVqQ0N/
2UlMoOHVAINSmBAdwgjy0u/8J7bAptz2y70IoFlofwTGm1RVZ2XNQPSilY2kngQO
ao6S+jWzsvoTdMdJAvFsYYsMHdRVFnlSz9qnqrDANiNsMpHIMEnPG2bhONMAjKjM
JHWRkPahkSyK3OjIqil8mgJ88DgLhFtzR+60Vfrg2X8nOrJDkDe4gP/pjfQPzQcp
w3ttCSmz0H0/vsYcWvOsrby/iw3/EFmjpC34r8lEQ10N3t1weDQm9wMluRP7Nkjd
Qg9xkoGJsoiajKbBYIlfFJ7K85WHFEZS89y+4FM/uMgOz5xVrL9s58L5ugT/x5XP
yVM/v9ZiWOuzjwmEobDF
=nYDY
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Sharing lots of little pieces of data across a cluster
Posted by Albert Kam <mo...@gmail.com>.
I am not sure this might suit you, but if i am currently thinking of
http://redis.io/
Key and value storage (binary value is supported), expiry is supported,
support scaling horizontally, can be set to be non-persistent (only in
memory, which is fast)
For more complex data structures, and more ability in queries, i would
suggest http://www.mongodb.org/
They have json-like storage, expiry is also supported, scaling horizontally
with sharding, support replicas for availability, and although it's always
in persistent mode, the active workset would be in the mapped-memory for
fast access.
On Thu, Mar 21, 2013 at 7:39 AM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> All,
>
> I have an in-process "service" that stores valid nonces on a server
> for a particular set of client operations. The nonces are created
> once, then expire after a certain amount of time. They never change.
> I'd like to make this in-process service into an out-of-process
> service that can be accessed by any node in my cluster, basically
> acting like a communal hash map.
>
> Memcached is the perfect application for this kind of thing, right? It
> is fast and simple, plus supports key expiration out of the box.
>
> Doing a bit of reading (I've never actually used memcached before), it
> seems like memcached is better-suited as a /cache/ -- that is,
> something that sits between a slow data source and clients. They
> suggest that you /not/ configure "failover" but instead allow a dying
> node in your memcached cluster to simply die and consider the data
> lost: go back to the canonical data source and re-fetch the data. In
> my case, I have no (other) canonical data source: I just want to use
> memcached.
>
> (Note that if the whole service were to fall-over and I had to restart
> the nonce-storage cluster and start with a completely empty
> "database", it wouldn't be the end of the world. There would be a lot
> of grumbling, because everyone would have to request new nonces and
> re-start any transactions that were using them.)
>
> Also, the memcached servers don't really know about each other, right?
> So, it's not really a big, shared hashtable. Instead, it's like a
> bunch of separate hash tables and the client knows which server ought
> to have the data when it requests it based upon the key.
>
> Am I barking up the wrong tree by looking at memcached? Is there
> something else that would be better for me? It's a simple enough set
> of requirements that writing it myself could be done easily. Then
> again, it's a simple enough set of requirements that someone /must/
> have done this before me.
>
> Thanks,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJRSla3AAoJEBzwKT+lPKRYGpsP+wW0kKEjm5k0p2cFrRDF9KAr
> y5SWi3GWCPSNRbOJkR487CRfFN9cUCuyiq1+wuQrtbhG2osdeLUIb4dS7NOCTRh8
> dknyZvGQw9BrBZXvUeVQnMsLrD02YE6qgEp8hAdnLZBoKLb8EOA1FACs2qdWaBW6
> 8XLxI6nw5yT/y6glP35syq/MfgjFsXdn8+2Wlu5KQdc6YciUMrG/L7ifB4Huxz+S
> NFqVCsXJeVQU6MGpL1Bucn135WE3dHrZWJlnnP38iq2cATzo+0SM6Yq4ul2APjye
> EoN252a5WXddEhzMyjRKC8U89XE8ELF44WiP9NN3niEyyHh035+iK3dawhpN40qi
> XUw87TbnbL/4cAk8wu2d+gD3BHAFl9SrmkAcJ8lPKpn+ExSzFcgXwldc/TJah+yh
> M8FOGNwF4FfOeVCkAEMEltsk9YJev8IQNPSAYKtI0tzzWtckmq0ujHQpBWd/2knw
> YtSQALT6cjx0oyJnRTG02Jx1I6OsOQgVaHcarb8ejAAqkK3I0pge1tNroMU2QaHD
> gNY5wl33msB5WpHRYlsmg0y+6bshjrOrJkpPoYpxuVgGZj5zrPls5WhuFk1zgZ6Z
> e7j1BMMF/xVykNKS/bO1T6hqGsjCFSXCfT7WURbkkQyQ85Yiyvcmcfrt2tnVFX0x
> dtB4zpCWED0XLw1CbG88
> =eJjT
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Do not pursue the past. Do not lose yourself in the future.
The past no longer is. The future has not yet come.
Looking deeply at life as it is in the very here and now,
the practitioner dwells in stability and freedom.
(Thich Nhat Hanh)