Replacing Redis

[Benson Margulies <bi...@gmail.com>]

I'm about to launch off on replacing redis with cassandra. I wonder if
anyone else has ever been there and done that.

Re: Replacing Redis

[David Strauss <da...@davidstrauss.net>]

On Sun, 2011-02-20 at 09:04 -0500, Benson Margulies wrote:
> There are two parts to the data footprint, one growing more rapidly
> than the other. The faster growing part has a natural 'redis-friendly'
> data model (keys map to values); but it would map to EHCache as well.

If you're coming into the non-relational world with the perspective that
the main consideration is your "data model," you're approaching the
problem from the wrong direction. The non-relational world isn't about
BCNF or foreign keys. Storing and retrieving individual items is the
easy part; we need to understand how you intend to *use* the data to
help you understand if Cassandra is a good fit.

      * How do you add new data?
      * Other than accessing individual items by their primary key (or
        equivalent), how do you read it?
      * What needs to be real-time? Asynchronous?
      * What correctness or consistency constraints can you relax?
      * What availability needs do you have?
      * What programming languages are you working in?

> However, the initial redis/jredis implementation used redis, rather
> than local ordinary JVM objects, for the slower growing part. The
> lpush/ltrim usage was in here.

It's not clear just from your use of lpush/ltrim what you're doing. Some
uses can map to Cassandra's APIs with some creative reinterpretation;
others can't.

-- 
David Strauss
   | david@davidstrauss.net
   | +1 512 577 5827 [mobile]

Re: Replacing Redis

[Benson Margulies <bi...@gmail.com>]

Thanks for replying, but it is not at all clear to me that redis was
ever the right solution to the problem at hand. I'm still working out
whether, in fact, Cassandra is.

Up to a fairly large scale, this particular problem needs no 'cache'
at all. It needs a write-behind persistence mechanism -- a log. The
simplest form of which is, of course, just a file, though that
solution poses problems for backup and some failure scenarios.

There are two parts to the data footprint, one growing more rapidly
than the other. The faster growing part has a natural 'redis-friendly'
data model (keys map to values); but it would map to EHCache as well.
However, the initial redis/jredis implementation used redis, rather
than local ordinary JVM objects, for the slower growing part. The
lpush/ltrim usage was in here.

At really gigantic scales, which I don't claim to have entirely
thought through, even the slow-growing part will stop fitting into
memory. However, it has a natural architecture for distribution, so
there is a dilemma in my head between following the logic of the
algorithm out to implement distribution, or letting something like
cassandra do that for me.

Re: Replacing Redis

[Gaurav Sharma <ga...@gmail.com>]

Try Jedis or better-still JOhm if you would like to have automatic object
mapping to Redis. You can select a connection pool of your choice even
though we recommend you use Apache Commons Pool.

Cassandra and Redis are very different beasts and work best at their
respective scales. If you just want a simple cache, Redis can do the job -
it is really fast and operations are atomic and with AOF, data-loss window
can be reduced to max transactions occurring in 1 second :)

All that being said, why are you doing an lpush (list-op) on your KV pairs
and not hset (with pipelining if possible)?

Also, try the v2.2 by directly building off of the redis main branch. It is
much faster than v2.0 and Jedis/JOhm are compatible with 2.2


On Fri, Feb 18, 2011 at 12:00 PM, Benson Margulies <bi...@gmail.com>wrote:

> typical experiment.
>
> Redis 2.0.4 deployed on my macbook pro.
>
> Saves enabled.
>
> appendfsync off.
>
> vm enabled, 1g max memory.
>
> 72 databases. Each database asked to store 13*N key-value pairs with
> lpush, bucket size not very big, N -> 500,000.
>
> Client jredis.
>
> Start running against a stream of inputs. Sooner or later, client
> times out 'last operation not performed'.
>
> These could be jredis bugs, but I don't care to find out.
>

Re: Replacing Redis

[Benson Margulies <bi...@gmail.com>]

typical experiment.

Redis 2.0.4 deployed on my macbook pro.

Saves enabled.

appendfsync off.

vm enabled, 1g max memory.

72 databases. Each database asked to store 13*N key-value pairs with
lpush, bucket size not very big, N -> 500,000.

Client jredis.

Start running against a stream of inputs. Sooner or later, client
times out 'last operation not performed'.

These could be jredis bugs, but I don't care to find out.

Re: Replacing Redis

[Jonathan Shook <js...@gmail.com>]

Benson,
I was considering using Redis for a specific project. Can you
elaborate a bit on your problem with it? What were the circumstances,
loading factors, etc?

On Fri, Feb 18, 2011 at 9:19 AM, Benson Margulies <bi...@gmail.com> wrote:
> redis times out at random regardless of what we configure for client
> timeouts; the platform-sensitive binaries are painful for us since we
> support many platform; just to name two reasons.
>
> On Fri, Feb 18, 2011 at 10:04 AM, Joshua Partogi <jo...@gmail.com> wrote:
>> Any reason why you want to do that?
>>
>> On Sat, Feb 19, 2011 at 1:32 AM, Benson Margulies <bi...@gmail.com> wrote:
>>> I'm about to launch off on replacing redis with cassandra. I wonder if
>>> anyone else has ever been there and done that.
>>>
>>
>>
>>
>> --
>> http://twitter.com/jpartogi
>>
>

Re: Replacing Redis

[Benson Margulies <bi...@gmail.com>]

redis times out at random regardless of what we configure for client
timeouts; the platform-sensitive binaries are painful for us since we
support many platform; just to name two reasons.

On Fri, Feb 18, 2011 at 10:04 AM, Joshua Partogi <jo...@gmail.com> wrote:
> Any reason why you want to do that?
>
> On Sat, Feb 19, 2011 at 1:32 AM, Benson Margulies <bi...@gmail.com> wrote:
>> I'm about to launch off on replacing redis with cassandra. I wonder if
>> anyone else has ever been there and done that.
>>
>
>
>
> --
> http://twitter.com/jpartogi
>

Re: Replacing Redis

[Joshua Partogi <jo...@gmail.com>]

Any reason why you want to do that?

On Sat, Feb 19, 2011 at 1:32 AM, Benson Margulies <bi...@gmail.com> wrote:
> I'm about to launch off on replacing redis with cassandra. I wonder if
> anyone else has ever been there and done that.
>



-- 
http://twitter.com/jpartogi