You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kevin Burton <bu...@spinn3r.com> on 2016/01/19 03:52:44 UTC
Using cassandra a BLOB store / web cache.
Internally we have the need for a blob store for web content. It's MOSTLY
key, ,value based but we'd like to have lookups by coarse grained tags.
This needs to store normal web content like HTML , CSS, JPEG, SVG, etc.
Highly doubt that anything over 5MB would need to be stored.
We also need the ability to store older versions of the same URL for
features like "time travel" where we can see what the web looks like over
time.
I initially wrote this for Elasticsearch (and it works well for that) but
it looks like binaries snuck into the set of requirements.
I could Base64 encode/decode them in ES I guess but that seems ugly.
I was thinking of porting this over to CS but I'm not up to date on the
current state of blobs in C*...
Any advice?
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Using cassandra a BLOB store / web cache.
Posted by DuyHai Doan <do...@gmail.com>.
There is also an excellent tutorial video done by Patrick McFadin and Aaron
Morton on the subject of data model for storing images into Cassandra:
http://youtu.be/gk-B75xgFUg
I guess it can be adapted to store binary objects other than images
On Tue, Jan 19, 2016 at 6:37 AM, Jack Krupansky <ja...@gmail.com>
wrote:
> Chunk the blobs and store them in a separate table from the metadata.
>
> Here's an old attempt at a chunked object store, for reference:
> https://github.com/Netflix/astyanax/wiki/Chunked-Object-Store
>
> Picking an appropriate chunk size may be key (or not). Somewhere between
> 8K and 512K, I would guess, but it probably doesn't matter a lot and could
> be tuned and even configured dynamically.
> With a smaller chunk size you have the option of reading lots of small
> chunks with separate requests or as a slice.
>
> It seems like there has been a fair amount of negative sentiment about
> using Cassandra as a blob/object store, but I personally do think it is
> workable to at least some extent.
>
> A lot of my background is with Solr for search, including a little with
> DSE Search.
>
>
> -- Jack Krupansky
>
> On Mon, Jan 18, 2016 at 9:52 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> Internally we have the need for a blob store for web content. It's
>> MOSTLY key, ,value based but we'd like to have lookups by coarse grained
>> tags.
>>
>> This needs to store normal web content like HTML , CSS, JPEG, SVG, etc.
>>
>> Highly doubt that anything over 5MB would need to be stored.
>>
>> We also need the ability to store older versions of the same URL for
>> features like "time travel" where we can see what the web looks like over
>> time.
>>
>> I initially wrote this for Elasticsearch (and it works well for that) but
>> it looks like binaries snuck into the set of requirements.
>>
>> I could Base64 encode/decode them in ES I guess but that seems ugly.
>>
>> I was thinking of porting this over to CS but I'm not up to date on the
>> current state of blobs in C*...
>>
>> Any advice?
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>>
>>
>
Re: Using cassandra a BLOB store / web cache.
Posted by Jack Krupansky <ja...@gmail.com>.
Chunk the blobs and store them in a separate table from the metadata.
Here's an old attempt at a chunked object store, for reference:
https://github.com/Netflix/astyanax/wiki/Chunked-Object-Store
Picking an appropriate chunk size may be key (or not). Somewhere between 8K
and 512K, I would guess, but it probably doesn't matter a lot and could be
tuned and even configured dynamically.
With a smaller chunk size you have the option of reading lots of small
chunks with separate requests or as a slice.
It seems like there has been a fair amount of negative sentiment about
using Cassandra as a blob/object store, but I personally do think it is
workable to at least some extent.
A lot of my background is with Solr for search, including a little with DSE
Search.
-- Jack Krupansky
On Mon, Jan 18, 2016 at 9:52 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Internally we have the need for a blob store for web content. It's MOSTLY
> key, ,value based but we'd like to have lookups by coarse grained tags.
>
> This needs to store normal web content like HTML , CSS, JPEG, SVG, etc.
>
> Highly doubt that anything over 5MB would need to be stored.
>
> We also need the ability to store older versions of the same URL for
> features like "time travel" where we can see what the web looks like over
> time.
>
> I initially wrote this for Elasticsearch (and it works well for that) but
> it looks like binaries snuck into the set of requirements.
>
> I could Base64 encode/decode them in ES I guess but that seems ugly.
>
> I was thinking of porting this over to CS but I'm not up to date on the
> current state of blobs in C*...
>
> Any advice?
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
>
Re: Using cassandra a BLOB store / web cache.
Posted by Eric Evans <ee...@wikimedia.org>.
On Mon, Jan 18, 2016 at 8:52 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Internally we have the need for a blob store for web content. It's MOSTLY
> key, ,value based but we'd like to have lookups by coarse grained tags.
>
> This needs to store normal web content like HTML , CSS, JPEG, SVG, etc.
>
> Highly doubt that anything over 5MB would need to be stored.
>
> We also need the ability to store older versions of the same URL for
> features like "time travel" where we can see what the web looks like over
> time.
>
> I initially wrote this for Elasticsearch (and it works well for that) but
> it looks like binaries snuck into the set of requirements.
>
> I could Base64 encode/decode them in ES I guess but that seems ugly.
>
> I was thinking of porting this over to CS but I'm not up to date on the
> current state of blobs in C*...
>
> Any advice?
>
We (Wikimedia Foundation) use Cassandra as a durable cache for HTML (with
history). A simplified version of the schema we use would look something
like:
CREATE TABLE data (
key text,
rev int,
tid timeuuid,
value blob,
PRIMARY KEY (("_domain", key), rev, tid)
)
In our case, a 'rev' represents a normative change to the document (read:
someone made an edit), and the 'tid' attribute allows for some arbitrary
number of HTML representations of that revision (say if for example some
transclusion would alter the final outcome). You could simplify this
further by removing the 'tid' attribute if this doesn't apply to you.
One concern here is the size of blobs. Where exactly the threshold on size
should be is probably debatable, but if you are using G1GC I would be
careful about what large blobs do to humongous allocations. G1 will
allocate anything over 1/2 the region size as humongous, and special-case
the handling of them, so humongous allocations should be the exception and
not the rule. Depending on your heap size and the distribution of blob
sizes, you might be able to get by with overriding the GC's choice of
region size, but if 5MB values are at all common, you'll need 16MB region
sizes, (which probably won't work well without a very large corresponding
max heap size).
Another concern is row width. With a data-model like this, rows will grow
relative to the number of versions stored. If versions are added at a low
rate, that might not pose an issue in practice, if it does though you'll
need to consider a different partitioning strategy.
TL;DR You need to understand what your data will look like. Min and max
value sizes aren't enough, you should have some idea of size distribution,
read/write rates, etc. Understand the implications of your data model.
And then test, test, test.
--
Eric Evans
eevans@wikimedia.org
Re: Using cassandra a BLOB store / web cache.
Posted by Jack Krupansky <ja...@gmail.com>.
I think the requirement was stated that old versions will be kept, which is
consistent with Cassandra and the LSM data model - it would avoid the need
for compactions of the actual chunked blob data.
Throughput mostly comes down to adequately provisioning your cluster.
-- Jack Krupansky
On Wed, Jan 20, 2016 at 1:05 PM, Mohit Anchlia <mo...@gmail.com>
wrote:
> The answer to this questions is very much dependent on the throughput,
> desired latency and access patters (R/W or R/O)? In general what I have
> seen working for high throughput environment is to either use a distributed
> file system like Ceph/Gluster or object store like S3 and keep the pointer
> in the NoSQL database Cassandra, Dynamo etc.NoSQL DBs are mostly log
> structured and require compaction frequently, which for high throughput
> environment proves to be quite devastating.
>
>
> On Wed, Jan 20, 2016 at 9:59 AM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> There's also the 'support' issue.. C* is hard enough as it is... maybe
>> you can bring in another system like ES or HDFS but the more you bring in
>> the more your complexity REALLY goes through the roof.
>>
>> Better to keep things simple.
>>
>> I really like the chunking idea for C*... seems like an easy way to store
>> tons of data.
>>
>> On Tue, Jan 19, 2016 at 4:13 PM, Robert Coli <rc...@eventbrite.com>
>> wrote:
>>
>>> On Tue, Jan 19, 2016 at 2:07 PM, Richard L. Burton III <
>>> mrburton@gmail.com> wrote:
>>>
>>>> I would ask why do this over say HDFS, S3, etc. seems like this problem
>>>> has been solved with other solutions that are specifically designed for
>>>> blob storage?
>>>>
>>>
>>> HDFS's default block size is 64mb. If you are storing objects smaller
>>> than this, that might be bad! It also doesn't have http transport, which
>>> other things do.
>>>
>>> Etc..
>>>
>>> =Rob
>>>
>>>
>>
>>
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>>
>>
>
Re: Using cassandra a BLOB store / web cache.
Posted by Mohit Anchlia <mo...@gmail.com>.
The answer to this questions is very much dependent on the throughput,
desired latency and access patters (R/W or R/O)? In general what I have
seen working for high throughput environment is to either use a distributed
file system like Ceph/Gluster or object store like S3 and keep the pointer
in the NoSQL database Cassandra, Dynamo etc.NoSQL DBs are mostly log
structured and require compaction frequently, which for high throughput
environment proves to be quite devastating.
On Wed, Jan 20, 2016 at 9:59 AM, Kevin Burton <bu...@spinn3r.com> wrote:
> There's also the 'support' issue.. C* is hard enough as it is... maybe you
> can bring in another system like ES or HDFS but the more you bring in the
> more your complexity REALLY goes through the roof.
>
> Better to keep things simple.
>
> I really like the chunking idea for C*... seems like an easy way to store
> tons of data.
>
> On Tue, Jan 19, 2016 at 4:13 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Tue, Jan 19, 2016 at 2:07 PM, Richard L. Burton III <
>> mrburton@gmail.com> wrote:
>>
>>> I would ask why do this over say HDFS, S3, etc. seems like this problem
>>> has been solved with other solutions that are specifically designed for
>>> blob storage?
>>>
>>
>> HDFS's default block size is 64mb. If you are storing objects smaller
>> than this, that might be bad! It also doesn't have http transport, which
>> other things do.
>>
>> Etc..
>>
>> =Rob
>>
>>
>
>
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
>
Re: Using cassandra a BLOB store / web cache.
Posted by Kevin Burton <bu...@spinn3r.com>.
There's also the 'support' issue.. C* is hard enough as it is... maybe you
can bring in another system like ES or HDFS but the more you bring in the
more your complexity REALLY goes through the roof.
Better to keep things simple.
I really like the chunking idea for C*... seems like an easy way to store
tons of data.
On Tue, Jan 19, 2016 at 4:13 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, Jan 19, 2016 at 2:07 PM, Richard L. Burton III <mrburton@gmail.com
> > wrote:
>
>> I would ask why do this over say HDFS, S3, etc. seems like this problem
>> has been solved with other solutions that are specifically designed for
>> blob storage?
>>
>
> HDFS's default block size is 64mb. If you are storing objects smaller than
> this, that might be bad! It also doesn't have http transport, which other
> things do.
>
> Etc..
>
> =Rob
>
>
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Using cassandra a BLOB store / web cache.
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Jan 19, 2016 at 2:07 PM, Richard L. Burton III <mr...@gmail.com>
wrote:
> I would ask why do this over say HDFS, S3, etc. seems like this problem
> has been solved with other solutions that are specifically designed for
> blob storage?
>
HDFS's default block size is 64mb. If you are storing objects smaller than
this, that might be bad! It also doesn't have http transport, which other
things do.
Etc..
=Rob
Re: Using cassandra a BLOB store / web cache.
Posted by Steve Robenalt <sr...@highwire.org>.
Just adding one more item to the discussion. I believe this was announced
on the list some time ago. I haven't tried it out yet, just pointing it out
since it's on the OP's topic:
http://pithos.io/
It's a Cassandra-backed object store using an S3 API.
On Tue, Jan 19, 2016 at 2:07 PM, Richard L. Burton III <mr...@gmail.com>
wrote:
> I would ask why do this over say HDFS, S3, etc. seems like this problem
> has been solved with other solutions that are specifically designed for
> blob storage?
>
> On Tue, Jan 19, 2016 at 4:23 PM, <li...@airstreamcomm.net> wrote:
>
>> I recently started noodling with this concept and built a working blob
>> storage service using node.js and C*. I setup a basic web server using the
>> express web server where you could POST binary files to the server where
>> they would get chunked and assigned to a user and bucket, in the spirit of
>> S3. Then when you retrieved the file with a GET it would reassemble the
>> chunks and push it out. What was really nice about node.js is I could
>> retrieve all the chunks in parallel asynchronously. It was just for fun,
>> but we have considered fleshing it out for use in our organization. Here
>> is the schema I developed:
>>
>> CREATE TABLE object.users (
>> name text PRIMARY KEY,
>> buckets set<text>,
>> password text
>> );
>>
>> CREATE TABLE object.objects (
>> user text,
>> bucket text,
>> name text,
>> chunks int,
>> id uuid,
>> size int,
>> type text,
>> PRIMARY KEY ((user, bucket), name)
>> );
>>
>> CREATE TABLE object.chunks (
>> file_id uuid,
>> num int,
>> data blob,
>> PRIMARY KEY (file_id, num)
>> );
>>
>> For your purposes you could modify the objects table to keep a revision
>> id or timeuuid for looking at previous versions. If you want some insight
>> into how the code worked give me a shout.
>>
>
>
>
> --
> -Richard L. Burton III
> @rburton
>
--
Steve Robenalt
Software Architect
srobenalt@highwire.org <bz...@highwire.org>
(office/cell): 916-505-1785
HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org
Technology for Scholarly Communication
Re: Using cassandra a BLOB store / web cache.
Posted by "Richard L. Burton III" <mr...@gmail.com>.
I would ask why do this over say HDFS, S3, etc. seems like this problem has
been solved with other solutions that are specifically designed for blob
storage?
On Tue, Jan 19, 2016 at 4:23 PM, <li...@airstreamcomm.net> wrote:
> I recently started noodling with this concept and built a working blob
> storage service using node.js and C*. I setup a basic web server using the
> express web server where you could POST binary files to the server where
> they would get chunked and assigned to a user and bucket, in the spirit of
> S3. Then when you retrieved the file with a GET it would reassemble the
> chunks and push it out. What was really nice about node.js is I could
> retrieve all the chunks in parallel asynchronously. It was just for fun,
> but we have considered fleshing it out for use in our organization. Here
> is the schema I developed:
>
> CREATE TABLE object.users (
> name text PRIMARY KEY,
> buckets set<text>,
> password text
> );
>
> CREATE TABLE object.objects (
> user text,
> bucket text,
> name text,
> chunks int,
> id uuid,
> size int,
> type text,
> PRIMARY KEY ((user, bucket), name)
> );
>
> CREATE TABLE object.chunks (
> file_id uuid,
> num int,
> data blob,
> PRIMARY KEY (file_id, num)
> );
>
> For your purposes you could modify the objects table to keep a revision id
> or timeuuid for looking at previous versions. If you want some insight
> into how the code worked give me a shout.
>
--
-Richard L. Burton III
@rburton
Re: Using cassandra a BLOB store / web cache.
Posted by li...@airstreamcomm.net.
I recently started noodling with this concept and built a working blob storage service using node.js and C*. I setup a basic web server using the express web server where you could POST binary files to the server where they would get chunked and assigned to a user and bucket, in the spirit of S3. Then when you retrieved the file with a GET it would reassemble the chunks and push it out. What was really nice about node.js is I could retrieve all the chunks in parallel asynchronously. It was just for fun, but we have considered fleshing it out for use in our organization. Here is the schema I developed:
CREATE TABLE object.users (
name text PRIMARY KEY,
buckets set<text>,
password text
);
CREATE TABLE object.objects (
user text,
bucket text,
name text,
chunks int,
id uuid,
size int,
type text,
PRIMARY KEY ((user, bucket), name)
);
CREATE TABLE object.chunks (
file_id uuid,
num int,
data blob,
PRIMARY KEY (file_id, num)
);
For your purposes you could modify the objects table to keep a revision id or timeuuid for looking at previous versions. If you want some insight into how the code worked give me a shout.
Re: Using cassandra a BLOB store / web cache.
Posted by Kevin Burton <bu...@spinn3r.com>.
Lots of interesting feedback.. I like the ideal of chunking the IO into
pages.. it would require more thinking but I could even do cassandra async
IO and async HTTP to serve the data and then use HTTP chunks for each
range.
On Tue, Jan 19, 2016 at 10:47 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Jan 18, 2016 at 6:52 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> Internally we have the need for a blob store for web content. It's
>> MOSTLY key, ,value based but we'd like to have lookups by coarse grained
>> tags.
>>
>
> I know you know how to operate and scale MySQL, so I suggest MogileFS for
> the actual blob storage :
>
> https://github.com/mogilefs
>
> Then do some simple indexing in some search store. Done.
>
> =Rob
>
>
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: Using cassandra a BLOB store / web cache.
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Jan 18, 2016 at 6:52 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Internally we have the need for a blob store for web content. It's MOSTLY
> key, ,value based but we'd like to have lookups by coarse grained tags.
>
I know you know how to operate and scale MySQL, so I suggest MogileFS for
the actual blob storage :
https://github.com/mogilefs
Then do some simple indexing in some search store. Done.
=Rob