Blob storage

["Buttler, David" <bu...@llnl.gov>]

Hi all,

I have just had a new requirement to store blob data.  I recall seeing this discussed in the past, but my search only turned up references to Lily and the hbase hackathon: http://search-hadoop.com/m/jVqj5aagW62/blob+storage&subj=Hackathon+notes+12+13+2010

Has someone implemented a complementary blob storage mechanism, or is this something still on the roadmap?

The Lily blob storage looks very interesting, but also tightly integrated into their CMS

(if the Lily guys are listening: would you like to reconsider your choice of solr over elastic search?  The solr sharding you describe seems a bit painful)

Dave

Re: Blob storage

[Jean-Daniel Cryans <jd...@apache.org>]

Yeah there's definitely something better we could do there, see "Too
easy to OOME a RS" https://issues.apache.org/jira/browse/HBASE-2506

J-D

On Wed, Mar 9, 2011 at 11:09 AM, Chris Tarnas <cf...@email.com> wrote:
> When I get a chance to catch my breath I'll see about writing up something on our experiences. One thing I will say - don't skimp on the nodes, you do not want to run out of RAM when using the large values. When running my dev environment in pseudo distributed mode on a laptop the system can have trouble (nothing that is not recoverable though) when the regionserver runs out of memory dealing with the large value.
>
> -chris
>
>
>
>
> On Mar 8, 2011, at 1:54 PM, Jean-Daniel Cryans wrote:
>
>> That's pretty good stuff Chris! You know, you could be my new BFF if
>> you wrote a blog post about your current HBase setup, experiences, etc
>> :)
>>
>> J-D
>>
>> On Tue, Mar 8, 2011 at 11:25 AM, Chris Tarnas <cf...@email.com> wrote:
>>> Yes, HBASE-3483 fixed the majority of our pauses, but not all - as JD points out we do experience issues related to inserting into several column families. Luckily inserts that have the really imbalanced column family sizes (mb vs kb) are few and far between, relatively speaking. We are also "throttled" by going through thrift, but even then I can push our 10 node cluster to over 200k requests a second.
>>>
>>> -chris
>>>
>>> On Mar 8, 2011, at 11:16 AM, Ryan Rawson wrote:
>>>
>>>> Probably the soft limit flushes, eh?
>>>> On Mar 8, 2011 11:15 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>>>>> On Tue, Mar 8, 2011 at 11:04 AM, Chris Tarnas <cf...@email.com> wrote:
>>>>>> Just as a point of reference, in one of our systems we have 500+million
>>>> rows that have a cell in its own column family that is about usually about
>>>> 100bytes, but in about 10,000 of rows the cell can get to 300mb (average is
>>>> probably about 30mb for the larger data). The jumbo sized data gets loaded
>>>> in separately from the smaller data, although it all goes through the same
>>>> pipeline. We are using cdh3b45 (0.90.1) GZ compression, region size of 1GB
>>>> and with a max value size of 500mb. So far we have had no problems with the
>>>> larger values.
>>>>>>
>>>>>> Our largest problem was performance related to inserting into several
>>>> column families for the small sized value loads and pauses when flushing the
>>>> memstores. 0.90.1 helped quite a bit with that.
>>>>>
>>>>> Flushing is done without blocking, were the pauses you were seeing
>>>>> related to the "too many stores" issue or about the global memstore
>>>>> size?
>>>>>
>>>>> In general inserting into many families is a bad idea unless the sizes
>>>>> are the same. The worst case is inserting a few kbs in one and a few
>>>>> mbs in the other. The reason being:
>>>>> https://issues.apache.org/jira/browse/HBASE-3149
>>>>>
>>>>> J-D
>>>
>>>
>
>

Re: Blob storage

[Chris Tarnas <cf...@email.com>]

When I get a chance to catch my breath I'll see about writing up something on our experiences. One thing I will say - don't skimp on the nodes, you do not want to run out of RAM when using the large values. When running my dev environment in pseudo distributed mode on a laptop the system can have trouble (nothing that is not recoverable though) when the regionserver runs out of memory dealing with the large value.

-chris




On Mar 8, 2011, at 1:54 PM, Jean-Daniel Cryans wrote:

> That's pretty good stuff Chris! You know, you could be my new BFF if
> you wrote a blog post about your current HBase setup, experiences, etc
> :)
> 
> J-D
> 
> On Tue, Mar 8, 2011 at 11:25 AM, Chris Tarnas <cf...@email.com> wrote:
>> Yes, HBASE-3483 fixed the majority of our pauses, but not all - as JD points out we do experience issues related to inserting into several column families. Luckily inserts that have the really imbalanced column family sizes (mb vs kb) are few and far between, relatively speaking. We are also "throttled" by going through thrift, but even then I can push our 10 node cluster to over 200k requests a second.
>> 
>> -chris
>> 
>> On Mar 8, 2011, at 11:16 AM, Ryan Rawson wrote:
>> 
>>> Probably the soft limit flushes, eh?
>>> On Mar 8, 2011 11:15 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>>>> On Tue, Mar 8, 2011 at 11:04 AM, Chris Tarnas <cf...@email.com> wrote:
>>>>> Just as a point of reference, in one of our systems we have 500+million
>>> rows that have a cell in its own column family that is about usually about
>>> 100bytes, but in about 10,000 of rows the cell can get to 300mb (average is
>>> probably about 30mb for the larger data). The jumbo sized data gets loaded
>>> in separately from the smaller data, although it all goes through the same
>>> pipeline. We are using cdh3b45 (0.90.1) GZ compression, region size of 1GB
>>> and with a max value size of 500mb. So far we have had no problems with the
>>> larger values.
>>>>> 
>>>>> Our largest problem was performance related to inserting into several
>>> column families for the small sized value loads and pauses when flushing the
>>> memstores. 0.90.1 helped quite a bit with that.
>>>> 
>>>> Flushing is done without blocking, were the pauses you were seeing
>>>> related to the "too many stores" issue or about the global memstore
>>>> size?
>>>> 
>>>> In general inserting into many families is a bad idea unless the sizes
>>>> are the same. The worst case is inserting a few kbs in one and a few
>>>> mbs in the other. The reason being:
>>>> https://issues.apache.org/jira/browse/HBASE-3149
>>>> 
>>>> J-D
>> 
>> 

Re: Blob storage

[Jean-Daniel Cryans <jd...@apache.org>]

That's pretty good stuff Chris! You know, you could be my new BFF if
you wrote a blog post about your current HBase setup, experiences, etc
:)

J-D

On Tue, Mar 8, 2011 at 11:25 AM, Chris Tarnas <cf...@email.com> wrote:
> Yes, HBASE-3483 fixed the majority of our pauses, but not all - as JD points out we do experience issues related to inserting into several column families. Luckily inserts that have the really imbalanced column family sizes (mb vs kb) are few and far between, relatively speaking. We are also "throttled" by going through thrift, but even then I can push our 10 node cluster to over 200k requests a second.
>
> -chris
>
> On Mar 8, 2011, at 11:16 AM, Ryan Rawson wrote:
>
>> Probably the soft limit flushes, eh?
>> On Mar 8, 2011 11:15 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>>> On Tue, Mar 8, 2011 at 11:04 AM, Chris Tarnas <cf...@email.com> wrote:
>>>> Just as a point of reference, in one of our systems we have 500+million
>> rows that have a cell in its own column family that is about usually about
>> 100bytes, but in about 10,000 of rows the cell can get to 300mb (average is
>> probably about 30mb for the larger data). The jumbo sized data gets loaded
>> in separately from the smaller data, although it all goes through the same
>> pipeline. We are using cdh3b45 (0.90.1) GZ compression, region size of 1GB
>> and with a max value size of 500mb. So far we have had no problems with the
>> larger values.
>>>>
>>>> Our largest problem was performance related to inserting into several
>> column families for the small sized value loads and pauses when flushing the
>> memstores. 0.90.1 helped quite a bit with that.
>>>
>>> Flushing is done without blocking, were the pauses you were seeing
>>> related to the "too many stores" issue or about the global memstore
>>> size?
>>>
>>> In general inserting into many families is a bad idea unless the sizes
>>> are the same. The worst case is inserting a few kbs in one and a few
>>> mbs in the other. The reason being:
>>> https://issues.apache.org/jira/browse/HBASE-3149
>>>
>>> J-D
>
>

Re: Blob storage

[Chris Tarnas <cf...@email.com>]

Yes, HBASE-3483 fixed the majority of our pauses, but not all - as JD points out we do experience issues related to inserting into several column families. Luckily inserts that have the really imbalanced column family sizes (mb vs kb) are few and far between, relatively speaking. We are also "throttled" by going through thrift, but even then I can push our 10 node cluster to over 200k requests a second.

-chris
 
On Mar 8, 2011, at 11:16 AM, Ryan Rawson wrote:

> Probably the soft limit flushes, eh?
> On Mar 8, 2011 11:15 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
>> On Tue, Mar 8, 2011 at 11:04 AM, Chris Tarnas <cf...@email.com> wrote:
>>> Just as a point of reference, in one of our systems we have 500+million
> rows that have a cell in its own column family that is about usually about
> 100bytes, but in about 10,000 of rows the cell can get to 300mb (average is
> probably about 30mb for the larger data). The jumbo sized data gets loaded
> in separately from the smaller data, although it all goes through the same
> pipeline. We are using cdh3b45 (0.90.1) GZ compression, region size of 1GB
> and with a max value size of 500mb. So far we have had no problems with the
> larger values.
>>> 
>>> Our largest problem was performance related to inserting into several
> column families for the small sized value loads and pauses when flushing the
> memstores. 0.90.1 helped quite a bit with that.
>> 
>> Flushing is done without blocking, were the pauses you were seeing
>> related to the "too many stores" issue or about the global memstore
>> size?
>> 
>> In general inserting into many families is a bad idea unless the sizes
>> are the same. The worst case is inserting a few kbs in one and a few
>> mbs in the other. The reason being:
>> https://issues.apache.org/jira/browse/HBASE-3149
>> 
>> J-D

Re: Blob storage

[Ryan Rawson <ry...@gmail.com>]

Probably the soft limit flushes, eh?
On Mar 8, 2011 11:15 AM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
> On Tue, Mar 8, 2011 at 11:04 AM, Chris Tarnas <cf...@email.com> wrote:
>> Just as a point of reference, in one of our systems we have 500+million
rows that have a cell in its own column family that is about usually about
100bytes, but in about 10,000 of rows the cell can get to 300mb (average is
probably about 30mb for the larger data). The jumbo sized data gets loaded
in separately from the smaller data, although it all goes through the same
pipeline. We are using cdh3b45 (0.90.1) GZ compression, region size of 1GB
and with a max value size of 500mb. So far we have had no problems with the
larger values.
>>
>> Our largest problem was performance related to inserting into several
column families for the small sized value loads and pauses when flushing the
memstores. 0.90.1 helped quite a bit with that.
>
> Flushing is done without blocking, were the pauses you were seeing
> related to the "too many stores" issue or about the global memstore
> size?
>
> In general inserting into many families is a bad idea unless the sizes
> are the same. The worst case is inserting a few kbs in one and a few
> mbs in the other. The reason being:
> https://issues.apache.org/jira/browse/HBASE-3149
>
> J-D

Re: Blob storage

[Jean-Daniel Cryans <jd...@apache.org>]

On Tue, Mar 8, 2011 at 11:04 AM, Chris Tarnas <cf...@email.com> wrote:
> Just as a point of reference, in one of our systems we have 500+million rows that have a cell in its own column family that is about usually about 100bytes, but in about 10,000 of rows the cell can get to 300mb (average is probably about 30mb for the larger data). The jumbo sized data gets loaded in separately from the smaller data, although it all goes through the same pipeline. We are using cdh3b45 (0.90.1) GZ compression, region size of 1GB and with a max value size of 500mb. So far we have had no problems with the larger values.
>
> Our largest problem was performance related to inserting into several column families for the small sized value loads and pauses when flushing the memstores. 0.90.1 helped quite a bit with that.

Flushing is done without blocking, were the pauses you were seeing
related to the "too many stores" issue or about the global memstore
size?

In general inserting into many families is a bad idea unless the sizes
are the same. The worst case is inserting a few kbs in one and a few
mbs in the other. The reason being:
https://issues.apache.org/jira/browse/HBASE-3149

J-D

Re: Blob storage

[Chris Tarnas <cf...@email.com>]

Just as a point of reference, in one of our systems we have 500+million rows that have a cell in its own column family that is about usually about 100bytes, but in about 10,000 of rows the cell can get to 300mb (average is probably about 30mb for the larger data). The jumbo sized data gets loaded in separately from the smaller data, although it all goes through the same pipeline. We are using cdh3b45 (0.90.1) GZ compression, region size of 1GB and with a max value size of 500mb. So far we have had no problems with the larger values.

Our largest problem was performance related to inserting into several column families for the small sized value loads and pauses when flushing the memstores. 0.90.1 helped quite a bit with that.

-chris



On Mar 8, 2011, at 10:54 AM, Jean-Daniel Cryans wrote:

>> The blobs vary in size from smallish (10K) to largish (20MB).
> 
> 20MB is quite large, but could be harmless if most of the rows are under 1MB
> 
>> They are too small to put into individual files in HDFS, but if I have too many largish rows in a region, I think I would suffer.
> 
> Yeah, need more info about the size distribution.
> 
>> 
>> Would it be possible to put the blobs in their own column family that has a significantly different block size (10x).  I hesitate to do this mostly because I already have too many column families, but since I don't expect the blobs to be touched very often, a separate column family would make them mostly harmless.
> 
> The block size is dynamic, if you store a single cell of 20MB then
> that will be 1 block of the same size. Instead of creating a new
> family, you could also create a new table.
> 
> J-D

Re: Blob storage

[Jean-Daniel Cryans <jd...@apache.org>]

> The blobs vary in size from smallish (10K) to largish (20MB).

20MB is quite large, but could be harmless if most of the rows are under 1MB

> They are too small to put into individual files in HDFS, but if I have too many largish rows in a region, I think I would suffer.

Yeah, need more info about the size distribution.

>
> Would it be possible to put the blobs in their own column family that has a significantly different block size (10x).  I hesitate to do this mostly because I already have too many column families, but since I don't expect the blobs to be touched very often, a separate column family would make them mostly harmless.

The block size is dynamic, if you store a single cell of 20MB then
that will be 1 block of the same size. Instead of creating a new
family, you could also create a new table.

J-D

RE: Blob storage

["Buttler, David" <bu...@llnl.gov>]

The blobs vary in size from smallish (10K) to largish (20MB).  
They are too small to put into individual files in HDFS, but if I have too many largish rows in a region, I think I would suffer.

Would it be possible to put the blobs in their own column family that has a significantly different block size (10x).  I hesitate to do this mostly because I already have too many column families, but since I don't expect the blobs to be touched very often, a separate column family would make them mostly harmless.

Dave


-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Tuesday, March 08, 2011 10:39 AM
To: user@hbase.apache.org
Subject: Re: Blob storage

How big are those blobs on average? I know of a few people that store
objects in the hundreds of KBs in HBase without too much tuning.

AFAIK the offline blob storage described by Todd wasn't implemented.

J-D

On Tue, Mar 8, 2011 at 10:33 AM, Buttler, David <bu...@llnl.gov> wrote:
> Hi all,
>
> I have just had a new requirement to store blob data.  I recall seeing this discussed in the past, but my search only turned up references to Lily and the hbase hackathon: http://search-hadoop.com/m/jVqj5aagW62/blob+storage&subj=Hackathon+notes+12+13+2010
>
> Has someone implemented a complementary blob storage mechanism, or is this something still on the roadmap?
>
> The Lily blob storage looks very interesting, but also tightly integrated into their CMS
>
> (if the Lily guys are listening: would you like to reconsider your choice of solr over elastic search?  The solr sharding you describe seems a bit painful)
>
> Dave
>
>

Re: Blob storage

[Jean-Daniel Cryans <jd...@apache.org>]

How big are those blobs on average? I know of a few people that store
objects in the hundreds of KBs in HBase without too much tuning.

AFAIK the offline blob storage described by Todd wasn't implemented.

J-D

On Tue, Mar 8, 2011 at 10:33 AM, Buttler, David <bu...@llnl.gov> wrote:
> Hi all,
>
> I have just had a new requirement to store blob data.  I recall seeing this discussed in the past, but my search only turned up references to Lily and the hbase hackathon: http://search-hadoop.com/m/jVqj5aagW62/blob+storage&subj=Hackathon+notes+12+13+2010
>
> Has someone implemented a complementary blob storage mechanism, or is this something still on the roadmap?
>
> The Lily blob storage looks very interesting, but also tightly integrated into their CMS
>
> (if the Lily guys are listening: would you like to reconsider your choice of solr over elastic search?  The solr sharding you describe seems a bit painful)
>
> Dave
>
>

Re: Blob storage

[Steven Noels <st...@outerthought.org>]

On Tue, Mar 8, 2011 at 7:33 PM, Buttler, David <bu...@llnl.gov> wrote:

Has someone implemented a complementary blob storage mechanism, or is this
> something still on the roadmap?
>


Well, yes. :-)



> The Lily blob storage looks very interesting, but also tightly integrated
> into their CMS
>


We prefer "data repository" but I catch your drift. We support blob storage
inline in the Lily record table, in a separate blob table, and in HDFS. The
client API also connects directly to HBase or HDFS for retrieving blobs for
reasons of efficiency.



> (if the Lily guys are listening:


We typically do, but are pretty busy with our 1.0 release.


> would you like to reconsider your choice of solr over elastic search?


Reconsider: no, but yes, we have 'somewhere along the roadmap' the intention
to support ES alongside SOLR. For our enterprise users, SOLR fits the bill
quite nicely. When running distributed, you can have heavy index calculation
nodes and only the computed index segments are then replicated to slaves,
whileas ES requires all nodes to be identical (AFAIU) because indexes are
computed locally (Shay: correct me if I'm wrong).


>  The solr sharding you describe seems a bit painful)
>

Again, we haven't encountered issues, yet. Resharding is painful, but we
provide index mgmt around that. Again, ES support is planned for later this
year, but I cannot confirm a hard date yet.

Kind regards,
-- 
Steven Noels
http://outerthought.org/
Scalable Smart Data
Makers of Kauri, Daisy CMS and Lily