You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by vishnupriyaa <vp...@gmail.com> on 2012/04/17 15:06:18 UTC
Storing extremely large size file
I want to save a file of size 12MB but an exception occuring like this
KeyValue size too large.
I have set the value of hbase.client.keyvalue.maxsize in hbase-site.xml and
hbase-default.xml to 3GB
but the default value 10MB is taking for hbase.client.keyvalue.maxsize.How
could I change the value of hbase.client.keyvalue.maxsize or how to store
the file of extremely large size.
--
View this message in context: http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33701522.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Storing extremely large size file
Posted by Andrey Stepachev <oc...@gmail.com>.
I think, that HBase should have something like
http://www.mongodb.org/display/DOCS/GridFS+Specification.
On Wed, Apr 18, 2012 at 10:55 AM, kim young ill <kh...@googlemail.com> wrote:
> +1 documentation please
>
> On Wed, Apr 18, 2012 at 7:21 AM, anil gupta <an...@gmail.com> wrote:
>
>> +1 for documentation. It will help a lot of people.
>>
>> On Tue, Apr 17, 2012 at 10:02 PM, lars hofhansl <lh...@yahoo.com>
>> wrote:
>>
>> > I disagree. This comes up frequently and some basic guidelines should be
>> > documented in the Reference Guide.
>> > If it is indeed not difficult than the section is the book will be short.
>> >
>> >
>> >
>> > ----- Original Message -----
>> > From: Michael Segel <mi...@hotmail.com>
>> > To: "user@hbase.apache.org" <us...@hbase.apache.org>
>> > Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
>> > Sent: Tuesday, April 17, 2012 3:43 PM
>> > Subject: Re: Storing extremely large size file
>> >
>> > -1. It's a boring topic.
>> > And it's one of those things that you either get it right or you end up
>> > hiring a voodoo witch doctor to curse the author of the chapter...
>> >
>> > I agree w Jack, it's not difficult just takes some planning and
>> > forethought.
>> >
>> > Also reading lots of blogs... And some practice...
>> >
>> >
>> > Sent from my iPhone
>> >
>> > On Apr 17, 2012, at 1:42 PM, "Dave Revell" <da...@urbanairship.com>
>> wrote:
>> >
>> > > +1 Jack :)
>> > >
>> > > On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
>> > >
>> > >> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
>> > >> wrote:
>> > >>> I think this is a popular topic that might deserve a section in The
>> > Book.
>> > >>>
>> > >>> By "this topic" I mean storing big binary chunks.
>> > >>>
>> > >>
>> > >> Get Jack Levin to write it (smile).
>> > >>
>> > >> And make sure the values are compressed that you send over from the
>> > >> client....
>> > >>
>> > >> St.Ack
>> > >>
>> >
>> >
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
--
Andrey.
Re: Storing extremely large size file
Posted by kim young ill <kh...@googlemail.com>.
+1 documentation please
On Wed, Apr 18, 2012 at 7:21 AM, anil gupta <an...@gmail.com> wrote:
> +1 for documentation. It will help a lot of people.
>
> On Tue, Apr 17, 2012 at 10:02 PM, lars hofhansl <lh...@yahoo.com>
> wrote:
>
> > I disagree. This comes up frequently and some basic guidelines should be
> > documented in the Reference Guide.
> > If it is indeed not difficult than the section is the book will be short.
> >
> >
> >
> > ----- Original Message -----
> > From: Michael Segel <mi...@hotmail.com>
> > To: "user@hbase.apache.org" <us...@hbase.apache.org>
> > Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
> > Sent: Tuesday, April 17, 2012 3:43 PM
> > Subject: Re: Storing extremely large size file
> >
> > -1. It's a boring topic.
> > And it's one of those things that you either get it right or you end up
> > hiring a voodoo witch doctor to curse the author of the chapter...
> >
> > I agree w Jack, it's not difficult just takes some planning and
> > forethought.
> >
> > Also reading lots of blogs... And some practice...
> >
> >
> > Sent from my iPhone
> >
> > On Apr 17, 2012, at 1:42 PM, "Dave Revell" <da...@urbanairship.com>
> wrote:
> >
> > > +1 Jack :)
> > >
> > > On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
> > >
> > >> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
> > >> wrote:
> > >>> I think this is a popular topic that might deserve a section in The
> > Book.
> > >>>
> > >>> By "this topic" I mean storing big binary chunks.
> > >>>
> > >>
> > >> Get Jack Levin to write it (smile).
> > >>
> > >> And make sure the values are compressed that you send over from the
> > >> client....
> > >>
> > >> St.Ack
> > >>
> >
> >
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: Storing extremely large size file
Posted by anil gupta <an...@gmail.com>.
+1 for documentation. It will help a lot of people.
On Tue, Apr 17, 2012 at 10:02 PM, lars hofhansl <lh...@yahoo.com> wrote:
> I disagree. This comes up frequently and some basic guidelines should be
> documented in the Reference Guide.
> If it is indeed not difficult than the section is the book will be short.
>
>
>
> ----- Original Message -----
> From: Michael Segel <mi...@hotmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Tuesday, April 17, 2012 3:43 PM
> Subject: Re: Storing extremely large size file
>
> -1. It's a boring topic.
> And it's one of those things that you either get it right or you end up
> hiring a voodoo witch doctor to curse the author of the chapter...
>
> I agree w Jack, it's not difficult just takes some planning and
> forethought.
>
> Also reading lots of blogs... And some practice...
>
>
> Sent from my iPhone
>
> On Apr 17, 2012, at 1:42 PM, "Dave Revell" <da...@urbanairship.com> wrote:
>
> > +1 Jack :)
> >
> > On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
> >
> >> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
> >> wrote:
> >>> I think this is a popular topic that might deserve a section in The
> Book.
> >>>
> >>> By "this topic" I mean storing big binary chunks.
> >>>
> >>
> >> Get Jack Levin to write it (smile).
> >>
> >> And make sure the values are compressed that you send over from the
> >> client....
> >>
> >> St.Ack
> >>
>
>
--
Thanks & Regards,
Anil Gupta
Re: Storing extremely large size file
Posted by Michel Segel <mi...@hotmail.com>.
Look, I don't want to be *that* guy, but just my $0.02 cents ...
I don't disagree that this topic doesn't come up all the time.
But you have a couple of issues. What do you consider to be large?
1kb? 10? 100? >1MB?
(and then there's that lie that size doesn't matter... But let's not go there... ;-)
Then there is the issue of region size, number of regions per RS...
(this question alone yields different answers from different people, meaning there isn't a single right answer or even a meaningful one.)
Then how about the heap size for the RS?
How about how much memory should have for your DNs...
You then Run in to some one saying that they read that they could build a DN with X hardware and yet this chapter now says you need to have Y and what's up with that...
Then you have the issue of alternatives like storing the blob in a sequence file while you store the index in HBase...
Just saying... :-)
Sent from a remote device. Please excuse any typos...
Mike Segel
On Apr 18, 2012, at 12:02 AM, lars hofhansl <lh...@yahoo.com> wrote:
> I disagree. This comes up frequently and some basic guidelines should be documented in the Reference Guide.
> If it is indeed not difficult than the section is the book will be short.
>
>
>
> ----- Original Message -----
> From: Michael Segel <mi...@hotmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Tuesday, April 17, 2012 3:43 PM
> Subject: Re: Storing extremely large size file
>
> -1. It's a boring topic.
> And it's one of those things that you either get it right or you end up hiring a voodoo witch doctor to curse the author of the chapter...
>
> I agree w Jack, it's not difficult just takes some planning and forethought.
>
> Also reading lots of blogs... And some practice...
>
>
> Sent from my iPhone
>
> On Apr 17, 2012, at 1:42 PM, "Dave Revell" <da...@urbanairship.com> wrote:
>
>> +1 Jack :)
>>
>> On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
>>
>>> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
>>> wrote:
>>>> I think this is a popular topic that might deserve a section in The Book.
>>>>
>>>> By "this topic" I mean storing big binary chunks.
>>>>
>>>
>>> Get Jack Levin to write it (smile).
>>>
>>> And make sure the values are compressed that you send over from the
>>> client....
>>>
>>> St.Ack
>>>
>
>
Re: Storing extremely large size file
Posted by Doug Meil <do...@explorysmedical.com>.
This would be a good entry for the new "Use Cases" chapter in the Schema
Design section.
On 4/18/12 1:02 AM, "lars hofhansl" <lh...@yahoo.com> wrote:
>I disagree. This comes up frequently and some basic guidelines should be
>documented in the Reference Guide.
>If it is indeed not difficult than the section is the book will be short.
>
>
>
>----- Original Message -----
>From: Michael Segel <mi...@hotmail.com>
>To: "user@hbase.apache.org" <us...@hbase.apache.org>
>Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
>Sent: Tuesday, April 17, 2012 3:43 PM
>Subject: Re: Storing extremely large size file
>
>-1. It's a boring topic.
>And it's one of those things that you either get it right or you end up
>hiring a voodoo witch doctor to curse the author of the chapter...
>
>I agree w Jack, it's not difficult just takes some planning and
>forethought.
>
>Also reading lots of blogs... And some practice...
>
>
>Sent from my iPhone
>
>On Apr 17, 2012, at 1:42 PM, "Dave Revell" <da...@urbanairship.com> wrote:
>
>> +1 Jack :)
>>
>> On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
>>
>>> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
>>> wrote:
>>>> I think this is a popular topic that might deserve a section in The
>>>>Book.
>>>>
>>>> By "this topic" I mean storing big binary chunks.
>>>>
>>>
>>> Get Jack Levin to write it (smile).
>>>
>>> And make sure the values are compressed that you send over from the
>>> client....
>>>
>>> St.Ack
>>>
>
>
Re: Storing extremely large size file
Posted by lars hofhansl <lh...@yahoo.com>.
I disagree. This comes up frequently and some basic guidelines should be documented in the Reference Guide.
If it is indeed not difficult than the section is the book will be short.
----- Original Message -----
From: Michael Segel <mi...@hotmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Tuesday, April 17, 2012 3:43 PM
Subject: Re: Storing extremely large size file
-1. It's a boring topic.
And it's one of those things that you either get it right or you end up hiring a voodoo witch doctor to curse the author of the chapter...
I agree w Jack, it's not difficult just takes some planning and forethought.
Also reading lots of blogs... And some practice...
Sent from my iPhone
On Apr 17, 2012, at 1:42 PM, "Dave Revell" <da...@urbanairship.com> wrote:
> +1 Jack :)
>
> On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
>
>> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
>> wrote:
>>> I think this is a popular topic that might deserve a section in The Book.
>>>
>>> By "this topic" I mean storing big binary chunks.
>>>
>>
>> Get Jack Levin to write it (smile).
>>
>> And make sure the values are compressed that you send over from the
>> client....
>>
>> St.Ack
>>
Re: Storing extremely large size file
Posted by Michael Segel <mi...@hotmail.com>.
-1. It's a boring topic.
And it's one of those things that you either get it right or you end up hiring a voodoo witch doctor to curse the author of the chapter...
I agree w Jack, it's not difficult just takes some planning and forethought.
Also reading lots of blogs... And some practice...
Sent from my iPhone
On Apr 17, 2012, at 1:42 PM, "Dave Revell" <da...@urbanairship.com> wrote:
> +1 Jack :)
>
> On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
>
>> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
>> wrote:
>>> I think this is a popular topic that might deserve a section in The Book.
>>>
>>> By "this topic" I mean storing big binary chunks.
>>>
>>
>> Get Jack Levin to write it (smile).
>>
>> And make sure the values are compressed that you send over from the
>> client....
>>
>> St.Ack
>>
Re: Storing extremely large size file
Posted by Andrew Purtell <ap...@apache.org>.
I've stored up to 100MB in cells, but only at the end of a long tail.This is a typical "webtable" type application.You can get away with it, but only if they are outliers, and you need to be careful with scan load with those things in the store like icebergs.
For many (most?) use cases, I'd say having individual values larger than a few megabytes mean you are doing something wrong.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Wei Shung Chung <we...@gmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Cc:
> Sent: Wednesday, April 18, 2012 2:49 AM
> Subject: Re: Storing extremely large size file
>
> What would be the max affordable size one could have ?
>
> Sent from my iPhone
>
> On Apr 17, 2012, at 1:42 PM, Dave Revell <da...@urbanairship.com> wrote:
>
>> +1 Jack :)
>>
>> On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
>>
>>> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell
> <da...@urbanairship.com>
>>> wrote:
>>>> I think this is a popular topic that might deserve a section in The
> Book.
>>>>
>>>> By "this topic" I mean storing big binary chunks.
>>>>
>>>
>>> Get Jack Levin to write it (smile).
>>>
>>> And make sure the values are compressed that you send over from the
>>> client....
>>>
>>> St.Ack
>>>
>
Re: Storing extremely large size file
Posted by Weishung Chung <we...@gmail.com>.
Thank you all. Practice makes perfect :)
On Tue, Apr 17, 2012 at 5:46 PM, Michael Segel <mi...@hotmail.com>wrote:
> In theory, you could go as large as a region size minus the key and
> overhead. (rows can't span regions)
>
> Realistically you'd want to go much smaller.
>
>
> Sent from my iPhone
>
> On Apr 17, 2012, at 1:49 PM, "Wei Shung Chung" <we...@gmail.com> wrote:
>
> > What would be the max affordable size one could have ?
> >
> > Sent from my iPhone
> >
> > On Apr 17, 2012, at 1:42 PM, Dave Revell <da...@urbanairship.com> wrote:
> >
> >> +1 Jack :)
> >>
> >> On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
> >>
> >>> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
> >>> wrote:
> >>>> I think this is a popular topic that might deserve a section in The
> Book.
> >>>>
> >>>> By "this topic" I mean storing big binary chunks.
> >>>>
> >>>
> >>> Get Jack Levin to write it (smile).
> >>>
> >>> And make sure the values are compressed that you send over from the
> >>> client....
> >>>
> >>> St.Ack
> >>>
>
Re: Storing extremely large size file
Posted by Michael Segel <mi...@hotmail.com>.
In theory, you could go as large as a region size minus the key and overhead. (rows can't span regions)
Realistically you'd want to go much smaller.
Sent from my iPhone
On Apr 17, 2012, at 1:49 PM, "Wei Shung Chung" <we...@gmail.com> wrote:
> What would be the max affordable size one could have ?
>
> Sent from my iPhone
>
> On Apr 17, 2012, at 1:42 PM, Dave Revell <da...@urbanairship.com> wrote:
>
>> +1 Jack :)
>>
>> On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
>>
>>> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
>>> wrote:
>>>> I think this is a popular topic that might deserve a section in The Book.
>>>>
>>>> By "this topic" I mean storing big binary chunks.
>>>>
>>>
>>> Get Jack Levin to write it (smile).
>>>
>>> And make sure the values are compressed that you send over from the
>>> client....
>>>
>>> St.Ack
>>>
Re: Storing extremely large size file
Posted by Wei Shung Chung <we...@gmail.com>.
What would be the max affordable size one could have ?
Sent from my iPhone
On Apr 17, 2012, at 1:42 PM, Dave Revell <da...@urbanairship.com> wrote:
> +1 Jack :)
>
> On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
>
>> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
>> wrote:
>>> I think this is a popular topic that might deserve a section in The Book.
>>>
>>> By "this topic" I mean storing big binary chunks.
>>>
>>
>> Get Jack Levin to write it (smile).
>>
>> And make sure the values are compressed that you send over from the
>> client....
>>
>> St.Ack
>>
Re: Storing extremely large size file
Posted by Dave Revell <da...@urbanairship.com>.
+1 Jack :)
On Tue, Apr 17, 2012 at 11:38 AM, Stack <st...@duboce.net> wrote:
> On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com>
> wrote:
> > I think this is a popular topic that might deserve a section in The Book.
> >
> > By "this topic" I mean storing big binary chunks.
> >
>
> Get Jack Levin to write it (smile).
>
> And make sure the values are compressed that you send over from the
> client....
>
> St.Ack
>
Re: Storing extremely large size file
Posted by Stack <st...@duboce.net>.
On Tue, Apr 17, 2012 at 11:18 AM, Dave Revell <da...@urbanairship.com> wrote:
> I think this is a popular topic that might deserve a section in The Book.
>
> By "this topic" I mean storing big binary chunks.
>
Get Jack Levin to write it (smile).
And make sure the values are compressed that you send over from the client....
St.Ack
Re: Storing extremely large size file
Posted by Dave Revell <da...@urbanairship.com>.
I think this is a popular topic that might deserve a section in The Book.
By "this topic" I mean storing big binary chunks.
-Dave
On Tue, Apr 17, 2012 at 11:07 AM, kim young ill <kh...@googlemail.com>wrote:
> i plan to move some data from relational-db to hbase, most of them are
> binary with some hundreds KBs, would like to hear some
> best-practice/tuning for storing this kind of data
> thanx
>
> On Tue, Apr 17, 2012 at 7:50 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
> > Yes, you fine tuned it properly :) But in general I wouldn't
> > recommend it to new users.
> >
> > J-D
> >
> > On Tue, Apr 17, 2012 at 10:47 AM, Jack Levin <ma...@gmail.com> wrote:
> > > Whats wrong with that size? We store > 15MB routinely into our image
> > hbase.
> > >
> > > -Jack
> > >
> > > On Tue, Apr 17, 2012 at 10:46 AM, Jean-Daniel Cryans
> > > <jd...@apache.org> wrote:
> > >> Make sure the config is changed client-side not server-side.
> > >>
> > >> Also you might not want to store 12MB values in HBase.
> > >>
> > >> J-D
> > >>
> > >> On Tue, Apr 17, 2012 at 6:06 AM, vishnupriyaa <vp...@gmail.com>
> > wrote:
> > >>>
> > >>> I want to save a file of size 12MB but an exception occuring like
> this
> > >>> KeyValue size too large.
> > >>> I have set the value of hbase.client.keyvalue.maxsize in
> > hbase-site.xml and
> > >>> hbase-default.xml to 3GB
> > >>> but the default value 10MB is taking for
> > hbase.client.keyvalue.maxsize.How
> > >>> could I change the value of hbase.client.keyvalue.maxsize or how to
> > store
> > >>> the file of extremely large size.
> > >>> --
> > >>> View this message in context:
> >
> http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33701522.html
> > >>> Sent from the HBase User mailing list archive at Nabble.com.
> > >>>
> >
>
Re: Storing extremely large size file
Posted by kim young ill <kh...@googlemail.com>.
i plan to move some data from relational-db to hbase, most of them are
binary with some hundreds KBs, would like to hear some
best-practice/tuning for storing this kind of data
thanx
On Tue, Apr 17, 2012 at 7:50 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Yes, you fine tuned it properly :) But in general I wouldn't
> recommend it to new users.
>
> J-D
>
> On Tue, Apr 17, 2012 at 10:47 AM, Jack Levin <ma...@gmail.com> wrote:
> > Whats wrong with that size? We store > 15MB routinely into our image
> hbase.
> >
> > -Jack
> >
> > On Tue, Apr 17, 2012 at 10:46 AM, Jean-Daniel Cryans
> > <jd...@apache.org> wrote:
> >> Make sure the config is changed client-side not server-side.
> >>
> >> Also you might not want to store 12MB values in HBase.
> >>
> >> J-D
> >>
> >> On Tue, Apr 17, 2012 at 6:06 AM, vishnupriyaa <vp...@gmail.com>
> wrote:
> >>>
> >>> I want to save a file of size 12MB but an exception occuring like this
> >>> KeyValue size too large.
> >>> I have set the value of hbase.client.keyvalue.maxsize in
> hbase-site.xml and
> >>> hbase-default.xml to 3GB
> >>> but the default value 10MB is taking for
> hbase.client.keyvalue.maxsize.How
> >>> could I change the value of hbase.client.keyvalue.maxsize or how to
> store
> >>> the file of extremely large size.
> >>> --
> >>> View this message in context:
> http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33701522.html
> >>> Sent from the HBase User mailing list archive at Nabble.com.
> >>>
>
Re: Storing extremely large size file
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yes, you fine tuned it properly :) But in general I wouldn't
recommend it to new users.
J-D
On Tue, Apr 17, 2012 at 10:47 AM, Jack Levin <ma...@gmail.com> wrote:
> Whats wrong with that size? We store > 15MB routinely into our image hbase.
>
> -Jack
>
> On Tue, Apr 17, 2012 at 10:46 AM, Jean-Daniel Cryans
> <jd...@apache.org> wrote:
>> Make sure the config is changed client-side not server-side.
>>
>> Also you might not want to store 12MB values in HBase.
>>
>> J-D
>>
>> On Tue, Apr 17, 2012 at 6:06 AM, vishnupriyaa <vp...@gmail.com> wrote:
>>>
>>> I want to save a file of size 12MB but an exception occuring like this
>>> KeyValue size too large.
>>> I have set the value of hbase.client.keyvalue.maxsize in hbase-site.xml and
>>> hbase-default.xml to 3GB
>>> but the default value 10MB is taking for hbase.client.keyvalue.maxsize.How
>>> could I change the value of hbase.client.keyvalue.maxsize or how to store
>>> the file of extremely large size.
>>> --
>>> View this message in context: http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33701522.html
>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>
Re: Storing extremely large size file
Posted by Jack Levin <ma...@gmail.com>.
Whats wrong with that size? We store > 15MB routinely into our image hbase.
-Jack
On Tue, Apr 17, 2012 at 10:46 AM, Jean-Daniel Cryans
<jd...@apache.org> wrote:
> Make sure the config is changed client-side not server-side.
>
> Also you might not want to store 12MB values in HBase.
>
> J-D
>
> On Tue, Apr 17, 2012 at 6:06 AM, vishnupriyaa <vp...@gmail.com> wrote:
>>
>> I want to save a file of size 12MB but an exception occuring like this
>> KeyValue size too large.
>> I have set the value of hbase.client.keyvalue.maxsize in hbase-site.xml and
>> hbase-default.xml to 3GB
>> but the default value 10MB is taking for hbase.client.keyvalue.maxsize.How
>> could I change the value of hbase.client.keyvalue.maxsize or how to store
>> the file of extremely large size.
>> --
>> View this message in context: http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33701522.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
Re: Storing extremely large size file
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Make sure the config is changed client-side not server-side.
Also you might not want to store 12MB values in HBase.
J-D
On Tue, Apr 17, 2012 at 6:06 AM, vishnupriyaa <vp...@gmail.com> wrote:
>
> I want to save a file of size 12MB but an exception occuring like this
> KeyValue size too large.
> I have set the value of hbase.client.keyvalue.maxsize in hbase-site.xml and
> hbase-default.xml to 3GB
> but the default value 10MB is taking for hbase.client.keyvalue.maxsize.How
> could I change the value of hbase.client.keyvalue.maxsize or how to store
> the file of extremely large size.
> --
> View this message in context: http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33701522.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: Storing extremely large size file
Posted by Brock Noland <br...@cloudera.com>.
This should be about right:
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
OutputStream os = fs.create(new Path("/images/img.jpg"));
os.write()...
os.close();
Or even:
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("/filer/img.jpg"), new Path("/images/img.jpg"));
http://hadoop.apache.org/common/docs/r1.0.2/api/org/apache/hadoop/fs/FileSystem.html
On Fri, Apr 20, 2012 at 7:38 AM, vishnupriyaa <vp...@gmail.com> wrote:
>
> Hi Brock,
> I am new to hbase ,can you help me with some sort of code to store
> directly in HDFS.
>
> Brock Noland-2 wrote:
>>
>> Hi,
>>
>> Any reason you cannot have an exception marker for large files and
>> then store them directly in HDFS?
>>
>> Brock
>>
>> On Tue, Apr 17, 2012 at 1:06 PM, vishnupriyaa <vp...@gmail.com>
>> wrote:
>>>
>>> I want to save a file of size 12MB but an exception occuring like this
>>> KeyValue size too large.
>>> I have set the value of hbase.client.keyvalue.maxsize in hbase-site.xml
>>> and
>>> hbase-default.xml to 3GB
>>> but the default value 10MB is taking for
>>> hbase.client.keyvalue.maxsize.How
>>> could I change the value of hbase.client.keyvalue.maxsize or how to store
>>> the file of extremely large size.
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33701522.html
>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33720293.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Storing extremely large size file
Posted by vishnupriyaa <vp...@gmail.com>.
Hi Brock,
I am new to hbase ,can you help me with some sort of code to store
directly in HDFS.
Brock Noland-2 wrote:
>
> Hi,
>
> Any reason you cannot have an exception marker for large files and
> then store them directly in HDFS?
>
> Brock
>
> On Tue, Apr 17, 2012 at 1:06 PM, vishnupriyaa <vp...@gmail.com>
> wrote:
>>
>> I want to save a file of size 12MB but an exception occuring like this
>> KeyValue size too large.
>> I have set the value of hbase.client.keyvalue.maxsize in hbase-site.xml
>> and
>> hbase-default.xml to 3GB
>> but the default value 10MB is taking for
>> hbase.client.keyvalue.maxsize.How
>> could I change the value of hbase.client.keyvalue.maxsize or how to store
>> the file of extremely large size.
>> --
>> View this message in context:
>> http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33701522.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
>
--
View this message in context: http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33720293.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Storing extremely large size file
Posted by Brock Noland <br...@cloudera.com>.
Hi,
Any reason you cannot have an exception marker for large files and
then store them directly in HDFS?
Brock
On Tue, Apr 17, 2012 at 1:06 PM, vishnupriyaa <vp...@gmail.com> wrote:
>
> I want to save a file of size 12MB but an exception occuring like this
> KeyValue size too large.
> I have set the value of hbase.client.keyvalue.maxsize in hbase-site.xml and
> hbase-default.xml to 3GB
> but the default value 10MB is taking for hbase.client.keyvalue.maxsize.How
> could I change the value of hbase.client.keyvalue.maxsize or how to store
> the file of extremely large size.
> --
> View this message in context: http://old.nabble.com/Storing-extremely-large-size-file-tp33701522p33701522.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/