You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Sadasivam, Srinivasan" <sr...@qualcomm.com> on 2011/01/14 19:36:34 UTC
KeyValue size too large
Hi,
Running a throughput test with various sample sizes but getting "KeyValue size too large" for 10MB files. Could not remember any setting related to cell max size.
Thanks
+ Srini Sadasivam
11/01/14 18:26:25 INFO perfclient.Main: Read:0 req/s Write: 0 req/s Total Througput: 0 req/s
Exception in thread "Thread-10" java.lang.IllegalArgumentException: KeyValue size too large
at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:688)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:544)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
at com.qualcomm.qis.hbase.perfclient.WriteHBaseThread.run(WriteHBaseThread.java:152)
Re: Newbie question about application development
Posted by Chris Tarnas <cf...@email.com>.
When you submit a job via hadoop you include the jar with the submission. Hadoop takes care of the distribution of the jar to each of the nodes and the cleanup when all processing is done. You can also include other needed files and archives with the job as well. You would use the 'hadoop' command for this:
http://hadoop.apache.org/common/docs/r0.20.2/commands_manual.html
-chris
On Jan 17, 2011, at 11:47 AM, Jérôme Verstrynge wrote:
> Hi,
>
> Trying to understand HBase/Hadoop here. Let's imagine I write some MapReduce Java code in my application. Let's imagine I deploy my application on 1 node of a ring of 20. Let's imagine the data is distributed on the 20 nodes.
>
> Do I need to distribute my JAR on the 20 to run the MapReduce Java code? Or can I launch this code from my node and HBase/Hadoop will distribute it automatically to other nodes for local processing? How does the magic happen?
>
> Thanks,
>
> JVerstry
>
Newbie question about application development
Posted by Jérôme Verstrynge <jv...@gmail.com>.
Hi,
Trying to understand HBase/Hadoop here. Let's imagine I write some
MapReduce Java code in my application. Let's imagine I deploy my
application on 1 node of a ring of 20. Let's imagine the data is
distributed on the 20 nodes.
Do I need to distribute my JAR on the 20 to run the MapReduce Java code?
Or can I launch this code from my node and HBase/Hadoop will distribute
it automatically to other nodes for local processing? How does the magic
happen?
Thanks,
JVerstry
Re: KeyValue size too large
Posted by Stack <st...@duboce.net>.
Ok. 1k -> 2MB or so... maybe up to 10MB MIGHT be OK in HBase but
bigger and you are into an area little tested; there is probably work
to do to make it smooth-running (yeah, 3 nodes is a little
underpowered too).
Thanks for writing back Eric,
St.Ack
On Mon, Jan 17, 2011 at 4:38 AM, Eric <er...@gmail.com> wrote:
> St.Ack, my files ranged from 1KB up to 80MB sometimes. Maybe the cluster was
> also underpowered (3 nodes). I'm now looking at writing the files to map
> files and only storing meta information.
>
> 2011/1/15 Stack <st...@duboce.net>
>
>> Tell us more Eric. Which HBase version? What kinda sizes. We should
>> make it work reliably for at least cells of 1 or 2 MB.
>> Thanks,
>> St.Ack
>>
>> On Sat, Jan 15, 2011 at 2:55 AM, Eric <er...@gmail.com> wrote:
>> > I would also recommend against storing large files in HBase. Your regions
>> > get filled up very quickly, meaning you will get a lot of regions. You
>> can
>> > increase the max region size but I've seen very unstable behaviour while
>> > insering tens of thousands of small to large files into HBase with both
>> > small and large max region sizes.
>> >
>> > 2011/1/15 Ryan Rawson <ry...@gmail.com>
>> >
>> >> This is a client side check yes, requested as a double check to make
>> >> sure that applications aren't going crazy.
>> >>
>> >> Its a combination size of row, qualifier, HBase metadata, and your
>> >> actual value. If the combo exceeds that limit you get that exception.
>> >>
>> >> Disable it by setting it to 0. Note that disabling this check doesn't
>> >> actually lift any intrinsic limits that are due to use integer and
>> >> other fields to store lengths. That is, HBase is not appropriate for
>> >> storing cells > 2GB, and you will hit performance problems before then
>> >> anyways.
>> >>
>> >> For extremely large data, consider storing the meta data in HBase and
>> >> the actual data in HDFS.
>> >>
>> >> -ryan
>> >>
>> >> On Fri, Jan 14, 2011 at 4:09 PM, Sean Bigdatafun
>> >> <se...@gmail.com> wrote:
>> >> > And I guess if your k-v exceed that number. The HBase client library
>> will
>> >> > cut at the the last byte on the boundary? (my question is if this
>> >> parameter
>> >> > is a client side parameter)
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Jan 14, 2011 at 11:15 AM, Christopher Tarnas <cf...@email.com>
>> >> wrote:
>> >> >
>> >> >> You need to increase the hbase.client.keyvalue.maxsize property in
>> >> >> hbase-site.xml. It defaults to 10 MB.
>> >> >>
>> >> >> -chris
>> >> >>
>> >> >> On Fri, Jan 14, 2011 at 12:36 PM, Sadasivam, Srinivasan
>> >> >> <sr...@qualcomm.com>wrote:
>> >> >>
>> >> >> > Hi,
>> >> >> > Running a throughput test with various sample sizes but getting
>> >> "KeyValue
>> >> >> > size too large" for 10MB files. Could not remember any setting
>> related
>> >> to
>> >> >> > cell max size.
>> >> >> > Thanks
>> >> >> > + Srini Sadasivam
>> >> >> >
>> >> >> > 11/01/14 18:26:25 INFO perfclient.Main: Read:0 req/s Write: 0 req/s
>> >> Total
>> >> >> > Througput: 0 req/s
>> >> >> > Exception in thread "Thread-10" java.lang.IllegalArgumentException:
>> >> >> > KeyValue size too large
>> >> >> > at
>> >> >> > org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:688)
>> >> >> > at
>> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:544)
>> >> >> > at
>> org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
>> >> >> > at
>> >> >> >
>> >> >>
>> >>
>> com.qualcomm.qis.hbase.perfclient.WriteHBaseThread.run(WriteHBaseThread.java:152)
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > --Sean
>> >> >
>> >>
>> >
>>
>
Re: KeyValue size too large
Posted by Eric <er...@gmail.com>.
St.Ack, my files ranged from 1KB up to 80MB sometimes. Maybe the cluster was
also underpowered (3 nodes). I'm now looking at writing the files to map
files and only storing meta information.
2011/1/15 Stack <st...@duboce.net>
> Tell us more Eric. Which HBase version? What kinda sizes. We should
> make it work reliably for at least cells of 1 or 2 MB.
> Thanks,
> St.Ack
>
> On Sat, Jan 15, 2011 at 2:55 AM, Eric <er...@gmail.com> wrote:
> > I would also recommend against storing large files in HBase. Your regions
> > get filled up very quickly, meaning you will get a lot of regions. You
> can
> > increase the max region size but I've seen very unstable behaviour while
> > insering tens of thousands of small to large files into HBase with both
> > small and large max region sizes.
> >
> > 2011/1/15 Ryan Rawson <ry...@gmail.com>
> >
> >> This is a client side check yes, requested as a double check to make
> >> sure that applications aren't going crazy.
> >>
> >> Its a combination size of row, qualifier, HBase metadata, and your
> >> actual value. If the combo exceeds that limit you get that exception.
> >>
> >> Disable it by setting it to 0. Note that disabling this check doesn't
> >> actually lift any intrinsic limits that are due to use integer and
> >> other fields to store lengths. That is, HBase is not appropriate for
> >> storing cells > 2GB, and you will hit performance problems before then
> >> anyways.
> >>
> >> For extremely large data, consider storing the meta data in HBase and
> >> the actual data in HDFS.
> >>
> >> -ryan
> >>
> >> On Fri, Jan 14, 2011 at 4:09 PM, Sean Bigdatafun
> >> <se...@gmail.com> wrote:
> >> > And I guess if your k-v exceed that number. The HBase client library
> will
> >> > cut at the the last byte on the boundary? (my question is if this
> >> parameter
> >> > is a client side parameter)
> >> >
> >> >
> >> >
> >> > On Fri, Jan 14, 2011 at 11:15 AM, Christopher Tarnas <cf...@email.com>
> >> wrote:
> >> >
> >> >> You need to increase the hbase.client.keyvalue.maxsize property in
> >> >> hbase-site.xml. It defaults to 10 MB.
> >> >>
> >> >> -chris
> >> >>
> >> >> On Fri, Jan 14, 2011 at 12:36 PM, Sadasivam, Srinivasan
> >> >> <sr...@qualcomm.com>wrote:
> >> >>
> >> >> > Hi,
> >> >> > Running a throughput test with various sample sizes but getting
> >> "KeyValue
> >> >> > size too large" for 10MB files. Could not remember any setting
> related
> >> to
> >> >> > cell max size.
> >> >> > Thanks
> >> >> > + Srini Sadasivam
> >> >> >
> >> >> > 11/01/14 18:26:25 INFO perfclient.Main: Read:0 req/s Write: 0 req/s
> >> Total
> >> >> > Througput: 0 req/s
> >> >> > Exception in thread "Thread-10" java.lang.IllegalArgumentException:
> >> >> > KeyValue size too large
> >> >> > at
> >> >> > org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:688)
> >> >> > at
> org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:544)
> >> >> > at
> org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
> >> >> > at
> >> >> >
> >> >>
> >>
> com.qualcomm.qis.hbase.perfclient.WriteHBaseThread.run(WriteHBaseThread.java:152)
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > --Sean
> >> >
> >>
> >
>
Re: KeyValue size too large
Posted by Stack <st...@duboce.net>.
Tell us more Eric. Which HBase version? What kinda sizes. We should
make it work reliably for at least cells of 1 or 2 MB.
Thanks,
St.Ack
On Sat, Jan 15, 2011 at 2:55 AM, Eric <er...@gmail.com> wrote:
> I would also recommend against storing large files in HBase. Your regions
> get filled up very quickly, meaning you will get a lot of regions. You can
> increase the max region size but I've seen very unstable behaviour while
> insering tens of thousands of small to large files into HBase with both
> small and large max region sizes.
>
> 2011/1/15 Ryan Rawson <ry...@gmail.com>
>
>> This is a client side check yes, requested as a double check to make
>> sure that applications aren't going crazy.
>>
>> Its a combination size of row, qualifier, HBase metadata, and your
>> actual value. If the combo exceeds that limit you get that exception.
>>
>> Disable it by setting it to 0. Note that disabling this check doesn't
>> actually lift any intrinsic limits that are due to use integer and
>> other fields to store lengths. That is, HBase is not appropriate for
>> storing cells > 2GB, and you will hit performance problems before then
>> anyways.
>>
>> For extremely large data, consider storing the meta data in HBase and
>> the actual data in HDFS.
>>
>> -ryan
>>
>> On Fri, Jan 14, 2011 at 4:09 PM, Sean Bigdatafun
>> <se...@gmail.com> wrote:
>> > And I guess if your k-v exceed that number. The HBase client library will
>> > cut at the the last byte on the boundary? (my question is if this
>> parameter
>> > is a client side parameter)
>> >
>> >
>> >
>> > On Fri, Jan 14, 2011 at 11:15 AM, Christopher Tarnas <cf...@email.com>
>> wrote:
>> >
>> >> You need to increase the hbase.client.keyvalue.maxsize property in
>> >> hbase-site.xml. It defaults to 10 MB.
>> >>
>> >> -chris
>> >>
>> >> On Fri, Jan 14, 2011 at 12:36 PM, Sadasivam, Srinivasan
>> >> <sr...@qualcomm.com>wrote:
>> >>
>> >> > Hi,
>> >> > Running a throughput test with various sample sizes but getting
>> "KeyValue
>> >> > size too large" for 10MB files. Could not remember any setting related
>> to
>> >> > cell max size.
>> >> > Thanks
>> >> > + Srini Sadasivam
>> >> >
>> >> > 11/01/14 18:26:25 INFO perfclient.Main: Read:0 req/s Write: 0 req/s
>> Total
>> >> > Througput: 0 req/s
>> >> > Exception in thread "Thread-10" java.lang.IllegalArgumentException:
>> >> > KeyValue size too large
>> >> > at
>> >> > org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:688)
>> >> > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:544)
>> >> > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
>> >> > at
>> >> >
>> >>
>> com.qualcomm.qis.hbase.perfclient.WriteHBaseThread.run(WriteHBaseThread.java:152)
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > --Sean
>> >
>>
>
Re: KeyValue size too large
Posted by Eric <er...@gmail.com>.
I would also recommend against storing large files in HBase. Your regions
get filled up very quickly, meaning you will get a lot of regions. You can
increase the max region size but I've seen very unstable behaviour while
insering tens of thousands of small to large files into HBase with both
small and large max region sizes.
2011/1/15 Ryan Rawson <ry...@gmail.com>
> This is a client side check yes, requested as a double check to make
> sure that applications aren't going crazy.
>
> Its a combination size of row, qualifier, HBase metadata, and your
> actual value. If the combo exceeds that limit you get that exception.
>
> Disable it by setting it to 0. Note that disabling this check doesn't
> actually lift any intrinsic limits that are due to use integer and
> other fields to store lengths. That is, HBase is not appropriate for
> storing cells > 2GB, and you will hit performance problems before then
> anyways.
>
> For extremely large data, consider storing the meta data in HBase and
> the actual data in HDFS.
>
> -ryan
>
> On Fri, Jan 14, 2011 at 4:09 PM, Sean Bigdatafun
> <se...@gmail.com> wrote:
> > And I guess if your k-v exceed that number. The HBase client library will
> > cut at the the last byte on the boundary? (my question is if this
> parameter
> > is a client side parameter)
> >
> >
> >
> > On Fri, Jan 14, 2011 at 11:15 AM, Christopher Tarnas <cf...@email.com>
> wrote:
> >
> >> You need to increase the hbase.client.keyvalue.maxsize property in
> >> hbase-site.xml. It defaults to 10 MB.
> >>
> >> -chris
> >>
> >> On Fri, Jan 14, 2011 at 12:36 PM, Sadasivam, Srinivasan
> >> <sr...@qualcomm.com>wrote:
> >>
> >> > Hi,
> >> > Running a throughput test with various sample sizes but getting
> "KeyValue
> >> > size too large" for 10MB files. Could not remember any setting related
> to
> >> > cell max size.
> >> > Thanks
> >> > + Srini Sadasivam
> >> >
> >> > 11/01/14 18:26:25 INFO perfclient.Main: Read:0 req/s Write: 0 req/s
> Total
> >> > Througput: 0 req/s
> >> > Exception in thread "Thread-10" java.lang.IllegalArgumentException:
> >> > KeyValue size too large
> >> > at
> >> > org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:688)
> >> > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:544)
> >> > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
> >> > at
> >> >
> >>
> com.qualcomm.qis.hbase.perfclient.WriteHBaseThread.run(WriteHBaseThread.java:152)
> >> >
> >>
> >
> >
> >
> > --
> > --Sean
> >
>
Re: KeyValue size too large
Posted by Ryan Rawson <ry...@gmail.com>.
This is a client side check yes, requested as a double check to make
sure that applications aren't going crazy.
Its a combination size of row, qualifier, HBase metadata, and your
actual value. If the combo exceeds that limit you get that exception.
Disable it by setting it to 0. Note that disabling this check doesn't
actually lift any intrinsic limits that are due to use integer and
other fields to store lengths. That is, HBase is not appropriate for
storing cells > 2GB, and you will hit performance problems before then
anyways.
For extremely large data, consider storing the meta data in HBase and
the actual data in HDFS.
-ryan
On Fri, Jan 14, 2011 at 4:09 PM, Sean Bigdatafun
<se...@gmail.com> wrote:
> And I guess if your k-v exceed that number. The HBase client library will
> cut at the the last byte on the boundary? (my question is if this parameter
> is a client side parameter)
>
>
>
> On Fri, Jan 14, 2011 at 11:15 AM, Christopher Tarnas <cf...@email.com> wrote:
>
>> You need to increase the hbase.client.keyvalue.maxsize property in
>> hbase-site.xml. It defaults to 10 MB.
>>
>> -chris
>>
>> On Fri, Jan 14, 2011 at 12:36 PM, Sadasivam, Srinivasan
>> <sr...@qualcomm.com>wrote:
>>
>> > Hi,
>> > Running a throughput test with various sample sizes but getting "KeyValue
>> > size too large" for 10MB files. Could not remember any setting related to
>> > cell max size.
>> > Thanks
>> > + Srini Sadasivam
>> >
>> > 11/01/14 18:26:25 INFO perfclient.Main: Read:0 req/s Write: 0 req/s Total
>> > Througput: 0 req/s
>> > Exception in thread "Thread-10" java.lang.IllegalArgumentException:
>> > KeyValue size too large
>> > at
>> > org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:688)
>> > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:544)
>> > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
>> > at
>> >
>> com.qualcomm.qis.hbase.perfclient.WriteHBaseThread.run(WriteHBaseThread.java:152)
>> >
>>
>
>
>
> --
> --Sean
>
Re: KeyValue size too large
Posted by Sean Bigdatafun <se...@gmail.com>.
And I guess if your k-v exceed that number. The HBase client library will
cut at the the last byte on the boundary? (my question is if this parameter
is a client side parameter)
On Fri, Jan 14, 2011 at 11:15 AM, Christopher Tarnas <cf...@email.com> wrote:
> You need to increase the hbase.client.keyvalue.maxsize property in
> hbase-site.xml. It defaults to 10 MB.
>
> -chris
>
> On Fri, Jan 14, 2011 at 12:36 PM, Sadasivam, Srinivasan
> <sr...@qualcomm.com>wrote:
>
> > Hi,
> > Running a throughput test with various sample sizes but getting "KeyValue
> > size too large" for 10MB files. Could not remember any setting related to
> > cell max size.
> > Thanks
> > + Srini Sadasivam
> >
> > 11/01/14 18:26:25 INFO perfclient.Main: Read:0 req/s Write: 0 req/s Total
> > Througput: 0 req/s
> > Exception in thread "Thread-10" java.lang.IllegalArgumentException:
> > KeyValue size too large
> > at
> > org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:688)
> > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:544)
> > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
> > at
> >
> com.qualcomm.qis.hbase.perfclient.WriteHBaseThread.run(WriteHBaseThread.java:152)
> >
>
--
--Sean
Re: KeyValue size too large
Posted by Christopher Tarnas <cf...@email.com>.
You need to increase the hbase.client.keyvalue.maxsize property in
hbase-site.xml. It defaults to 10 MB.
-chris
On Fri, Jan 14, 2011 at 12:36 PM, Sadasivam, Srinivasan
<sr...@qualcomm.com>wrote:
> Hi,
> Running a throughput test with various sample sizes but getting "KeyValue
> size too large" for 10MB files. Could not remember any setting related to
> cell max size.
> Thanks
> + Srini Sadasivam
>
> 11/01/14 18:26:25 INFO perfclient.Main: Read:0 req/s Write: 0 req/s Total
> Througput: 0 req/s
> Exception in thread "Thread-10" java.lang.IllegalArgumentException:
> KeyValue size too large
> at
> org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:688)
> at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:544)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:535)
> at
> com.qualcomm.qis.hbase.perfclient.WriteHBaseThread.run(WriteHBaseThread.java:152)
>