You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Calvin <ca...@gmail.com> on 2009/12/01 00:33:25 UTC

Re: hbase bulk writes

Thanks for the responses.  If I can avoid writing a map-reduce job that
would be preferable (getting map-reduce to work with / depend on my existing
infrastructure is turning out to be annoying).

I have no good way of randomizing my dataset since it's a very large stream
of sequential data (ordered by some key).  I have a fair number of column
families (~25) and every column is a long or a double.  Having a standalone
program that writes rows using the HTable / Put API seems to run at ~2-5000
rows/sec, which seems ridiculously slow.  Is it possible I am doing
something terribly wrong?

-Calvin

On Mon, Nov 30, 2009 at 5:47 PM, Ryan Rawson <ry...@gmail.com> wrote:

> Sequentially ordered rows is the worst insert case in HBase - you end
> up writing all to 1 server even if you have 500.  If you could
> randomize your input, and I have pasted a Randomize.java map reduce
> that will randomize lines of a file, then your performance will
> improve.
>
> I have seen sustained inserts of 100-300k rows/sec on small rows
> before.  Obviously large blob rows will be slower, since the limiting
> factor is how fast we can write data to HDFS, thus it isnt the actual
> row count, but the amount of data involved.
>
> Try the randomize.java, see where that gets you. I think it's on the
> list archives.
>
> -ryan
>
>
> On Mon, Nov 30, 2009 at 2:41 PM, Jean-Daniel Cryans <jd...@apache.org>
> wrote:
> > Could you put your data in HDFS and load it from there with a MapReduce
> job?
> >
> > J-D
> >
> > On Mon, Nov 30, 2009 at 2:33 PM, Calvin <ca...@gmail.com> wrote:
> >> I have a large amount of sequential ordered rows I would like to write
> to an
> >> HBase table.  What is the preferred way to do bulk writes of
> multi-column
> >> tables in HBase?  Using the get/put interface seems fairly slow even if
> I
> >> bulk writes with table.put(List<Put>).
> >>
> >> I have followed the directions on:
> >>   * http://wiki.apache.org/hadoop/PerformanceTuning
> >>   *
> >>
> http://ryantwopointoh.blogspot.com/2009/01/performance-of-hbase-importing.html
> >>
> >> Are there any other resources for improving the throughput of my bulk
> >> writes?  On
> >>
> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/mapreduce/package-summary.htmlI
> >> see there's a way to write HFiles directly, but HFileOutputFormat can
> >> only
> >> write a single column famly at a time (
> >> https://issues.apache.org/jira/browse/HBASE-1861).
> >>
> >> Thanks!
> >>
> >> -Calvin
> >>
> >
>

Re: hbase bulk writes

Posted by Stack <sa...@gmail.com>.
I do not know of schemas where there are more than 5 or 6 families. My  
guess is that there will be issues. One issue for sure is that we do  
not parallelize queries across families yet.  The queries run in  
series so will be slow when lots of families involved.  It shouldn't  
be hard to address. It just has not been a priority.

You might be ok if only 5 families queried at a time as part of a mr job

Going by the content of your first mail my guess is that you are clear  
on the difference between a column family and a column.  If you are  
not clear I would suggest you lok into it. Maybe you do not need that  
many families

On Dec 1, 2009, at 4:03 PM, Calvin <ca...@gmail.com> wrote:

> Thanks again for the responses.
>
> Stack: What's the issue with 25 families?  I will mostly be  
> accessing HBase
> as a map-reduce source and will be looking at ~5 column families at  
> a time.
> Is there any documentation on column family limits in practice?
>
> -Calvin
>
> On Mon, Nov 30, 2009 at 7:38 PM, stack <st...@duboce.net> wrote:
>
>> There is little art to the HFileOutputFormat.  You could play with  
>> it to
>> make it support multiple column families (As soon as any family  
>> hits the
>> region size boundary, start up a new set of files).  If keys are  
>> already
>> sorted, could hook up HFileOutputFormat to the map (I think) and  
>> avoid the
>> mapreduce framework sort.
>>
>> 25 families is probably too many for current hbase -- depends on  
>> how you'll
>> be accessing them.
>>
>> Can you put your input files under an http server and then write a
>> mapreduce
>> that pulls via HTTP?
>>
>> St.Ack
>>
>> On Mon, Nov 30, 2009 at 3:33 PM, Calvin <ca...@gmail.com>  
>> wrote:
>>
>>> Thanks for the responses.  If I can avoid writing a map-reduce job  
>>> that
>>> would be preferable (getting map-reduce to work with / depend on my
>>> existing
>>> infrastructure is turning out to be annoying).
>>>
>>> I have no good way of randomizing my dataset since it's a very large
>> stream
>>> of sequential data (ordered by some key).  I have a fair number of  
>>> column
>>> families (~25) and every column is a long or a double.  Having a
>> standalone
>>> program that writes rows using the HTable / Put API seems to run at
>> ~2-5000
>>> rows/sec, which seems ridiculously slow.  Is it possible I am doing
>>> something terribly wrong?
>>>
>>> -Calvin
>>>
>>> On Mon, Nov 30, 2009 at 5:47 PM, Ryan Rawson <ry...@gmail.com>  
>>> wrote:
>>>
>>>> Sequentially ordered rows is the worst insert case in HBase - you  
>>>> end
>>>> up writing all to 1 server even if you have 500.  If you could
>>>> randomize your input, and I have pasted a Randomize.java map reduce
>>>> that will randomize lines of a file, then your performance will
>>>> improve.
>>>>
>>>> I have seen sustained inserts of 100-300k rows/sec on small rows
>>>> before.  Obviously large blob rows will be slower, since the  
>>>> limiting
>>>> factor is how fast we can write data to HDFS, thus it isnt the  
>>>> actual
>>>> row count, but the amount of data involved.
>>>>
>>>> Try the randomize.java, see where that gets you. I think it's on  
>>>> the
>>>> list archives.
>>>>
>>>> -ryan
>>>>
>>>>
>>>> On Mon, Nov 30, 2009 at 2:41 PM, Jean-Daniel Cryans <
>> jdcryans@apache.org
>>>>
>>>> wrote:
>>>>> Could you put your data in HDFS and load it from there with a
>> MapReduce
>>>> job?
>>>>>
>>>>> J-D
>>>>>
>>>>> On Mon, Nov 30, 2009 at 2:33 PM, Calvin <ca...@gmail.com>
>>> wrote:
>>>>>> I have a large amount of sequential ordered rows I would like to
>> write
>>>> to an
>>>>>> HBase table.  What is the preferred way to do bulk writes of
>>>> multi-column
>>>>>> tables in HBase?  Using the get/put interface seems fairly slow  
>>>>>> even
>>> if
>>>> I
>>>>>> bulk writes with table.put(List<Put>).
>>>>>>
>>>>>> I have followed the directions on:
>>>>>>  * http://wiki.apache.org/hadoop/PerformanceTuning
>>>>>>  *
>>>>>>
>>>>
>>>
>> http://ryantwopointoh.blogspot.com/2009/01/performance-of-hbase-importing.html
>>>>>>
>>>>>> Are there any other resources for improving the throughput of my
>> bulk
>>>>>> writes?  On
>>>>>>
>>>>
>>>
>> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/mapreduce/package-summary.htmlI
>>>>>> see there's a way to write HFiles directly, but HFileOutputFormat
>> can
>>>>>> only
>>>>>> write a single column famly at a time (
>>>>>> https://issues.apache.org/jira/browse/HBASE-1861).
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> -Calvin
>>>>>>
>>>>>
>>>>
>>>
>>

Re: hbase bulk writes

Posted by Calvin <ca...@gmail.com>.
Thanks again for the responses.

Stack: What's the issue with 25 families?  I will mostly be accessing HBase
as a map-reduce source and will be looking at ~5 column families at a time.
 Is there any documentation on column family limits in practice?

-Calvin

On Mon, Nov 30, 2009 at 7:38 PM, stack <st...@duboce.net> wrote:

> There is little art to the HFileOutputFormat.  You could play with it to
> make it support multiple column families (As soon as any family hits the
> region size boundary, start up a new set of files).  If keys are already
> sorted, could hook up HFileOutputFormat to the map (I think) and avoid the
> mapreduce framework sort.
>
> 25 families is probably too many for current hbase -- depends on how you'll
> be accessing them.
>
> Can you put your input files under an http server and then write a
> mapreduce
> that pulls via HTTP?
>
> St.Ack
>
> On Mon, Nov 30, 2009 at 3:33 PM, Calvin <ca...@gmail.com> wrote:
>
> > Thanks for the responses.  If I can avoid writing a map-reduce job that
> > would be preferable (getting map-reduce to work with / depend on my
> > existing
> > infrastructure is turning out to be annoying).
> >
> > I have no good way of randomizing my dataset since it's a very large
> stream
> > of sequential data (ordered by some key).  I have a fair number of column
> > families (~25) and every column is a long or a double.  Having a
> standalone
> > program that writes rows using the HTable / Put API seems to run at
> ~2-5000
> > rows/sec, which seems ridiculously slow.  Is it possible I am doing
> > something terribly wrong?
> >
> > -Calvin
> >
> > On Mon, Nov 30, 2009 at 5:47 PM, Ryan Rawson <ry...@gmail.com> wrote:
> >
> > > Sequentially ordered rows is the worst insert case in HBase - you end
> > > up writing all to 1 server even if you have 500.  If you could
> > > randomize your input, and I have pasted a Randomize.java map reduce
> > > that will randomize lines of a file, then your performance will
> > > improve.
> > >
> > > I have seen sustained inserts of 100-300k rows/sec on small rows
> > > before.  Obviously large blob rows will be slower, since the limiting
> > > factor is how fast we can write data to HDFS, thus it isnt the actual
> > > row count, but the amount of data involved.
> > >
> > > Try the randomize.java, see where that gets you. I think it's on the
> > > list archives.
> > >
> > > -ryan
> > >
> > >
> > > On Mon, Nov 30, 2009 at 2:41 PM, Jean-Daniel Cryans <
> jdcryans@apache.org
> > >
> > > wrote:
> > > > Could you put your data in HDFS and load it from there with a
> MapReduce
> > > job?
> > > >
> > > > J-D
> > > >
> > > > On Mon, Nov 30, 2009 at 2:33 PM, Calvin <ca...@gmail.com>
> > wrote:
> > > >> I have a large amount of sequential ordered rows I would like to
> write
> > > to an
> > > >> HBase table.  What is the preferred way to do bulk writes of
> > > multi-column
> > > >> tables in HBase?  Using the get/put interface seems fairly slow even
> > if
> > > I
> > > >> bulk writes with table.put(List<Put>).
> > > >>
> > > >> I have followed the directions on:
> > > >>   * http://wiki.apache.org/hadoop/PerformanceTuning
> > > >>   *
> > > >>
> > >
> >
> http://ryantwopointoh.blogspot.com/2009/01/performance-of-hbase-importing.html
> > > >>
> > > >> Are there any other resources for improving the throughput of my
> bulk
> > > >> writes?  On
> > > >>
> > >
> >
> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/mapreduce/package-summary.htmlI
> > > >> see there's a way to write HFiles directly, but HFileOutputFormat
> can
> > > >> only
> > > >> write a single column famly at a time (
> > > >> https://issues.apache.org/jira/browse/HBASE-1861).
> > > >>
> > > >> Thanks!
> > > >>
> > > >> -Calvin
> > > >>
> > > >
> > >
> >
>

Re: hbase bulk writes

Posted by stack <st...@duboce.net>.
There is little art to the HFileOutputFormat.  You could play with it to
make it support multiple column families (As soon as any family hits the
region size boundary, start up a new set of files).  If keys are already
sorted, could hook up HFileOutputFormat to the map (I think) and avoid the
mapreduce framework sort.

25 families is probably too many for current hbase -- depends on how you'll
be accessing them.

Can you put your input files under an http server and then write a mapreduce
that pulls via HTTP?

St.Ack

On Mon, Nov 30, 2009 at 3:33 PM, Calvin <ca...@gmail.com> wrote:

> Thanks for the responses.  If I can avoid writing a map-reduce job that
> would be preferable (getting map-reduce to work with / depend on my
> existing
> infrastructure is turning out to be annoying).
>
> I have no good way of randomizing my dataset since it's a very large stream
> of sequential data (ordered by some key).  I have a fair number of column
> families (~25) and every column is a long or a double.  Having a standalone
> program that writes rows using the HTable / Put API seems to run at ~2-5000
> rows/sec, which seems ridiculously slow.  Is it possible I am doing
> something terribly wrong?
>
> -Calvin
>
> On Mon, Nov 30, 2009 at 5:47 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
> > Sequentially ordered rows is the worst insert case in HBase - you end
> > up writing all to 1 server even if you have 500.  If you could
> > randomize your input, and I have pasted a Randomize.java map reduce
> > that will randomize lines of a file, then your performance will
> > improve.
> >
> > I have seen sustained inserts of 100-300k rows/sec on small rows
> > before.  Obviously large blob rows will be slower, since the limiting
> > factor is how fast we can write data to HDFS, thus it isnt the actual
> > row count, but the amount of data involved.
> >
> > Try the randomize.java, see where that gets you. I think it's on the
> > list archives.
> >
> > -ryan
> >
> >
> > On Mon, Nov 30, 2009 at 2:41 PM, Jean-Daniel Cryans <jdcryans@apache.org
> >
> > wrote:
> > > Could you put your data in HDFS and load it from there with a MapReduce
> > job?
> > >
> > > J-D
> > >
> > > On Mon, Nov 30, 2009 at 2:33 PM, Calvin <ca...@gmail.com>
> wrote:
> > >> I have a large amount of sequential ordered rows I would like to write
> > to an
> > >> HBase table.  What is the preferred way to do bulk writes of
> > multi-column
> > >> tables in HBase?  Using the get/put interface seems fairly slow even
> if
> > I
> > >> bulk writes with table.put(List<Put>).
> > >>
> > >> I have followed the directions on:
> > >>   * http://wiki.apache.org/hadoop/PerformanceTuning
> > >>   *
> > >>
> >
> http://ryantwopointoh.blogspot.com/2009/01/performance-of-hbase-importing.html
> > >>
> > >> Are there any other resources for improving the throughput of my bulk
> > >> writes?  On
> > >>
> >
> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/mapreduce/package-summary.htmlI
> > >> see there's a way to write HFiles directly, but HFileOutputFormat can
> > >> only
> > >> write a single column famly at a time (
> > >> https://issues.apache.org/jira/browse/HBASE-1861).
> > >>
> > >> Thanks!
> > >>
> > >> -Calvin
> > >>
> > >
> >
>

Re: hbase bulk writes

Posted by Ryan Rawson <ry...@gmail.com>.
Seems a little slow, but you will have a hard time getting a single
thread/process to do more than 10k rows/sec. Make your write buffer
larger (20MB?), dont forget to flush it too on shutdown!  Use more
processes or threads.  The usual tricks.

Good luck!

On Mon, Nov 30, 2009 at 3:33 PM, Calvin <ca...@gmail.com> wrote:
> Thanks for the responses.  If I can avoid writing a map-reduce job that
> would be preferable (getting map-reduce to work with / depend on my existing
> infrastructure is turning out to be annoying).
>
> I have no good way of randomizing my dataset since it's a very large stream
> of sequential data (ordered by some key).  I have a fair number of column
> families (~25) and every column is a long or a double.  Having a standalone
> program that writes rows using the HTable / Put API seems to run at ~2-5000
> rows/sec, which seems ridiculously slow.  Is it possible I am doing
> something terribly wrong?
>
> -Calvin
>
> On Mon, Nov 30, 2009 at 5:47 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> Sequentially ordered rows is the worst insert case in HBase - you end
>> up writing all to 1 server even if you have 500.  If you could
>> randomize your input, and I have pasted a Randomize.java map reduce
>> that will randomize lines of a file, then your performance will
>> improve.
>>
>> I have seen sustained inserts of 100-300k rows/sec on small rows
>> before.  Obviously large blob rows will be slower, since the limiting
>> factor is how fast we can write data to HDFS, thus it isnt the actual
>> row count, but the amount of data involved.
>>
>> Try the randomize.java, see where that gets you. I think it's on the
>> list archives.
>>
>> -ryan
>>
>>
>> On Mon, Nov 30, 2009 at 2:41 PM, Jean-Daniel Cryans <jd...@apache.org>
>> wrote:
>> > Could you put your data in HDFS and load it from there with a MapReduce
>> job?
>> >
>> > J-D
>> >
>> > On Mon, Nov 30, 2009 at 2:33 PM, Calvin <ca...@gmail.com> wrote:
>> >> I have a large amount of sequential ordered rows I would like to write
>> to an
>> >> HBase table.  What is the preferred way to do bulk writes of
>> multi-column
>> >> tables in HBase?  Using the get/put interface seems fairly slow even if
>> I
>> >> bulk writes with table.put(List<Put>).
>> >>
>> >> I have followed the directions on:
>> >>   * http://wiki.apache.org/hadoop/PerformanceTuning
>> >>   *
>> >>
>> http://ryantwopointoh.blogspot.com/2009/01/performance-of-hbase-importing.html
>> >>
>> >> Are there any other resources for improving the throughput of my bulk
>> >> writes?  On
>> >>
>> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/mapreduce/package-summary.htmlI
>> >> see there's a way to write HFiles directly, but HFileOutputFormat can
>> >> only
>> >> write a single column famly at a time (
>> >> https://issues.apache.org/jira/browse/HBASE-1861).
>> >>
>> >> Thanks!
>> >>
>> >> -Calvin
>> >>
>> >
>>
>