You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by S Ahmed <sa...@gmail.com> on 2010/07/22 20:22:48 UTC
java driver versus thrift
Can someone explain, at a high level, how the hbase service is exposed?
Is it a Java socket or? (sorry not that well versed in this)
Does anyone have any numbers on the performance differences between using
the native java driver (that presumably connects 'directly') versus the
Thrift route?
Re: java driver versus thrift
Posted by Ryan Rawson <ry...@gmail.com>.
Hi,
The situation is fairly complex and there is no super clear answer.
Here are some facts:
- Thrift requires the use of a shared-infrastructure client/server app
that is another scaling factor
- Thrift servers live a long time and thus can more effective amortize
the HTable cache across multiple short client runs
- Thrift servers dont have as advanced batch put and you could run
into scaling issues
- The Java API parallelizes as the number of client JVMs you have -
there is no real limit (other than your cluster's ability to handle
the requests)
- The API is probably the way to go if you are in Java
- Stumbleupon uses the Thrift API with PHP and it works like a charm.
The overhead it adds is surprisingly negligible. We deploy thrift
servers on all the regionservers
On Thu, Jul 22, 2010 at 2:18 PM, Sylvain Hellegouarch <sh...@defuze.org> wrote:
> On Thu, Jul 22, 2010 at 8:22 PM, S Ahmed <sa...@gmail.com> wrote:
>
>> Can someone explain, at a high level, how the hbase service is exposed?
>>
>> Is it a Java socket or? (sorry not that well versed in this)
>>
>>
>> Does anyone have any numbers on the performance differences between using
>> the native java driver (that presumably connects 'directly') versus the
>> Thrift route?
>>
>
>
> Basically when you use the thrift API, you're conversing with a thrift
> server that is written using the Java API. In other words, it's similar to
> some RPC mechanism, which means you'll introduce some overhead over using
> directly the Java API.
>
> I don't have numbers at hand unfortunately.
>
> --
> - Sylvain
> http://www.defuze.org
> http://twitter.com/lawouach
>
Re: java driver versus thrift
Posted by Sylvain Hellegouarch <sh...@defuze.org>.
On Thu, Jul 22, 2010 at 8:22 PM, S Ahmed <sa...@gmail.com> wrote:
> Can someone explain, at a high level, how the hbase service is exposed?
>
> Is it a Java socket or? (sorry not that well versed in this)
>
>
> Does anyone have any numbers on the performance differences between using
> the native java driver (that presumably connects 'directly') versus the
> Thrift route?
>
Basically when you use the thrift API, you're conversing with a thrift
server that is written using the Java API. In other words, it's similar to
some RPC mechanism, which means you'll introduce some overhead over using
directly the Java API.
I don't have numbers at hand unfortunately.
--
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach
Re: HBase performace & bulk load
Posted by Jean-Daniel Cryans <jd...@apache.org>.
HBase uses a write ahead log, if you don't write to it you will lose
all the data that's only in the memstores when a region server fails.
Useful for importing a lot of data.
J-D
On Fri, Jul 23, 2010 at 11:33 AM, Han Liu <ha...@andrew.cmu.edu> wrote:
> Hi SJ,
>
> Awesome setup. I tested with your configurations and the performance is 6 times better. :) Thanks a lot.
> What role does 'setWriteToWAL(false)' play here? I hear that if it's set to false then there will be data loss in case of RegionServer crash. How would the performance be affected if I set it to true?
>
> Thanks,
>
> Han
> On Jul 23, 2010, at 9:57 AM, Samuru Jackson wrote:
>
>> Hi,
>>
>> For testing purposes I have to make some bulk loads as well.
>>
>> What I do is to insert the data in bulks (for instance 10.000 rows every time).
>>
>> I create a Put List out of those records:
>>
>> List<Put> pList = new ArrayList<Put>();
>>
>> where each Put has WriteToWAL set to false;
>>
>> put.setWriteToWAL(false);
>> pList.add(p);
>>
>> Then I set autoflush to false and create a larger writebuffer:
>>
>> hTable.setAutoFlush(false);
>> hTable.setWriteBufferSize(1024*1024*12);
>> hTable.put(pList);
>> hTable.setAutoFlush(true);
>>
>> The following settings have boosted my load performance 5times -
>> without any bigger performance tunings, no special HW configuration I
>> achieve 8000-9000 records per second:
>> p.setWriteToWAL(false);
>> hTable.setAutoFlush(false);
>> hTable.setWriteBufferSize(1024*1024*12);
>>
>>
>> /SJ
>>
>>
>> On Thu, Jul 22, 2010 at 6:31 PM, Jean-Daniel Cryans <jd...@apache.org> wrote:
>>> Yes, then you should really look at using the write buffer.
>>>
>>> J-D
>>>
>>> On Thu, Jul 22, 2010 at 3:22 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
>>>> Thanks J-D.
>>>>
>>>> The only place where I create an HTable is in the constructor of my Mapper. The constructor is called only once for each map task right?
>>>>
>>>> Han
>>>> On Jul 22, 2010, at 4:43 PM, Jean-Daniel Cryans wrote:
>>>>
>>>>> Han,
>>>>>
>>>>> This is bad, you must be doing something slow like creating a new
>>>>> HTable for each put call. Also you need to use the write buffer
>>>>> (disable auto flushing, then set the write buffer size on HTable
>>>>> during the map configuration) if since you manage the HTable yourself.
>>>>>
>>>>> The bulk load tool usage is wide-spread, you should give it a try if
>>>>> you only have 1 family.
>>>>>
>>>>> J-D
>>>>>
>>>>> On Thu, Jul 22, 2010 at 1:06 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
>>>>>> Hi Guys,
>>>>>>
>>>>>> I've been doing some data insertion from HDFS to HBase and the performance seems to be really bad. It took about 3 hours to insert 15 GB of data. The mapreduce job is launched from one machine which grabs data from HDFS and insert them into an HTable located at 3 other machines (1 master and 2 regionservers). There are 17 map job in total (no reduce jobs), representing 17 files each about 1GB in size. The mapper simply extracts the useful information from each of these files and insert them into HBase. In the end there are about 22 million rows added in the table, and with my implementation (pretty low-efficient I think), for each of these row a 'table.put(Put p)' method is called once, so in the end there are 22 million 'table.put()' calls.
>>>>>>
>>>>>> Does it make sense that these many 'table.put' calls talks 3 hours? Because I have played with my code and I have determined that the bottleneck is these 'table.put()' calls, because if I remove them, the rest of the code (doing every part of the job except for committing the updates via 'table.put()' )only takes 2 minutes to run. I am really inexperienced in HBase, so how do you guys usually do data insertion? What could be the tricks to enhance performance?
>>>>>>
>>>>>> I am thinking about using the bulk load feature to batch insert data into HBase. Is this a popular method out there in the HBase community?
>>>>>>
>>>>>> Really sorry about asking so much help for my problems but not helping other people with theirs. I really would like to offer help once I get more experienced with HBase.
>>>>>>
>>>>>> Thanks a lot in advance :)
>>>>>>
>>>>>>
>>>>>> ----
>>>>>> Han Liu
>>>>>> SCS & HCI Institute
>>>>>> Undergrad. Class of 2012
>>>>>> Carnegie Mellon University
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Han Liu
>>>> SCS & HCI Institute
>>>> Undergrad. Class of 2012
>>>> Carnegie Mellon University
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
> Han Liu
> SCS & HCI Institute
> Undergrad. Class of 2012
> Carnegie Mellon University
>
>
>
>
Re: HBase performace & bulk load
Posted by Han Liu <ha...@andrew.cmu.edu>.
Hi SJ,
Awesome setup. I tested with your configurations and the performance is 6 times better. :) Thanks a lot.
What role does 'setWriteToWAL(false)' play here? I hear that if it's set to false then there will be data loss in case of RegionServer crash. How would the performance be affected if I set it to true?
Thanks,
Han
On Jul 23, 2010, at 9:57 AM, Samuru Jackson wrote:
> Hi,
>
> For testing purposes I have to make some bulk loads as well.
>
> What I do is to insert the data in bulks (for instance 10.000 rows every time).
>
> I create a Put List out of those records:
>
> List<Put> pList = new ArrayList<Put>();
>
> where each Put has WriteToWAL set to false;
>
> put.setWriteToWAL(false);
> pList.add(p);
>
> Then I set autoflush to false and create a larger writebuffer:
>
> hTable.setAutoFlush(false);
> hTable.setWriteBufferSize(1024*1024*12);
> hTable.put(pList);
> hTable.setAutoFlush(true);
>
> The following settings have boosted my load performance 5times -
> without any bigger performance tunings, no special HW configuration I
> achieve 8000-9000 records per second:
> p.setWriteToWAL(false);
> hTable.setAutoFlush(false);
> hTable.setWriteBufferSize(1024*1024*12);
>
>
> /SJ
>
>
> On Thu, Jul 22, 2010 at 6:31 PM, Jean-Daniel Cryans <jd...@apache.org> wrote:
>> Yes, then you should really look at using the write buffer.
>>
>> J-D
>>
>> On Thu, Jul 22, 2010 at 3:22 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
>>> Thanks J-D.
>>>
>>> The only place where I create an HTable is in the constructor of my Mapper. The constructor is called only once for each map task right?
>>>
>>> Han
>>> On Jul 22, 2010, at 4:43 PM, Jean-Daniel Cryans wrote:
>>>
>>>> Han,
>>>>
>>>> This is bad, you must be doing something slow like creating a new
>>>> HTable for each put call. Also you need to use the write buffer
>>>> (disable auto flushing, then set the write buffer size on HTable
>>>> during the map configuration) if since you manage the HTable yourself.
>>>>
>>>> The bulk load tool usage is wide-spread, you should give it a try if
>>>> you only have 1 family.
>>>>
>>>> J-D
>>>>
>>>> On Thu, Jul 22, 2010 at 1:06 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
>>>>> Hi Guys,
>>>>>
>>>>> I've been doing some data insertion from HDFS to HBase and the performance seems to be really bad. It took about 3 hours to insert 15 GB of data. The mapreduce job is launched from one machine which grabs data from HDFS and insert them into an HTable located at 3 other machines (1 master and 2 regionservers). There are 17 map job in total (no reduce jobs), representing 17 files each about 1GB in size. The mapper simply extracts the useful information from each of these files and insert them into HBase. In the end there are about 22 million rows added in the table, and with my implementation (pretty low-efficient I think), for each of these row a 'table.put(Put p)' method is called once, so in the end there are 22 million 'table.put()' calls.
>>>>>
>>>>> Does it make sense that these many 'table.put' calls talks 3 hours? Because I have played with my code and I have determined that the bottleneck is these 'table.put()' calls, because if I remove them, the rest of the code (doing every part of the job except for committing the updates via 'table.put()' )only takes 2 minutes to run. I am really inexperienced in HBase, so how do you guys usually do data insertion? What could be the tricks to enhance performance?
>>>>>
>>>>> I am thinking about using the bulk load feature to batch insert data into HBase. Is this a popular method out there in the HBase community?
>>>>>
>>>>> Really sorry about asking so much help for my problems but not helping other people with theirs. I really would like to offer help once I get more experienced with HBase.
>>>>>
>>>>> Thanks a lot in advance :)
>>>>>
>>>>>
>>>>> ----
>>>>> Han Liu
>>>>> SCS & HCI Institute
>>>>> Undergrad. Class of 2012
>>>>> Carnegie Mellon University
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> Han Liu
>>> SCS & HCI Institute
>>> Undergrad. Class of 2012
>>> Carnegie Mellon University
>>>
>>>
>>>
>>>
>>
>
Han Liu
SCS & HCI Institute
Undergrad. Class of 2012
Carnegie Mellon University
Re: HBase performace & bulk load
Posted by Samuru Jackson <sa...@googlemail.com>.
Hi,
For testing purposes I have to make some bulk loads as well.
What I do is to insert the data in bulks (for instance 10.000 rows every time).
I create a Put List out of those records:
List<Put> pList = new ArrayList<Put>();
where each Put has WriteToWAL set to false;
put.setWriteToWAL(false);
pList.add(p);
Then I set autoflush to false and create a larger writebuffer:
hTable.setAutoFlush(false);
hTable.setWriteBufferSize(1024*1024*12);
hTable.put(pList);
hTable.setAutoFlush(true);
The following settings have boosted my load performance 5times -
without any bigger performance tunings, no special HW configuration I
achieve 8000-9000 records per second:
p.setWriteToWAL(false);
hTable.setAutoFlush(false);
hTable.setWriteBufferSize(1024*1024*12);
/SJ
On Thu, Jul 22, 2010 at 6:31 PM, Jean-Daniel Cryans <jd...@apache.org> wrote:
> Yes, then you should really look at using the write buffer.
>
> J-D
>
> On Thu, Jul 22, 2010 at 3:22 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
>> Thanks J-D.
>>
>> The only place where I create an HTable is in the constructor of my Mapper. The constructor is called only once for each map task right?
>>
>> Han
>> On Jul 22, 2010, at 4:43 PM, Jean-Daniel Cryans wrote:
>>
>>> Han,
>>>
>>> This is bad, you must be doing something slow like creating a new
>>> HTable for each put call. Also you need to use the write buffer
>>> (disable auto flushing, then set the write buffer size on HTable
>>> during the map configuration) if since you manage the HTable yourself.
>>>
>>> The bulk load tool usage is wide-spread, you should give it a try if
>>> you only have 1 family.
>>>
>>> J-D
>>>
>>> On Thu, Jul 22, 2010 at 1:06 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
>>>> Hi Guys,
>>>>
>>>> I've been doing some data insertion from HDFS to HBase and the performance seems to be really bad. It took about 3 hours to insert 15 GB of data. The mapreduce job is launched from one machine which grabs data from HDFS and insert them into an HTable located at 3 other machines (1 master and 2 regionservers). There are 17 map job in total (no reduce jobs), representing 17 files each about 1GB in size. The mapper simply extracts the useful information from each of these files and insert them into HBase. In the end there are about 22 million rows added in the table, and with my implementation (pretty low-efficient I think), for each of these row a 'table.put(Put p)' method is called once, so in the end there are 22 million 'table.put()' calls.
>>>>
>>>> Does it make sense that these many 'table.put' calls talks 3 hours? Because I have played with my code and I have determined that the bottleneck is these 'table.put()' calls, because if I remove them, the rest of the code (doing every part of the job except for committing the updates via 'table.put()' )only takes 2 minutes to run. I am really inexperienced in HBase, so how do you guys usually do data insertion? What could be the tricks to enhance performance?
>>>>
>>>> I am thinking about using the bulk load feature to batch insert data into HBase. Is this a popular method out there in the HBase community?
>>>>
>>>> Really sorry about asking so much help for my problems but not helping other people with theirs. I really would like to offer help once I get more experienced with HBase.
>>>>
>>>> Thanks a lot in advance :)
>>>>
>>>>
>>>> ----
>>>> Han Liu
>>>> SCS & HCI Institute
>>>> Undergrad. Class of 2012
>>>> Carnegie Mellon University
>>>>
>>>>
>>>>
>>>>
>>>
>>
>> Han Liu
>> SCS & HCI Institute
>> Undergrad. Class of 2012
>> Carnegie Mellon University
>>
>>
>>
>>
>
Re: HBase performace & bulk load
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yes, then you should really look at using the write buffer.
J-D
On Thu, Jul 22, 2010 at 3:22 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
> Thanks J-D.
>
> The only place where I create an HTable is in the constructor of my Mapper. The constructor is called only once for each map task right?
>
> Han
> On Jul 22, 2010, at 4:43 PM, Jean-Daniel Cryans wrote:
>
>> Han,
>>
>> This is bad, you must be doing something slow like creating a new
>> HTable for each put call. Also you need to use the write buffer
>> (disable auto flushing, then set the write buffer size on HTable
>> during the map configuration) if since you manage the HTable yourself.
>>
>> The bulk load tool usage is wide-spread, you should give it a try if
>> you only have 1 family.
>>
>> J-D
>>
>> On Thu, Jul 22, 2010 at 1:06 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
>>> Hi Guys,
>>>
>>> I've been doing some data insertion from HDFS to HBase and the performance seems to be really bad. It took about 3 hours to insert 15 GB of data. The mapreduce job is launched from one machine which grabs data from HDFS and insert them into an HTable located at 3 other machines (1 master and 2 regionservers). There are 17 map job in total (no reduce jobs), representing 17 files each about 1GB in size. The mapper simply extracts the useful information from each of these files and insert them into HBase. In the end there are about 22 million rows added in the table, and with my implementation (pretty low-efficient I think), for each of these row a 'table.put(Put p)' method is called once, so in the end there are 22 million 'table.put()' calls.
>>>
>>> Does it make sense that these many 'table.put' calls talks 3 hours? Because I have played with my code and I have determined that the bottleneck is these 'table.put()' calls, because if I remove them, the rest of the code (doing every part of the job except for committing the updates via 'table.put()' )only takes 2 minutes to run. I am really inexperienced in HBase, so how do you guys usually do data insertion? What could be the tricks to enhance performance?
>>>
>>> I am thinking about using the bulk load feature to batch insert data into HBase. Is this a popular method out there in the HBase community?
>>>
>>> Really sorry about asking so much help for my problems but not helping other people with theirs. I really would like to offer help once I get more experienced with HBase.
>>>
>>> Thanks a lot in advance :)
>>>
>>>
>>> ----
>>> Han Liu
>>> SCS & HCI Institute
>>> Undergrad. Class of 2012
>>> Carnegie Mellon University
>>>
>>>
>>>
>>>
>>
>
> Han Liu
> SCS & HCI Institute
> Undergrad. Class of 2012
> Carnegie Mellon University
>
>
>
>
Re: HBase performace & bulk load
Posted by HAN LIU <ha...@andrew.cmu.edu>.
Thanks J-D.
The only place where I create an HTable is in the constructor of my Mapper. The constructor is called only once for each map task right?
Han
On Jul 22, 2010, at 4:43 PM, Jean-Daniel Cryans wrote:
> Han,
>
> This is bad, you must be doing something slow like creating a new
> HTable for each put call. Also you need to use the write buffer
> (disable auto flushing, then set the write buffer size on HTable
> during the map configuration) if since you manage the HTable yourself.
>
> The bulk load tool usage is wide-spread, you should give it a try if
> you only have 1 family.
>
> J-D
>
> On Thu, Jul 22, 2010 at 1:06 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
>> Hi Guys,
>>
>> I've been doing some data insertion from HDFS to HBase and the performance seems to be really bad. It took about 3 hours to insert 15 GB of data. The mapreduce job is launched from one machine which grabs data from HDFS and insert them into an HTable located at 3 other machines (1 master and 2 regionservers). There are 17 map job in total (no reduce jobs), representing 17 files each about 1GB in size. The mapper simply extracts the useful information from each of these files and insert them into HBase. In the end there are about 22 million rows added in the table, and with my implementation (pretty low-efficient I think), for each of these row a 'table.put(Put p)' method is called once, so in the end there are 22 million 'table.put()' calls.
>>
>> Does it make sense that these many 'table.put' calls talks 3 hours? Because I have played with my code and I have determined that the bottleneck is these 'table.put()' calls, because if I remove them, the rest of the code (doing every part of the job except for committing the updates via 'table.put()' )only takes 2 minutes to run. I am really inexperienced in HBase, so how do you guys usually do data insertion? What could be the tricks to enhance performance?
>>
>> I am thinking about using the bulk load feature to batch insert data into HBase. Is this a popular method out there in the HBase community?
>>
>> Really sorry about asking so much help for my problems but not helping other people with theirs. I really would like to offer help once I get more experienced with HBase.
>>
>> Thanks a lot in advance :)
>>
>>
>> ----
>> Han Liu
>> SCS & HCI Institute
>> Undergrad. Class of 2012
>> Carnegie Mellon University
>>
>>
>>
>>
>
Han Liu
SCS & HCI Institute
Undergrad. Class of 2012
Carnegie Mellon University
Re: HBase performace & bulk load
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Han,
This is bad, you must be doing something slow like creating a new
HTable for each put call. Also you need to use the write buffer
(disable auto flushing, then set the write buffer size on HTable
during the map configuration) if since you manage the HTable yourself.
The bulk load tool usage is wide-spread, you should give it a try if
you only have 1 family.
J-D
On Thu, Jul 22, 2010 at 1:06 PM, HAN LIU <ha...@andrew.cmu.edu> wrote:
> Hi Guys,
>
> I've been doing some data insertion from HDFS to HBase and the performance seems to be really bad. It took about 3 hours to insert 15 GB of data. The mapreduce job is launched from one machine which grabs data from HDFS and insert them into an HTable located at 3 other machines (1 master and 2 regionservers). There are 17 map job in total (no reduce jobs), representing 17 files each about 1GB in size. The mapper simply extracts the useful information from each of these files and insert them into HBase. In the end there are about 22 million rows added in the table, and with my implementation (pretty low-efficient I think), for each of these row a 'table.put(Put p)' method is called once, so in the end there are 22 million 'table.put()' calls.
>
> Does it make sense that these many 'table.put' calls talks 3 hours? Because I have played with my code and I have determined that the bottleneck is these 'table.put()' calls, because if I remove them, the rest of the code (doing every part of the job except for committing the updates via 'table.put()' )only takes 2 minutes to run. I am really inexperienced in HBase, so how do you guys usually do data insertion? What could be the tricks to enhance performance?
>
> I am thinking about using the bulk load feature to batch insert data into HBase. Is this a popular method out there in the HBase community?
>
> Really sorry about asking so much help for my problems but not helping other people with theirs. I really would like to offer help once I get more experienced with HBase.
>
> Thanks a lot in advance :)
>
>
> ----
> Han Liu
> SCS & HCI Institute
> Undergrad. Class of 2012
> Carnegie Mellon University
>
>
>
>
HBase performace & bulk load
Posted by HAN LIU <ha...@andrew.cmu.edu>.
Hi Guys,
I've been doing some data insertion from HDFS to HBase and the performance seems to be really bad. It took about 3 hours to insert 15 GB of data. The mapreduce job is launched from one machine which grabs data from HDFS and insert them into an HTable located at 3 other machines (1 master and 2 regionservers). There are 17 map job in total (no reduce jobs), representing 17 files each about 1GB in size. The mapper simply extracts the useful information from each of these files and insert them into HBase. In the end there are about 22 million rows added in the table, and with my implementation (pretty low-efficient I think), for each of these row a 'table.put(Put p)' method is called once, so in the end there are 22 million 'table.put()' calls.
Does it make sense that these many 'table.put' calls talks 3 hours? Because I have played with my code and I have determined that the bottleneck is these 'table.put()' calls, because if I remove them, the rest of the code (doing every part of the job except for committing the updates via 'table.put()' )only takes 2 minutes to run. I am really inexperienced in HBase, so how do you guys usually do data insertion? What could be the tricks to enhance performance?
I am thinking about using the bulk load feature to batch insert data into HBase. Is this a popular method out there in the HBase community?
Really sorry about asking so much help for my problems but not helping other people with theirs. I really would like to offer help once I get more experienced with HBase.
Thanks a lot in advance :)
----
Han Liu
SCS & HCI Institute
Undergrad. Class of 2012
Carnegie Mellon University