You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ey-Chih chow <ey...@gmail.com> on 2012/05/25 20:03:16 UTC
improve performance of a MapReduce job with HBase input
Hi,
We have a MapReduce job of which input data is from HBase. We would like to improve performance of the job. According to the HBase book, we can do that by setting scan caching to a number higher than default. We use TableInputFormat to read data from the job. I look at the implementation of the class. The class does not set caching when a scan object is created. Is there anybody know how to externally set caching for the scan created in TableInputFormat? Thanks.
Ey-Chih Chow
Re: improve performance of a MapReduce job with HBase input
Posted by Ey-Chih chow <ey...@gmail.com>.
Thanks. This help.
Ey-Chih Chow
On May 25, 2012, at 11:23 AM, Dave Revell wrote:
> Here's what I do:
>
> Scan scan = new Scan(...)
> scan.setCaching(5000);
> scan.setWhatever(...);
>
> TableMapReduceUtil.initTableMapperJob(tablename, scan, mapClass,
> mapOutKeyClass, mapOutValueClass, job);
>
> Does that help?
>
> -Dave
>
> On Fri, May 25, 2012 at 11:03 AM, Ey-Chih chow <ey...@gmail.com> wrote:
>
>> Hi,
>>
>> We have a MapReduce job of which input data is from HBase. We would like
>> to improve performance of the job. According to the HBase book, we can do
>> that by setting scan caching to a number higher than default. We use
>> TableInputFormat to read data from the job. I look at the implementation
>> of the class. The class does not set caching when a scan object is
>> created. Is there anybody know how to externally set caching for the scan
>> created in TableInputFormat? Thanks.
>>
>> Ey-Chih Chow
Re: improve performance of a MapReduce job with HBase input
Posted by Dave Revell <da...@urbanairship.com>.
Here's what I do:
Scan scan = new Scan(...)
scan.setCaching(5000);
scan.setWhatever(...);
TableMapReduceUtil.initTableMapperJob(tablename, scan, mapClass,
mapOutKeyClass, mapOutValueClass, job);
Does that help?
-Dave
On Fri, May 25, 2012 at 11:03 AM, Ey-Chih chow <ey...@gmail.com> wrote:
> Hi,
>
> We have a MapReduce job of which input data is from HBase. We would like
> to improve performance of the job. According to the HBase book, we can do
> that by setting scan caching to a number higher than default. We use
> TableInputFormat to read data from the job. I look at the implementation
> of the class. The class does not set caching when a scan object is
> created. Is there anybody know how to externally set caching for the scan
> created in TableInputFormat? Thanks.
>
> Ey-Chih Chow
Re: improve performance of a MapReduce job with HBase input
Posted by Ey-Chih chow <ey...@gmail.com>.
Thanks. This works.
Ey-Chih Chow
On May 25, 2012, at 11:33 AM, Jean-Daniel Cryans wrote:
> TIF should be configured via TableMapReduceUtil.initTableMapperJob
> which takes a Scan object.
>
> J-D
>
> On Fri, May 25, 2012 at 11:30 AM, Ey-Chih chow <ey...@gmail.com> wrote:
>> Thanks. Since we use TableInputFormat in our map/reduce job. The scan object is created inside TableInputFormat. Is there any way to get the scan object to set caching?
>>
>> Ey-Chih Chow
>>
>> On May 25, 2012, at 11:24 AM, Alok Kumar wrote:
>>
>>> Hi,
>>>
>>> you can make use of 'setCaching' method of your scan object.
>>>
>>> Eg:
>>> Scan objScan = new Scan();
>>> objScan.setCaching(100); // set it to some integer, as per ur use case.
>>>
>>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setCaching(int)
>>>
>>> thanks,
>>> Alok
>>>
>>> On Fri, May 25, 2012 at 11:33 PM, Ey-Chih chow <ey...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have a MapReduce job of which input data is from HBase. We would like
>>>> to improve performance of the job. According to the HBase book, we can do
>>>> that by setting scan caching to a number higher than default. We use
>>>> TableInputFormat to read data from the job. I look at the implementation
>>>> of the class. The class does not set caching when a scan object is
>>>> created. Is there anybody know how to externally set caching for the scan
>>>> created in TableInputFormat? Thanks.
>>>>
>>>> Ey-Chih Chow
>>
Re: improve performance of a MapReduce job with HBase input
Posted by Jean-Daniel Cryans <jd...@apache.org>.
TIF should be configured via TableMapReduceUtil.initTableMapperJob
which takes a Scan object.
J-D
On Fri, May 25, 2012 at 11:30 AM, Ey-Chih chow <ey...@gmail.com> wrote:
> Thanks. Since we use TableInputFormat in our map/reduce job. The scan object is created inside TableInputFormat. Is there any way to get the scan object to set caching?
>
> Ey-Chih Chow
>
> On May 25, 2012, at 11:24 AM, Alok Kumar wrote:
>
>> Hi,
>>
>> you can make use of 'setCaching' method of your scan object.
>>
>> Eg:
>> Scan objScan = new Scan();
>> objScan.setCaching(100); // set it to some integer, as per ur use case.
>>
>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setCaching(int)
>>
>> thanks,
>> Alok
>>
>> On Fri, May 25, 2012 at 11:33 PM, Ey-Chih chow <ey...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We have a MapReduce job of which input data is from HBase. We would like
>>> to improve performance of the job. According to the HBase book, we can do
>>> that by setting scan caching to a number higher than default. We use
>>> TableInputFormat to read data from the job. I look at the implementation
>>> of the class. The class does not set caching when a scan object is
>>> created. Is there anybody know how to externally set caching for the scan
>>> created in TableInputFormat? Thanks.
>>>
>>> Ey-Chih Chow
>
Re: improve performance of a MapReduce job with HBase input
Posted by Ey-Chih chow <ey...@gmail.com>.
Thanks. Since we use TableInputFormat in our map/reduce job. The scan object is created inside TableInputFormat. Is there any way to get the scan object to set caching?
Ey-Chih Chow
On May 25, 2012, at 11:24 AM, Alok Kumar wrote:
> Hi,
>
> you can make use of 'setCaching' method of your scan object.
>
> Eg:
> Scan objScan = new Scan();
> objScan.setCaching(100); // set it to some integer, as per ur use case.
>
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setCaching(int)
>
> thanks,
> Alok
>
> On Fri, May 25, 2012 at 11:33 PM, Ey-Chih chow <ey...@gmail.com> wrote:
>
>> Hi,
>>
>> We have a MapReduce job of which input data is from HBase. We would like
>> to improve performance of the job. According to the HBase book, we can do
>> that by setting scan caching to a number higher than default. We use
>> TableInputFormat to read data from the job. I look at the implementation
>> of the class. The class does not set caching when a scan object is
>> created. Is there anybody know how to externally set caching for the scan
>> created in TableInputFormat? Thanks.
>>
>> Ey-Chih Chow
Re: improve performance of a MapReduce job with HBase input
Posted by Alok Kumar <al...@gmail.com>.
Hi,
you can make use of 'setCaching' method of your scan object.
Eg:
Scan objScan = new Scan();
objScan.setCaching(100); // set it to some integer, as per ur use case.
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#setCaching(int)
thanks,
Alok
On Fri, May 25, 2012 at 11:33 PM, Ey-Chih chow <ey...@gmail.com> wrote:
> Hi,
>
> We have a MapReduce job of which input data is from HBase. We would like
> to improve performance of the job. According to the HBase book, we can do
> that by setting scan caching to a number higher than default. We use
> TableInputFormat to read data from the job. I look at the implementation
> of the class. The class does not set caching when a scan object is
> created. Is there anybody know how to externally set caching for the scan
> created in TableInputFormat? Thanks.
>
> Ey-Chih Chow