You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by samar kumar <sa...@gmail.com> on 2012/06/27 12:49:19 UTC
direct Hfile Read and Writes
Hi Hbase Users,
I have seen API's supporting HFile direct reads and write. I Do understand
it would create Hfiles in the location specified and it should be much
faster since we would skip all the look ups to ZK. catalog table . RS , but
can anyone point me to a particular case when we would like to read/write
directly .
1. Since the data we might need would be distributed across regions how
would direct reading of Hfile be helpful.
2. Any use-case for direct writes of Hfiles. If we write Hfiles will
that data be accessible to the hbase shell.
Regards,
Samar
Re: direct Hfile Read and Writes
Posted by Stack <st...@duboce.net>.
On Thu, Jun 28, 2012 at 1:40 AM, samar kumar <sa...@gmail.com> wrote:
> Thanks for the replies . I am aware of the apis but can anyone give me
> little bit more insight on the details. After creating the HFiles and
> calling IncrementalLoadHFile how does it internally change the RS, Catalog
> tables etc?
> Can anyone explain the flow.
It does not change catalog tables; catalog table only has region
layout, not what files make up a region. The loading of an hfile into
a region just atomically updates the set of hfiles the currently open
region is using swapping in the new one.
Study the bulk load tool though. Notice how it will make the
proffered set of hfiles fit the current region layout splitting hfiles
if necessary so they fit w/i the current region confines. See how it
also threads the updating because update is slow done serially. Back
up more to see how the files are created using total order
partitioner. Leverage what bulk loader does if you want to avoid long
solved issues.
St.Ack
Re: direct Hfile Read and Writes
Posted by samar kumar <sa...@gmail.com>.
Thanks for the replies . I am aware of the apis but can anyone give me
little bit more insight on the details. After creating the HFiles and
calling IncrementalLoadHFile how does it internally change the RS, Catalog
tables etc?
Can anyone explain the flow.
Thanks ,
Samar
On Thu, Jun 28, 2012 at 9:07 AM, Anoop Sam John <an...@huawei.com> wrote:
> When there is a need of bulk loading huge amount of data into HBase at one
> time, it will be better go with the direct HFile write.
> Here 1st using the MR framework HFiles are directly written (Into HDFS)..
> For this HBase provides the utility classes and the ImportTSV tool itself.
> Then using the IncrementalLoadHFile , these files are loaded into the
> regions managed by RS.
> Once these 2 steps are over client can read the data normally.
> For loading these much data in a normal way of HTable#put() will take lot
> of time.
>
>
> -Anoop-
> ________________________________________
> From: Jerry Lam [chilinglam@gmail.com]
> Sent: Wednesday, June 27, 2012 10:52 PM
> To: user@hbase.apache.org
> Subject: Re: direct Hfile Read and Writes
>
> Hi Samar:
>
> I have used IncrementalLoadHFile successfully in the past. Basically, once
> you have written hfile youreself you can use the IncrementalLoadHFile to
> merge with the HFile currently managed by HBase. Once it is loaded to
> HBase, the records in the increment hfile are accessible by clients.
>
> HTH,
>
> Jerry
>
> On Wed, Jun 27, 2012 at 10:33 AM, shixing <pa...@gmail.com> wrote:
>
> > 1. Since the data we might need would be distributed across regions how
> > would direct reading of Hfile be helpful.
> >
> > You can read the HFilePrettyPrinter, it shows how to create a
> HFile.Reader
> > and use it to read the HFile.
> > Or you can use the ./hbase org.apache.hadoop.hbase.io.hfile.HFile -p -f
> > hdfs://xxxx/xxx/hfile to print some info to have a look.
> >
> > 2. Any use-case for direct writes of Hfiles. If we write Hfiles will
> > that data be accessible to the hbase shell.
> >
> > You can read the HFileOutputFormat, it shows how to create a HFile.Writer
> > and use it to directly write kvs the HFile.
> > If you want to read the data by hbase shell, you should firstly load the
> > HFile to regionservers, details for bulkload
> > http://hbase.apache.org/book.html#arch.bulk.load .
> >
> >
> > On Wed, Jun 27, 2012 at 6:49 PM, samar kumar <samar.opensource@gmail.com
> > >wrote:
> >
> > > Hi Hbase Users,
> > > I have seen API's supporting HFile direct reads and write. I Do
> > understand
> > > it would create Hfiles in the location specified and it should be much
> > > faster since we would skip all the look ups to ZK. catalog table . RS ,
> > but
> > > can anyone point me to a particular case when we would like to
> read/write
> > > directly .
> > >
> > >
> > > 1. Since the data we might need would be distributed across regions
> how
> > > would direct reading of Hfile be helpful.
> > > 2. Any use-case for direct writes of Hfiles. If we write Hfiles will
> > > that data be accessible to the hbase shell.
> > >
> > >
> > > Regards,
> > > Samar
> > >
> >
> >
> >
> > --
> > Best wishes!
> > My Friend~
> >
>
RE: direct Hfile Read and Writes
Posted by Anoop Sam John <an...@huawei.com>.
When there is a need of bulk loading huge amount of data into HBase at one time, it will be better go with the direct HFile write.
Here 1st using the MR framework HFiles are directly written (Into HDFS).. For this HBase provides the utility classes and the ImportTSV tool itself.
Then using the IncrementalLoadHFile , these files are loaded into the regions managed by RS.
Once these 2 steps are over client can read the data normally.
For loading these much data in a normal way of HTable#put() will take lot of time.
-Anoop-
________________________________________
From: Jerry Lam [chilinglam@gmail.com]
Sent: Wednesday, June 27, 2012 10:52 PM
To: user@hbase.apache.org
Subject: Re: direct Hfile Read and Writes
Hi Samar:
I have used IncrementalLoadHFile successfully in the past. Basically, once
you have written hfile youreself you can use the IncrementalLoadHFile to
merge with the HFile currently managed by HBase. Once it is loaded to
HBase, the records in the increment hfile are accessible by clients.
HTH,
Jerry
On Wed, Jun 27, 2012 at 10:33 AM, shixing <pa...@gmail.com> wrote:
> 1. Since the data we might need would be distributed across regions how
> would direct reading of Hfile be helpful.
>
> You can read the HFilePrettyPrinter, it shows how to create a HFile.Reader
> and use it to read the HFile.
> Or you can use the ./hbase org.apache.hadoop.hbase.io.hfile.HFile -p -f
> hdfs://xxxx/xxx/hfile to print some info to have a look.
>
> 2. Any use-case for direct writes of Hfiles. If we write Hfiles will
> that data be accessible to the hbase shell.
>
> You can read the HFileOutputFormat, it shows how to create a HFile.Writer
> and use it to directly write kvs the HFile.
> If you want to read the data by hbase shell, you should firstly load the
> HFile to regionservers, details for bulkload
> http://hbase.apache.org/book.html#arch.bulk.load .
>
>
> On Wed, Jun 27, 2012 at 6:49 PM, samar kumar <samar.opensource@gmail.com
> >wrote:
>
> > Hi Hbase Users,
> > I have seen API's supporting HFile direct reads and write. I Do
> understand
> > it would create Hfiles in the location specified and it should be much
> > faster since we would skip all the look ups to ZK. catalog table . RS ,
> but
> > can anyone point me to a particular case when we would like to read/write
> > directly .
> >
> >
> > 1. Since the data we might need would be distributed across regions how
> > would direct reading of Hfile be helpful.
> > 2. Any use-case for direct writes of Hfiles. If we write Hfiles will
> > that data be accessible to the hbase shell.
> >
> >
> > Regards,
> > Samar
> >
>
>
>
> --
> Best wishes!
> My Friend~
>
Re: direct Hfile Read and Writes
Posted by Jerry Lam <ch...@gmail.com>.
Hi Samar:
I have used IncrementalLoadHFile successfully in the past. Basically, once
you have written hfile youreself you can use the IncrementalLoadHFile to
merge with the HFile currently managed by HBase. Once it is loaded to
HBase, the records in the increment hfile are accessible by clients.
HTH,
Jerry
On Wed, Jun 27, 2012 at 10:33 AM, shixing <pa...@gmail.com> wrote:
> 1. Since the data we might need would be distributed across regions how
> would direct reading of Hfile be helpful.
>
> You can read the HFilePrettyPrinter, it shows how to create a HFile.Reader
> and use it to read the HFile.
> Or you can use the ./hbase org.apache.hadoop.hbase.io.hfile.HFile -p -f
> hdfs://xxxx/xxx/hfile to print some info to have a look.
>
> 2. Any use-case for direct writes of Hfiles. If we write Hfiles will
> that data be accessible to the hbase shell.
>
> You can read the HFileOutputFormat, it shows how to create a HFile.Writer
> and use it to directly write kvs the HFile.
> If you want to read the data by hbase shell, you should firstly load the
> HFile to regionservers, details for bulkload
> http://hbase.apache.org/book.html#arch.bulk.load .
>
>
> On Wed, Jun 27, 2012 at 6:49 PM, samar kumar <samar.opensource@gmail.com
> >wrote:
>
> > Hi Hbase Users,
> > I have seen API's supporting HFile direct reads and write. I Do
> understand
> > it would create Hfiles in the location specified and it should be much
> > faster since we would skip all the look ups to ZK. catalog table . RS ,
> but
> > can anyone point me to a particular case when we would like to read/write
> > directly .
> >
> >
> > 1. Since the data we might need would be distributed across regions how
> > would direct reading of Hfile be helpful.
> > 2. Any use-case for direct writes of Hfiles. If we write Hfiles will
> > that data be accessible to the hbase shell.
> >
> >
> > Regards,
> > Samar
> >
>
>
>
> --
> Best wishes!
> My Friend~
>
Re: direct Hfile Read and Writes
Posted by shixing <pa...@gmail.com>.
1. Since the data we might need would be distributed across regions how
would direct reading of Hfile be helpful.
You can read the HFilePrettyPrinter, it shows how to create a HFile.Reader
and use it to read the HFile.
Or you can use the ./hbase org.apache.hadoop.hbase.io.hfile.HFile -p -f
hdfs://xxxx/xxx/hfile to print some info to have a look.
2. Any use-case for direct writes of Hfiles. If we write Hfiles will
that data be accessible to the hbase shell.
You can read the HFileOutputFormat, it shows how to create a HFile.Writer
and use it to directly write kvs the HFile.
If you want to read the data by hbase shell, you should firstly load the
HFile to regionservers, details for bulkload
http://hbase.apache.org/book.html#arch.bulk.load .
On Wed, Jun 27, 2012 at 6:49 PM, samar kumar <sa...@gmail.com>wrote:
> Hi Hbase Users,
> I have seen API's supporting HFile direct reads and write. I Do understand
> it would create Hfiles in the location specified and it should be much
> faster since we would skip all the look ups to ZK. catalog table . RS , but
> can anyone point me to a particular case when we would like to read/write
> directly .
>
>
> 1. Since the data we might need would be distributed across regions how
> would direct reading of Hfile be helpful.
> 2. Any use-case for direct writes of Hfiles. If we write Hfiles will
> that data be accessible to the hbase shell.
>
>
> Regards,
> Samar
>
--
Best wishes!
My Friend~