You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dmitriy Ryaboy <dm...@twitter.com> on 2010/05/03 21:23:14 UTC
HBase / Pig integration
Hi folks,
I recently rewrote the Pig HBase loader to work with binary data, push down
filters, and do other things that make it more versatile.
If you use, or plan to use, both Pig and HBase, please try it out, take a
look at the code, let me know what you think. I am just starting to learn
about HBase, so I am especially interested to learn if there are HBase
capabilities I am not using and should be.
The code is part of our "ElephantBird" project, here:
http://github.com/kevinweil/elephant-bird/
and more specifically:
http://github.com/kevinweil/elephant-bird/tree/master/src/java/com/twitter/elephantbird/pig/load/
Thanks,
-Dmitriy
Re: HBase / Pig integration
Posted by Dmitriy Ryaboy <dm...@twitter.com>.
Stack,
Having an estimated row count per region would be great. Should I open a
ticket for that?
I'll check out the HBase counters.
Now that you mention it, we do set LZO compression for our tables, I just
forgot it's a manual thing :).
-Dmitriy
On Tue, May 4, 2010 at 7:34 AM, Stack <st...@duboce.net> wrote:
> On Tue, May 4, 2010 at 12:02 AM, Dmitriy Ryaboy <dm...@twitter.com>
> wrote:
> >
> > We have an apache license file in the root of the project; I am not sure
> if
> > we need to put it in every file. Will check with the lawyers.
> >
> Generally you put notice at head of each src file. I was particularly
> referring to this that we have in all hbase files:
>
> " * Copyright 2010 The Apache Software Foundation"
>
> We got this practise from hadoop but looking there, they no longer
> seem to do it (I need to talk to lawyers too -- smile).
>
> > Regarding the first and last slice, the problem is that I have no way of
> > knowing what the first and last, respectively, key values are. With the
> > first slice I can maybe cache the first key I see, and use that in
> > conjunction with the end of the region to calculate the size of the
> > keyspace; but with that last region, the max is infinity, so I can't
> really
> > estimate how much more I have left until I have none.. do regions store
> any
> > metadata that countains a rough count of the number of records they hold?
>
> Regions no. StoreFiles yes. They have number of entries but this is
> not really available via API. We should expose it or something like
> it. It could only be an estimate since delete and put both records
> and a delete can remove cell, column or family.
>
>
> I
> > guess they only keep track of the byte size of the data, not the number
> of
> > records per se. Maybe I can get the total byte size of the region, and
> > calculate offsets based on the size of the returned data? This would be
> > likely wrong due to pushed down projections and filters, of course. Any
> > other ideas? How do people normally handle this when writing regular MR
> jobs
> > that scan HBase tables?
> >
>
> I think most tasks that go against hbase should 0% progress and then
> 100% when done.
>
> We could expose a getLastRowInRegion or what if we added an estimated
> row count to the Split (Maybe thats not the right place to expose this
> info? What is the canonical way?).
>
> > I suspect this is actually a bit of a problem, btw -- since I don't
> report
> > the amount of remaining work for these slices accurately, and I
> (hopefully)
> > do a reasonable job for the ones where I can calculate the size of the
> > keyspace, speculative execution may get overeager with these two slices.
> >
> Good point.
>
> We should fix this. Keeping a counter of how many rows in a region
> wouldn't be hard. It could be updated on compaction, etc. A row
> count would be good enough.
>
>
> > PigCounterHelper just deals with some oddities of Hadoop counters (they
> may
> > not be available when you first try to increment a counter -- the helper
> > buffers increment requests until the reporter becomes available). Are
> HBase
> > counters special things or also just Hadoop counters under the covers?
> >
> Check them out. They are not hadoop counters. Keep up a count on
> anything. Might be of use given what you are doing. Update it
> thousands of times a second, etc.
>
>
> > The lzo files are probably unrelated.. there shouldn't be anything
> > LZO-specific in the HBase code. We are, in fact, lzo'ing hbase content in
> > the sense that that's the compression we have for HDFS, and I think HBase
> is
> > supposed to inherit that.
>
> No. You need to enable it on the column family. See how COMPRESSION
> can be NONE, GZ, or LZO. LZO needs to be installed as per hadoop.
> Search the hbase wiki home page for lzo. The hbase+lzo page has had
> some experience baked in to it so may be of some use to you.
>
> St.Ack
>
Re: HBase / Pig integration
Posted by Stack <st...@duboce.net>.
On Tue, May 4, 2010 at 7:34 AM, Stack <st...@duboce.net> wrote:
> On Tue, May 4, 2010 at 12:02 AM, Dmitriy Ryaboy <dm...@twitter.com> wrote:
>
> We should fix this. Keeping a counter of how many rows in a region
> wouldn't be hard. It could be updated on compaction, etc. A row
> count would be good enough.
>
<<Post-Coffee>>
On second thoughts, keeping a counter wouldn't be that easy
particularly if multiple column families.
I wonder what happens if you do a getClosestRowBefore on the last
region passing in the "" key. Will it return the last row in the
region? (I'll try it later if you don't get to it first).
St.Ack
Re: HBase / Pig integration
Posted by Stack <st...@duboce.net>.
On Tue, May 4, 2010 at 12:02 AM, Dmitriy Ryaboy <dm...@twitter.com> wrote:
>
> We have an apache license file in the root of the project; I am not sure if
> we need to put it in every file. Will check with the lawyers.
>
Generally you put notice at head of each src file. I was particularly
referring to this that we have in all hbase files:
" * Copyright 2010 The Apache Software Foundation"
We got this practise from hadoop but looking there, they no longer
seem to do it (I need to talk to lawyers too -- smile).
> Regarding the first and last slice, the problem is that I have no way of
> knowing what the first and last, respectively, key values are. With the
> first slice I can maybe cache the first key I see, and use that in
> conjunction with the end of the region to calculate the size of the
> keyspace; but with that last region, the max is infinity, so I can't really
> estimate how much more I have left until I have none.. do regions store any
> metadata that countains a rough count of the number of records they hold?
Regions no. StoreFiles yes. They have number of entries but this is
not really available via API. We should expose it or something like
it. It could only be an estimate since delete and put both records
and a delete can remove cell, column or family.
I
> guess they only keep track of the byte size of the data, not the number of
> records per se. Maybe I can get the total byte size of the region, and
> calculate offsets based on the size of the returned data? This would be
> likely wrong due to pushed down projections and filters, of course. Any
> other ideas? How do people normally handle this when writing regular MR jobs
> that scan HBase tables?
>
I think most tasks that go against hbase should 0% progress and then
100% when done.
We could expose a getLastRowInRegion or what if we added an estimated
row count to the Split (Maybe thats not the right place to expose this
info? What is the canonical way?).
> I suspect this is actually a bit of a problem, btw -- since I don't report
> the amount of remaining work for these slices accurately, and I (hopefully)
> do a reasonable job for the ones where I can calculate the size of the
> keyspace, speculative execution may get overeager with these two slices.
>
Good point.
We should fix this. Keeping a counter of how many rows in a region
wouldn't be hard. It could be updated on compaction, etc. A row
count would be good enough.
> PigCounterHelper just deals with some oddities of Hadoop counters (they may
> not be available when you first try to increment a counter -- the helper
> buffers increment requests until the reporter becomes available). Are HBase
> counters special things or also just Hadoop counters under the covers?
>
Check them out. They are not hadoop counters. Keep up a count on
anything. Might be of use given what you are doing. Update it
thousands of times a second, etc.
> The lzo files are probably unrelated.. there shouldn't be anything
> LZO-specific in the HBase code. We are, in fact, lzo'ing hbase content in
> the sense that that's the compression we have for HDFS, and I think HBase is
> supposed to inherit that.
No. You need to enable it on the column family. See how COMPRESSION
can be NONE, GZ, or LZO. LZO needs to be installed as per hadoop.
Search the hbase wiki home page for lzo. The hbase+lzo page has had
some experience baked in to it so may be of some use to you.
St.Ack
Re: HBase / Pig integration
Posted by Dmitriy Ryaboy <dm...@twitter.com>.
Hi Stack,
Thanks for taking a look!
We have an apache license file in the root of the project; I am not sure if
we need to put it in every file. Will check with the lawyers.
Regarding the first and last slice, the problem is that I have no way of
knowing what the first and last, respectively, key values are. With the
first slice I can maybe cache the first key I see, and use that in
conjunction with the end of the region to calculate the size of the
keyspace; but with that last region, the max is infinity, so I can't really
estimate how much more I have left until I have none.. do regions store any
metadata that countains a rough count of the number of records they hold? I
guess they only keep track of the byte size of the data, not the number of
records per se. Maybe I can get the total byte size of the region, and
calculate offsets based on the size of the returned data? This would be
likely wrong due to pushed down projections and filters, of course. Any
other ideas? How do people normally handle this when writing regular MR jobs
that scan HBase tables?
I suspect this is actually a bit of a problem, btw -- since I don't report
the amount of remaining work for these slices accurately, and I (hopefully)
do a reasonable job for the ones where I can calculate the size of the
keyspace, speculative execution may get overeager with these two slices.
Will take care of the HBaseConfiguration thing.
PigCounterHelper just deals with some oddities of Hadoop counters (they may
not be available when you first try to increment a counter -- the helper
buffers increment requests until the reporter becomes available). Are HBase
counters special things or also just Hadoop counters under the covers?
The lzo files are probably unrelated.. there shouldn't be anything
LZO-specific in the HBase code. We are, in fact, lzo'ing hbase content in
the sense that that's the compression we have for HDFS, and I think HBase is
supposed to inherit that. But that's at the HDFS level. The ElephantBird
library is a grab-bag of all kinds of hadoop stuff, so you can find lots of
things unrelated to HBase in there, mostly dealing with protocol buffers and
LZO compression.
The only 2 HBase files are HBaseLoader and HBaseSlice.
Thanks again for checking it out!
-Dmitriy
On Mon, May 3, 2010 at 8:48 PM, Stack <st...@duboce.net> wrote:
> Hey Dmitry:
>
> I took a quick look.
>
> Your files are missing a copyright?
>
> I like your using of BinaryComparatory and the lte, gte, options in
> skipRegion setting up filters.
>
> Regards:
>
> " // No way to know max.. just return 0. Sorry, reporting on the
> last slice is janky.
> // So is reporting on the first slice, by the way -- it will start
> out too high, possibly at 100%.
> if (endRow_.length==0) return 0;
> "
>
>
> ...if your keys are kinda regular, you might be able to do better in a
> slice. See in Bytes where there are methods that do BigDecimal math.
> You can ask them to divide the slice. Might work. Then you could do
> progress (Looks like you are doing some later in the file -- does it
> work?).
>
> Try to use the same version of " HBaseConfiguration conf = new
> HBaseConfiguration();" throughout rather than create a new one each
> time. Can be more costly.
>
> Whats this?
>
> if (counterHelper_ == null) counterHelper_ = new PigCounterHelper();
>
> A pig counter? You don't want to use hbase counters?
>
> Whats the lzo stuff about? It seems to be for loading files. Are you
> lzo'ing your hbase content?
>
> Oh man ... base64'ing....
>
> There are two files w/ mention of hbase, is that right?
>
> St.Ack
>
>
>
> On Mon, May 3, 2010 at 12:23 PM, Dmitriy Ryaboy <dm...@twitter.com>
> wrote:
> > Hi folks,
> > I recently rewrote the Pig HBase loader to work with binary data, push
> down
> > filters, and do other things that make it more versatile.
> > If you use, or plan to use, both Pig and HBase, please try it out, take a
> > look at the code, let me know what you think. I am just starting to
> learn
> > about HBase, so I am especially interested to learn if there are HBase
> > capabilities I am not using and should be.
> >
> > The code is part of our "ElephantBird" project, here:
> >
> > http://github.com/kevinweil/elephant-bird/
> > and more specifically:
> >
> http://github.com/kevinweil/elephant-bird/tree/master/src/java/com/twitter/elephantbird/pig/load/
> >
> > Thanks,
> > -Dmitriy
> >
>
Re: HBase / Pig integration
Posted by Stack <st...@duboce.net>.
Hey Dmitry:
I took a quick look.
Your files are missing a copyright?
I like your using of BinaryComparatory and the lte, gte, options in
skipRegion setting up filters.
Regards:
" // No way to know max.. just return 0. Sorry, reporting on the
last slice is janky.
// So is reporting on the first slice, by the way -- it will start
out too high, possibly at 100%.
if (endRow_.length==0) return 0;
"
...if your keys are kinda regular, you might be able to do better in a
slice. See in Bytes where there are methods that do BigDecimal math.
You can ask them to divide the slice. Might work. Then you could do
progress (Looks like you are doing some later in the file -- does it
work?).
Try to use the same version of " HBaseConfiguration conf = new
HBaseConfiguration();" throughout rather than create a new one each
time. Can be more costly.
Whats this?
if (counterHelper_ == null) counterHelper_ = new PigCounterHelper();
A pig counter? You don't want to use hbase counters?
Whats the lzo stuff about? It seems to be for loading files. Are you
lzo'ing your hbase content?
Oh man ... base64'ing....
There are two files w/ mention of hbase, is that right?
St.Ack
On Mon, May 3, 2010 at 12:23 PM, Dmitriy Ryaboy <dm...@twitter.com> wrote:
> Hi folks,
> I recently rewrote the Pig HBase loader to work with binary data, push down
> filters, and do other things that make it more versatile.
> If you use, or plan to use, both Pig and HBase, please try it out, take a
> look at the code, let me know what you think. I am just starting to learn
> about HBase, so I am especially interested to learn if there are HBase
> capabilities I am not using and should be.
>
> The code is part of our "ElephantBird" project, here:
>
> http://github.com/kevinweil/elephant-bird/
> and more specifically:
> http://github.com/kevinweil/elephant-bird/tree/master/src/java/com/twitter/elephantbird/pig/load/
>
> Thanks,
> -Dmitriy
>