You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Daniel Leffel <da...@gmail.com> on 2008/04/29 04:04:48 UTC
Problem I'm having with Release Candidate 0.1.2
Ever since upgrading to the first 0.1.2 release candidate, I cannot alter or
drop a table. The shell reports the operation succeeded, but the master node
starts failing to respond to any requests (although it's still running).
Stopping and starting hbase causes the table to be unchanged. Below is a
DEBUG output from an attempt to drop a table (you'll note no interesting
messages).
2008-04-28 20:02:20,133 INFO org.apache.hadoop.hbase.HMaster: deleted table:
sample_table
2008-04-28 20:02:23,937 INFO org.apache.hadoop.hbase.HMaster:
127.0.0.1:60020 no longer serving regionname: sample_table,,1209431861956,
startKey: <>, endKey: <>, encodedName: 665384872, tableDesc: {name:
sample_table, families: {categories:={name: categories, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, condition:={name: condition, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, fax:={name: fax,
max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, image_10:={name: image_10, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, image_1:={name: image_1, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, image_2:={name:
image_2, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, image_3:={name: image_3, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, image_4:={name: image_4, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, image_5:={name:
image_5, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, image_6:={name: image_6, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, image_7:={name: image_7, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, image_8:={name:
image_8, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, image_9:={name: image_9, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, location:={name: location, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none},
manufacturer:={name: manufacturer, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, model:={name:
model, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, phone:={name: phone, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, price:={name: price, max versions: 3, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}, seller:={name: seller,
max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, serial_number:={name: serial_number, max
versions: 3, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, stock_number:={name: stock_number, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, title:={name: title, max versions: 3, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}}}
2008-04-28 20:02:23,939 INFO org.apache.hadoop.hbase.HMaster: region closed:
sample_table,,1209431861956
2008-04-28 20:02:38,674 INFO org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey:
<>, server: 127.0.0.1:60020}
2008-04-28 20:02:38,688 INFO org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
<>, server: 127.0.0.1:60020} complete
2008-04-28 20:02:44,661 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scanning meta region {regionname: .META.,,1, startKey:
<>, server: 127.0.0.1:60020}
2008-04-28 20:02:44,680 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scan of meta region {regionname: .META.,,1, startKey:
<>, server: 127.0.0.1:60020} complete
2008-04-28 20:02:44,680 INFO org.apache.hadoop.hbase.HMaster: all meta
regions scanned
2008-04-28 20:02:44,969 INFO org.apache.hadoop.hbase.HMaster: assigning
region sample_table,,1209431861956 to the only server 127.0.0.1:60020
2008-04-28 20:02:47,975 INFO org.apache.hadoop.hbase.HMaster:
127.0.0.1:60020 serving sample_table,,1209431861956
2008-04-28 20:02:47,976 INFO org.apache.hadoop.hbase.HMaster: regionname:
sample_table,,1209431861956, startKey: <>, endKey: <>, encodedName:
665384872, tableDesc: {name: sample_table, families: {categories:={name:
categories, max versions: 3, compression: NONE, in memory: false, max
length: 2147483647, bloom filter: none}, condition:={name: condition, max
versions: 3, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, fax:={name: fax, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, image_10:={name:
image_10, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, image_1:={name: image_1, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, image_2:={name: image_2, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, image_3:={name:
image_3, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, image_4:={name: image_4, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, image_5:={name: image_5, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, image_6:={name:
image_6, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, image_7:={name: image_7, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, image_8:={name: image_8, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, image_9:={name:
image_9, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, location:={name: location, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}, manufacturer:={name: manufacturer, max versions: 3, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none},
model:={name: model, max versions: 3, compression: NONE, in memory: false,
max length: 2147483647, bloom filter: none}, phone:={name: phone, max
versions: 3, compression: NONE, in memory: false, max length: 2147483647,
bloom filter: none}, price:={name: price, max versions: 3, compression:
NONE, in memory: false, max length: 2147483647, bloom filter: none},
seller:={name: seller, max versions: 3, compression: NONE, in memory: false,
max length: 2147483647, bloom filter: none}, serial_number:={name:
serial_number, max versions: 3, compression: NONE, in memory: false, max
length: 2147483647, bloom filter: none}, stock_number:={name: stock_number,
max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, title:={name: title, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}}} open on 127.0.0.1:60020
2008-04-28 20:02:47,976 INFO org.apache.hadoop.hbase.HMaster: updating row
sample_table,,1209431861956 in table .META.,,1 with startcode 1209437914462
and server 127.0.0.1:60020
2008-04-28 20:03:38,676 INFO org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey:
<>, server: 127.0.0.1:60020}
2008-04-28 20:03:38,692 INFO org.apache.hadoop.hbase.HMaster:
HMaster.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
<>, server: 127.0.0.1:60020} complete
2008-04-28 20:03:44,662 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scanning meta region {regionname: .META.,,1, startKey:
<>, server: 127.0.0.1:60020}
2008-04-28 20:03:44,683 INFO org.apache.hadoop.hbase.HMaster:
HMaster.metaScanner scan of meta region {regionname: .META.,,1, startKey:
<>, server: 127.0.0.1:60020} complete
2008-04-28 20:03:44,683 INFO org.apache.hadoop.hbase.HMaster: all meta
regions scanned
Re: Problem I'm having with Release Candidate 0.1.2
Posted by Daniel Leffel <da...@gmail.com>.
I don't know about idle java consuming a load of 0.2... Maybe all that load
goes into figuring out a way to store an int in a few hundred bytes. ;-)
Clearly, you were right about the hadoop-site.xml fix. I don't know where I
read that this was something to do with TableReduce. I kept looking through
the source code and couldn't find where it was being set. This makes a lot
more sense now.
Yep. Dev is all one machine and production (being built as we speak) will be
a more traditional deployment.
Trying the job again now and will let you know what kind of speed
improvement I get.
Thanks,
Danny
On Tue, Apr 29, 2008 at 2:03 PM, stack <st...@duboce.net> wrote:
> Daniel Leffel wrote:
>
>> The Reduce phase is a simple reduce using the TableReduceOutputFormat.
>>
>>
> Try upping the number of reducers -- mapred.reduce.tasks in your
> hadoop-site.xml. Default is 1. Try with 4/8/16?
>
> Is your hbase cluster all running on the same node too?
>
> In this configuration, it writes 50-100 or so inserts per second (which
>> doesn't strike me as terrible), but given the low load factor of 0.20 on
>> the
>> machine, I don't understand what's keeping it from performing better.
>>
>>
>
> Yeah... doesn't java just idling consume a load of 0.2 (smile). Is this a
> multicore box?
>
>
> St.Ack
>
Re: Problem I'm having with Release Candidate 0.1.2
Posted by stack <st...@duboce.net>.
Daniel Leffel wrote:
> The Reduce phase is a simple reduce using the TableReduceOutputFormat.
>
Try upping the number of reducers -- mapred.reduce.tasks in your
hadoop-site.xml. Default is 1. Try with 4/8/16?
Is your hbase cluster all running on the same node too?
> In this configuration, it writes 50-100 or so inserts per second (which
> doesn't strike me as terrible), but given the low load factor of 0.20 on the
> machine, I don't understand what's keeping it from performing better.
>
Yeah... doesn't java just idling consume a load of 0.2 (smile). Is this
a multicore box?
St.Ack
Re: Problem I'm having with Release Candidate 0.1.2
Posted by Daniel Leffel <da...@gmail.com>.
Running a MapReduce job on a single node (as a test) in pseudo-distributed
node.
The Map job is a little intense and generates 221 million output records.
The Reduce phase is a simple reduce using the TableReduceOutputFormat.
It's single threaded during reduce because, as I understand it, TableReduce
uses only one reduce per node (or is there some other reason that reduce
only spawns one reduce task?).
In this configuration, it writes 50-100 or so inserts per second (which
doesn't strike me as terrible), but given the low load factor of 0.20 on the
machine, I don't understand what's keeping it from performing better.
On Tue, Apr 29, 2008 at 12:47 PM, stack <st...@duboce.net> wrote:
> Daniel Leffel wrote:
>
>> However, it baffles me a little that it's so slow. I don't quite
>> understand
>> the bottleneck - load on the machine is 0.20 for a pure load.
>> Nevertheless,
>> large batch loads that run a little slow is WAY better to me than crashing
>> with OOME.
>>
> Tell us more about how you are running your upload Daniel? Is it single
> client? Single threaded? What size are records? What size cluster (pardon
> if you've already said)?
> Thanks,
> St.Ack
>
Re: Problem I'm having with Release Candidate 0.1.2
Posted by stack <st...@duboce.net>.
Daniel Leffel wrote:
> However, it baffles me a little that it's so slow. I don't quite understand
> the bottleneck - load on the machine is 0.20 for a pure load. Nevertheless,
> large batch loads that run a little slow is WAY better to me than crashing
> with OOME.
Tell us more about how you are running your upload Daniel? Is it single
client? Single threaded? What size are records? What size cluster
(pardon if you've already said)?
Thanks,
St.Ack
Re: Problem I'm having with Release Candidate 0.1.2
Posted by Daniel Leffel <da...@gmail.com>.
Yes. Large uploads are working now in this release! Thanks!
However, it baffles me a little that it's so slow. I don't quite understand
the bottleneck - load on the machine is 0.20 for a pure load. Nevertheless,
large batch loads that run a little slow is WAY better to me than crashing
with OOME.
On Tue, Apr 29, 2008 at 11:25 AM, stack <st...@duboce.net> wrote:
> I just tried it w/ a decent sized table and ran into the same issue. For a
> while we've known offlining to be unreliable, HBASE-478, and intended fixing
> it for 0.2.0 but with this demonstration of how bad our offlining is
> currently working, I think it needs to be fixed in the branch and included
> in 0.1.2 release. I'll put up a new candidate in a few days.
>
> Did your upload work?
> St.Ack
>
>
>
> Daniel Leffel wrote:
>
>> New piece of info. Now a simple select count(*) from sample_table; results
>> in the following error:
>>
>> HRegionInfo was null or empty in .META.
>>
>>
>> On Mon, Apr 28, 2008 at 7:04 PM, Daniel Leffel <da...@gmail.com>
>> wrote:
>>
>>
>>
>>> Ever since upgrading to the first 0.1.2 release candidate, I cannot alter
>>> or drop a table. The shell reports the operation succeeded, but the
>>> master
>>> node starts failing to respond to any requests (although it's still
>>> running). Stopping and starting hbase causes the table to be unchanged.
>>> Below is a DEBUG output from an attempt to drop a table (you'll note no
>>> interesting messages).
>>>
>>> 2008-04-28 20:02:20,133 INFO org.apache.hadoop.hbase.HMaster: deleted
>>> table: sample_table
>>> 2008-04-28 20:02:23,937 INFO org.apache.hadoop.hbase.HMaster:
>>> 127.0.0.1:60020 no longer serving regionname:
>>> sample_table,,1209431861956,
>>> startKey: <>, endKey: <>, encodedName: 665384872, tableDesc: {name:
>>> sample_table, families: {categories:={name: categories, max versions: 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, condition:={name: condition, max versions: 3, compression: NONE,
>>> in
>>> memory: false, max length: 2147483647, bloom filter: none}, fax:={name:
>>> fax,
>>> max versions: 3, compression: NONE, in memory: false, max length:
>>> 2147483647, bloom filter: none}, image_10:={name: image_10, max versions:
>>> 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, image_1:={name: image_1, max versions: 3, compression: NONE, in
>>> memory: false, max length: 2147483647, bloom filter: none},
>>> image_2:={name:
>>> image_2, max versions: 3, compression: NONE, in memory: false, max
>>> length:
>>> 2147483647, bloom filter: none}, image_3:={name: image_3, max versions:
>>> 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, image_4:={name: image_4, max versions: 3, compression: NONE, in
>>> memory: false, max length: 2147483647, bloom filter: none},
>>> image_5:={name:
>>> image_5, max versions: 3, compression: NONE, in memory: false, max
>>> length:
>>> 2147483647, bloom filter: none}, image_6:={name: image_6, max versions:
>>> 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, image_7:={name: image_7, max versions: 3, compression: NONE, in
>>> memory: false, max length: 2147483647, bloom filter: none},
>>> image_8:={name:
>>> image_8, max versions: 3, compression: NONE, in memory: false, max
>>> length:
>>> 2147483647, bloom filter: none}, image_9:={name: image_9, max versions:
>>> 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, location:={name: location, max versions: 3, compression: NONE, in
>>> memory: false, max length: 2147483647, bloom filter: none},
>>> manufacturer:={name: manufacturer, max versions: 3, compression: NONE, in
>>> memory: false, max length: 2147483647, bloom filter: none}, model:={name:
>>> model, max versions: 3, compression: NONE, in memory: false, max length:
>>> 2147483647, bloom filter: none}, phone:={name: phone, max versions: 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, price:={name: price, max versions: 3, compression: NONE, in
>>> memory:
>>> false, max length: 2147483647, bloom filter: none}, seller:={name:
>>> seller,
>>> max versions: 3, compression: NONE, in memory: false, max length:
>>> 2147483647, bloom filter: none}, serial_number:={name: serial_number, max
>>> versions: 3, compression: NONE, in memory: false, max length: 2147483647,
>>> bloom filter: none}, stock_number:={name: stock_number, max versions: 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, title:={name: title, max versions: 3, compression: NONE, in
>>> memory:
>>> false, max length: 2147483647, bloom filter: none}}}
>>> 2008-04-28 20:02:23,939 INFO org.apache.hadoop.hbase.HMaster: region
>>> closed: sample_table,,1209431861956
>>> 2008-04-28 20:02:38,674 INFO org.apache.hadoop.hbase.HMaster:
>>> HMaster.rootScanner scanning meta region {regionname: -ROOT-,,0,
>>> startKey:
>>> <>, server: 127.0.0.1:60020}
>>> 2008-04-28 20:02:38,688 INFO org.apache.hadoop.hbase.HMaster:
>>> HMaster.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
>>> <>, server: 127.0.0.1:60020} complete
>>> 2008-04-28 20:02:44,661 INFO org.apache.hadoop.hbase.HMaster:
>>> HMaster.metaScanner scanning meta region {regionname: .META.,,1,
>>> startKey:
>>> <>, server: 127.0.0.1:60020}
>>> 2008-04-28 20:02:44,680 INFO org.apache.hadoop.hbase.HMaster:
>>> HMaster.metaScanner scan of meta region {regionname: .META.,,1, startKey:
>>> <>, server: 127.0.0.1:60020} complete
>>> 2008-04-28 20:02:44,680 INFO org.apache.hadoop.hbase.HMaster: all meta
>>> regions scanned
>>> 2008-04-28 20:02:44,969 INFO org.apache.hadoop.hbase.HMaster: assigning
>>> region sample_table,,1209431861956 to the only server 127.0.0.1:60020
>>> 2008-04-28 20:02:47,975 INFO org.apache.hadoop.hbase.HMaster:
>>> 127.0.0.1:60020 serving sample_table,,1209431861956
>>> 2008-04-28 20:02:47,976 INFO org.apache.hadoop.hbase.HMaster: regionname:
>>> sample_table,,1209431861956, startKey: <>, endKey: <>, encodedName:
>>> 665384872, tableDesc: {name: sample_table, families: {categories:={name:
>>> categories, max versions: 3, compression: NONE, in memory: false, max
>>> length: 2147483647, bloom filter: none}, condition:={name: condition, max
>>> versions: 3, compression: NONE, in memory: false, max length: 2147483647,
>>> bloom filter: none}, fax:={name: fax, max versions: 3, compression: NONE,
>>> in
>>> memory: false, max length: 2147483647, bloom filter: none},
>>> image_10:={name:
>>> image_10, max versions: 3, compression: NONE, in memory: false, max
>>> length:
>>> 2147483647, bloom filter: none}, image_1:={name: image_1, max versions:
>>> 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, image_2:={name: image_2, max versions: 3, compression: NONE, in
>>> memory: false, max length: 2147483647, bloom filter: none},
>>> image_3:={name:
>>> image_3, max versions: 3, compression: NONE, in memory: false, max
>>> length:
>>> 2147483647, bloom filter: none}, image_4:={name: image_4, max versions:
>>> 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, image_5:={name: image_5, max versions: 3, compression: NONE, in
>>> memory: false, max length: 2147483647, bloom filter: none},
>>> image_6:={name:
>>> image_6, max versions: 3, compression: NONE, in memory: false, max
>>> length:
>>> 2147483647, bloom filter: none}, image_7:={name: image_7, max versions:
>>> 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, image_8:={name: image_8, max versions: 3, compression: NONE, in
>>> memory: false, max length: 2147483647, bloom filter: none},
>>> image_9:={name:
>>> image_9, max versions: 3, compression: NONE, in memory: false, max
>>> length:
>>> 2147483647, bloom filter: none}, location:={name: location, max versions:
>>> 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}, manufacturer:={name: manufacturer, max versions: 3, compression:
>>> NONE, in memory: false, max length: 2147483647, bloom filter: none},
>>> model:={name: model, max versions: 3, compression: NONE, in memory:
>>> false,
>>> max length: 2147483647, bloom filter: none}, phone:={name: phone, max
>>> versions: 3, compression: NONE, in memory: false, max length: 2147483647,
>>> bloom filter: none}, price:={name: price, max versions: 3, compression:
>>> NONE, in memory: false, max length: 2147483647, bloom filter: none},
>>> seller:={name: seller, max versions: 3, compression: NONE, in memory:
>>> false,
>>> max length: 2147483647, bloom filter: none}, serial_number:={name:
>>> serial_number, max versions: 3, compression: NONE, in memory: false, max
>>> length: 2147483647, bloom filter: none}, stock_number:={name:
>>> stock_number,
>>> max versions: 3, compression: NONE, in memory: false, max length:
>>> 2147483647, bloom filter: none}, title:={name: title, max versions: 3,
>>> compression: NONE, in memory: false, max length: 2147483647, bloom
>>> filter:
>>> none}}} open on 127.0.0.1:60020
>>> 2008-04-28 20:02:47,976 INFO org.apache.hadoop.hbase.HMaster: updating
>>> row
>>> sample_table,,1209431861956 in table .META.,,1 with startcode
>>> 1209437914462
>>> and server 127.0.0.1:60020
>>> 2008-04-28 20:03:38,676 INFO org.apache.hadoop.hbase.HMaster:
>>> HMaster.rootScanner scanning meta region {regionname: -ROOT-,,0,
>>> startKey:
>>> <>, server: 127.0.0.1:60020}
>>> 2008-04-28 20:03:38,692 INFO org.apache.hadoop.hbase.HMaster:
>>> HMaster.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
>>> <>, server: 127.0.0.1:60020} complete
>>> 2008-04-28 20:03:44,662 INFO org.apache.hadoop.hbase.HMaster:
>>> HMaster.metaScanner scanning meta region {regionname: .META.,,1,
>>> startKey:
>>> <>, server: 127.0.0.1:60020}
>>> 2008-04-28 20:03:44,683 INFO org.apache.hadoop.hbase.HMaster:
>>> HMaster.metaScanner scan of meta region {regionname: .META.,,1, startKey:
>>> <>, server: 127.0.0.1:60020} complete
>>> 2008-04-28 20:03:44,683 INFO org.apache.hadoop.hbase.HMaster: all meta
>>> regions scanned
>>>
>>>
>>>
>>
>>
>>
>
>
Re: Problem I'm having with Release Candidate 0.1.2
Posted by stack <st...@duboce.net>.
I just tried it w/ a decent sized table and ran into the same issue.
For a while we've known offlining to be unreliable, HBASE-478, and
intended fixing it for 0.2.0 but with this demonstration of how bad our
offlining is currently working, I think it needs to be fixed in the
branch and included in 0.1.2 release. I'll put up a new candidate in a
few days.
Did your upload work?
St.Ack
Daniel Leffel wrote:
> New piece of info. Now a simple select count(*) from sample_table; results
> in the following error:
>
> HRegionInfo was null or empty in .META.
>
>
> On Mon, Apr 28, 2008 at 7:04 PM, Daniel Leffel <da...@gmail.com>
> wrote:
>
>
>> Ever since upgrading to the first 0.1.2 release candidate, I cannot alter
>> or drop a table. The shell reports the operation succeeded, but the master
>> node starts failing to respond to any requests (although it's still
>> running). Stopping and starting hbase causes the table to be unchanged.
>> Below is a DEBUG output from an attempt to drop a table (you'll note no
>> interesting messages).
>>
>> 2008-04-28 20:02:20,133 INFO org.apache.hadoop.hbase.HMaster: deleted
>> table: sample_table
>> 2008-04-28 20:02:23,937 INFO org.apache.hadoop.hbase.HMaster:
>> 127.0.0.1:60020 no longer serving regionname: sample_table,,1209431861956,
>> startKey: <>, endKey: <>, encodedName: 665384872, tableDesc: {name:
>> sample_table, families: {categories:={name: categories, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, condition:={name: condition, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none}, fax:={name: fax,
>> max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, image_10:={name: image_10, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, image_1:={name: image_1, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none}, image_2:={name:
>> image_2, max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, image_3:={name: image_3, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, image_4:={name: image_4, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none}, image_5:={name:
>> image_5, max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, image_6:={name: image_6, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, image_7:={name: image_7, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none}, image_8:={name:
>> image_8, max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, image_9:={name: image_9, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, location:={name: location, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none},
>> manufacturer:={name: manufacturer, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none}, model:={name:
>> model, max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, phone:={name: phone, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, price:={name: price, max versions: 3, compression: NONE, in memory:
>> false, max length: 2147483647, bloom filter: none}, seller:={name: seller,
>> max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, serial_number:={name: serial_number, max
>> versions: 3, compression: NONE, in memory: false, max length: 2147483647,
>> bloom filter: none}, stock_number:={name: stock_number, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, title:={name: title, max versions: 3, compression: NONE, in memory:
>> false, max length: 2147483647, bloom filter: none}}}
>> 2008-04-28 20:02:23,939 INFO org.apache.hadoop.hbase.HMaster: region
>> closed: sample_table,,1209431861956
>> 2008-04-28 20:02:38,674 INFO org.apache.hadoop.hbase.HMaster:
>> HMaster.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey:
>> <>, server: 127.0.0.1:60020}
>> 2008-04-28 20:02:38,688 INFO org.apache.hadoop.hbase.HMaster:
>> HMaster.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
>> <>, server: 127.0.0.1:60020} complete
>> 2008-04-28 20:02:44,661 INFO org.apache.hadoop.hbase.HMaster:
>> HMaster.metaScanner scanning meta region {regionname: .META.,,1, startKey:
>> <>, server: 127.0.0.1:60020}
>> 2008-04-28 20:02:44,680 INFO org.apache.hadoop.hbase.HMaster:
>> HMaster.metaScanner scan of meta region {regionname: .META.,,1, startKey:
>> <>, server: 127.0.0.1:60020} complete
>> 2008-04-28 20:02:44,680 INFO org.apache.hadoop.hbase.HMaster: all meta
>> regions scanned
>> 2008-04-28 20:02:44,969 INFO org.apache.hadoop.hbase.HMaster: assigning
>> region sample_table,,1209431861956 to the only server 127.0.0.1:60020
>> 2008-04-28 20:02:47,975 INFO org.apache.hadoop.hbase.HMaster:
>> 127.0.0.1:60020 serving sample_table,,1209431861956
>> 2008-04-28 20:02:47,976 INFO org.apache.hadoop.hbase.HMaster: regionname:
>> sample_table,,1209431861956, startKey: <>, endKey: <>, encodedName:
>> 665384872, tableDesc: {name: sample_table, families: {categories:={name:
>> categories, max versions: 3, compression: NONE, in memory: false, max
>> length: 2147483647, bloom filter: none}, condition:={name: condition, max
>> versions: 3, compression: NONE, in memory: false, max length: 2147483647,
>> bloom filter: none}, fax:={name: fax, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none}, image_10:={name:
>> image_10, max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, image_1:={name: image_1, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, image_2:={name: image_2, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none}, image_3:={name:
>> image_3, max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, image_4:={name: image_4, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, image_5:={name: image_5, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none}, image_6:={name:
>> image_6, max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, image_7:={name: image_7, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, image_8:={name: image_8, max versions: 3, compression: NONE, in
>> memory: false, max length: 2147483647, bloom filter: none}, image_9:={name:
>> image_9, max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, location:={name: location, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}, manufacturer:={name: manufacturer, max versions: 3, compression:
>> NONE, in memory: false, max length: 2147483647, bloom filter: none},
>> model:={name: model, max versions: 3, compression: NONE, in memory: false,
>> max length: 2147483647, bloom filter: none}, phone:={name: phone, max
>> versions: 3, compression: NONE, in memory: false, max length: 2147483647,
>> bloom filter: none}, price:={name: price, max versions: 3, compression:
>> NONE, in memory: false, max length: 2147483647, bloom filter: none},
>> seller:={name: seller, max versions: 3, compression: NONE, in memory: false,
>> max length: 2147483647, bloom filter: none}, serial_number:={name:
>> serial_number, max versions: 3, compression: NONE, in memory: false, max
>> length: 2147483647, bloom filter: none}, stock_number:={name: stock_number,
>> max versions: 3, compression: NONE, in memory: false, max length:
>> 2147483647, bloom filter: none}, title:={name: title, max versions: 3,
>> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
>> none}}} open on 127.0.0.1:60020
>> 2008-04-28 20:02:47,976 INFO org.apache.hadoop.hbase.HMaster: updating row
>> sample_table,,1209431861956 in table .META.,,1 with startcode 1209437914462
>> and server 127.0.0.1:60020
>> 2008-04-28 20:03:38,676 INFO org.apache.hadoop.hbase.HMaster:
>> HMaster.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey:
>> <>, server: 127.0.0.1:60020}
>> 2008-04-28 20:03:38,692 INFO org.apache.hadoop.hbase.HMaster:
>> HMaster.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
>> <>, server: 127.0.0.1:60020} complete
>> 2008-04-28 20:03:44,662 INFO org.apache.hadoop.hbase.HMaster:
>> HMaster.metaScanner scanning meta region {regionname: .META.,,1, startKey:
>> <>, server: 127.0.0.1:60020}
>> 2008-04-28 20:03:44,683 INFO org.apache.hadoop.hbase.HMaster:
>> HMaster.metaScanner scan of meta region {regionname: .META.,,1, startKey:
>> <>, server: 127.0.0.1:60020} complete
>> 2008-04-28 20:03:44,683 INFO org.apache.hadoop.hbase.HMaster: all meta
>> regions scanned
>>
>>
>
>
Re: Problem I'm having with Release Candidate 0.1.2
Posted by Daniel Leffel <da...@gmail.com>.
New piece of info. Now a simple select count(*) from sample_table; results
in the following error:
HRegionInfo was null or empty in .META.
On Mon, Apr 28, 2008 at 7:04 PM, Daniel Leffel <da...@gmail.com>
wrote:
> Ever since upgrading to the first 0.1.2 release candidate, I cannot alter
> or drop a table. The shell reports the operation succeeded, but the master
> node starts failing to respond to any requests (although it's still
> running). Stopping and starting hbase causes the table to be unchanged.
> Below is a DEBUG output from an attempt to drop a table (you'll note no
> interesting messages).
>
> 2008-04-28 20:02:20,133 INFO org.apache.hadoop.hbase.HMaster: deleted
> table: sample_table
> 2008-04-28 20:02:23,937 INFO org.apache.hadoop.hbase.HMaster:
> 127.0.0.1:60020 no longer serving regionname: sample_table,,1209431861956,
> startKey: <>, endKey: <>, encodedName: 665384872, tableDesc: {name:
> sample_table, families: {categories:={name: categories, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, condition:={name: condition, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none}, fax:={name: fax,
> max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, image_10:={name: image_10, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, image_1:={name: image_1, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none}, image_2:={name:
> image_2, max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, image_3:={name: image_3, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, image_4:={name: image_4, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none}, image_5:={name:
> image_5, max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, image_6:={name: image_6, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, image_7:={name: image_7, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none}, image_8:={name:
> image_8, max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, image_9:={name: image_9, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, location:={name: location, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none},
> manufacturer:={name: manufacturer, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none}, model:={name:
> model, max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, phone:={name: phone, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, price:={name: price, max versions: 3, compression: NONE, in memory:
> false, max length: 2147483647, bloom filter: none}, seller:={name: seller,
> max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, serial_number:={name: serial_number, max
> versions: 3, compression: NONE, in memory: false, max length: 2147483647,
> bloom filter: none}, stock_number:={name: stock_number, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, title:={name: title, max versions: 3, compression: NONE, in memory:
> false, max length: 2147483647, bloom filter: none}}}
> 2008-04-28 20:02:23,939 INFO org.apache.hadoop.hbase.HMaster: region
> closed: sample_table,,1209431861956
> 2008-04-28 20:02:38,674 INFO org.apache.hadoop.hbase.HMaster:
> HMaster.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey:
> <>, server: 127.0.0.1:60020}
> 2008-04-28 20:02:38,688 INFO org.apache.hadoop.hbase.HMaster:
> HMaster.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
> <>, server: 127.0.0.1:60020} complete
> 2008-04-28 20:02:44,661 INFO org.apache.hadoop.hbase.HMaster:
> HMaster.metaScanner scanning meta region {regionname: .META.,,1, startKey:
> <>, server: 127.0.0.1:60020}
> 2008-04-28 20:02:44,680 INFO org.apache.hadoop.hbase.HMaster:
> HMaster.metaScanner scan of meta region {regionname: .META.,,1, startKey:
> <>, server: 127.0.0.1:60020} complete
> 2008-04-28 20:02:44,680 INFO org.apache.hadoop.hbase.HMaster: all meta
> regions scanned
> 2008-04-28 20:02:44,969 INFO org.apache.hadoop.hbase.HMaster: assigning
> region sample_table,,1209431861956 to the only server 127.0.0.1:60020
> 2008-04-28 20:02:47,975 INFO org.apache.hadoop.hbase.HMaster:
> 127.0.0.1:60020 serving sample_table,,1209431861956
> 2008-04-28 20:02:47,976 INFO org.apache.hadoop.hbase.HMaster: regionname:
> sample_table,,1209431861956, startKey: <>, endKey: <>, encodedName:
> 665384872, tableDesc: {name: sample_table, families: {categories:={name:
> categories, max versions: 3, compression: NONE, in memory: false, max
> length: 2147483647, bloom filter: none}, condition:={name: condition, max
> versions: 3, compression: NONE, in memory: false, max length: 2147483647,
> bloom filter: none}, fax:={name: fax, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none}, image_10:={name:
> image_10, max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, image_1:={name: image_1, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, image_2:={name: image_2, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none}, image_3:={name:
> image_3, max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, image_4:={name: image_4, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, image_5:={name: image_5, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none}, image_6:={name:
> image_6, max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, image_7:={name: image_7, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, image_8:={name: image_8, max versions: 3, compression: NONE, in
> memory: false, max length: 2147483647, bloom filter: none}, image_9:={name:
> image_9, max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, location:={name: location, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}, manufacturer:={name: manufacturer, max versions: 3, compression:
> NONE, in memory: false, max length: 2147483647, bloom filter: none},
> model:={name: model, max versions: 3, compression: NONE, in memory: false,
> max length: 2147483647, bloom filter: none}, phone:={name: phone, max
> versions: 3, compression: NONE, in memory: false, max length: 2147483647,
> bloom filter: none}, price:={name: price, max versions: 3, compression:
> NONE, in memory: false, max length: 2147483647, bloom filter: none},
> seller:={name: seller, max versions: 3, compression: NONE, in memory: false,
> max length: 2147483647, bloom filter: none}, serial_number:={name:
> serial_number, max versions: 3, compression: NONE, in memory: false, max
> length: 2147483647, bloom filter: none}, stock_number:={name: stock_number,
> max versions: 3, compression: NONE, in memory: false, max length:
> 2147483647, bloom filter: none}, title:={name: title, max versions: 3,
> compression: NONE, in memory: false, max length: 2147483647, bloom filter:
> none}}} open on 127.0.0.1:60020
> 2008-04-28 20:02:47,976 INFO org.apache.hadoop.hbase.HMaster: updating row
> sample_table,,1209431861956 in table .META.,,1 with startcode 1209437914462
> and server 127.0.0.1:60020
> 2008-04-28 20:03:38,676 INFO org.apache.hadoop.hbase.HMaster:
> HMaster.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey:
> <>, server: 127.0.0.1:60020}
> 2008-04-28 20:03:38,692 INFO org.apache.hadoop.hbase.HMaster:
> HMaster.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey:
> <>, server: 127.0.0.1:60020} complete
> 2008-04-28 20:03:44,662 INFO org.apache.hadoop.hbase.HMaster:
> HMaster.metaScanner scanning meta region {regionname: .META.,,1, startKey:
> <>, server: 127.0.0.1:60020}
> 2008-04-28 20:03:44,683 INFO org.apache.hadoop.hbase.HMaster:
> HMaster.metaScanner scan of meta region {regionname: .META.,,1, startKey:
> <>, server: 127.0.0.1:60020} complete
> 2008-04-28 20:03:44,683 INFO org.apache.hadoop.hbase.HMaster: all meta
> regions scanned
>