You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Albert Shau <as...@yahoo-inc.com> on 2011/07/12 03:56:17 UTC
region max filesize
Hi everyone,
I'm using 0.90.3 and have the region max filesize set to 1gb in my hbase-site.xml like so:
<property>
<name>hbase.hregion.max.filesize</name>
<value>1073741824</value>
</property>
But when I look at the storefileSizeMB values listed on the regionserver UI, they're all around 256mb, which makes me think the default value is still being used for whatever reason. I'm also setting other properties like the handler count which are getting used so I'm pretty sure I have the conf in the right place and it's getting read. I've also tried restarting hbase and doing a major compact on the table but the region sizes stay the same. Any ideas on what might be wrong? Is there some way to verify the max filesize is getting set correctly?
Thanks,
Albert
Should we take a better safeguard in ServerShutdownHandler?
Posted by bijieshan <bi...@huawei.com>.
Hi,
We happened to several odd problems with little probability. Both of those problems come along with some un-wrapped exception in ServerShutdownHandler, like the following two issues:
1)" java.lang.AssertionError: Could not find target position 5368709120":
2011-07-07 13:28:06,118 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_SHUTDOWN
java.lang.AssertionError: Could not find target position 5368709120
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.getBlockAt(DFSClient.java:2157)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2246)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2441)
at java.io.DataInputStream.read(DataInputStream.java:132)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1984)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1884)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1930)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:198)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:172)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.parseHLog(HLogSplitter.java:429)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:262)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:188)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:201)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:114)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2)"IOException":
2011-07-09 01:38:31,836 ERROR org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting hdfs://162.2.16.6:9000/hbase/.logs/162-2-6-187,20020,1310107719056
java.io.IOException: hdfs://162.2.16.6:9000/hbase/.logs/162-2-6-187,20020,1310107719056/162-2-6-187%3A20020.1310143885352, entryStart=1878997244, pos=1879048192, end=2003890606, edit=80274
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.addFileInfoToException(SequenceFileLogReader.java:244)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:200)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:172)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.parseHLog(HLogSplitter.java:429)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:262)
at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:188)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:201)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:114)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:156)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
These exceptions could cause the ServerShutdownHandler process unexpect exit. So if the shutted regionserver carries the -ROOT- or .META. region, they won't be open again unless restart the cluster.
So should we take a better safeguard in it?
Regards,
Jieshan
Re: region max filesize
Posted by Stack <st...@duboce.net>.
Check out the copytables tool. See if that will work for you. Here is
its usage:
stack@sv4borg227:~/hadoop-0.22$ ./bin/hadoop jar
~/hbase-0.91.0-SNAPSHOT/hbase-0.91.0-SNAPSHOT.jar copytable
Usage: CopyTable [--rs.class=CLASS] [--rs.impl=IMPL] [--starttime=X]
[--endtime=Y] [--new.name=NEW] [--peer.adr=ADR] <tablename>
Options:
rs.class hbase.regionserver.class of the peer cluster
specify if different from current cluster
rs.impl hbase.regionserver.impl of the peer cluster
starttime beginning of the time range
without endtime means from starttime to forever
endtime end of the time range
new.name new table's name
peer.adr Address of the peer cluster given in the format
hbase.zookeeer.quorum:hbase.zookeeper.client.port:zookeeper.znode.parent
families comma-seperated list of families to copy
Args:
tablename Name of the table to copy
Examples:
To copy 'TestTable' to a cluster that uses replication for a 1 hour window:
$ bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable
--rs.class=org.apache.hadoop.hbase.ipc.ReplicationRegionInterface
--rs.impl=org.apache.hadoop.hbase.regionserver.replication.ReplicationRegionServer
--starttime=1265875194289 --endtime=1265878794289
--peer.adr=server1,server2,server3:2181:/hbase TestTable
stack@sv4borg227:~/hadoop-0.22$
St.Ack
On Tue, Jul 12, 2011 at 5:17 PM, 陈加俊 <cj...@gmail.com> wrote:
> create simple m/r job to migrate data to new tables and drop the old ones。
>
>
> Could you put your code paste to here?
>
> On Wed, Jul 13, 2011 at 7:43 AM, Ravi Veeramachaneni <
> ravi.veeramachaneni@gmail.com> wrote:
>
>> Albert,
>>
>> You doing partially right. The change will take into effect only to new
>> tables created going forward. There are two options I'm aware of, 1. Use
>> alter command on HBase shell to specify new region size. 2. Create new set
>> of tables and create simple m/r job to migrate data to new tables and drop
>> the old ones.
>>
>> We followed the second approach when we bump the region size. The first
>> approach leaves the current region count But fills up until region gets to
>> 1GB. Our goal was to reduce number of regions to minimum, second approach
>> worked well for us.
>>
>> Thanks,
>> Ravi
>>
>> Sent from my mobile phone.
>>
>> On Jul 11, 2011, at 8:56 PM, Albert Shau <as...@yahoo-inc.com> wrote:
>>
>> > Hi everyone,
>> >
>> > I'm using 0.90.3 and have the region max filesize set to 1gb in my
>> hbase-site.xml like so:
>> >
>> > <property>
>> > <name>hbase.hregion.max.filesize</name>
>> > <value>1073741824</value>
>> > </property>
>> >
>> > But when I look at the storefileSizeMB values listed on the regionserver
>> UI, they're all around 256mb, which makes me think the default value is
>> still being used for whatever reason. I'm also setting other properties
>> like the handler count which are getting used so I'm pretty sure I have the
>> conf in the right place and it's getting read. I've also tried restarting
>> hbase and doing a major compact on the table but the region sizes stay the
>> same. Any ideas on what might be wrong? Is there some way to verify the
>> max filesize is getting set correctly?
>> >
>> > Thanks,
>> > Albert
>> >
>>
>
>
>
> --
> Thanks & Best regards
> jiajun
>
Re: region max filesize
Posted by 陈加俊 <cj...@gmail.com>.
create simple m/r job to migrate data to new tables and drop the old ones。
Could you put your code paste to here?
On Wed, Jul 13, 2011 at 7:43 AM, Ravi Veeramachaneni <
ravi.veeramachaneni@gmail.com> wrote:
> Albert,
>
> You doing partially right. The change will take into effect only to new
> tables created going forward. There are two options I'm aware of, 1. Use
> alter command on HBase shell to specify new region size. 2. Create new set
> of tables and create simple m/r job to migrate data to new tables and drop
> the old ones.
>
> We followed the second approach when we bump the region size. The first
> approach leaves the current region count But fills up until region gets to
> 1GB. Our goal was to reduce number of regions to minimum, second approach
> worked well for us.
>
> Thanks,
> Ravi
>
> Sent from my mobile phone.
>
> On Jul 11, 2011, at 8:56 PM, Albert Shau <as...@yahoo-inc.com> wrote:
>
> > Hi everyone,
> >
> > I'm using 0.90.3 and have the region max filesize set to 1gb in my
> hbase-site.xml like so:
> >
> > <property>
> > <name>hbase.hregion.max.filesize</name>
> > <value>1073741824</value>
> > </property>
> >
> > But when I look at the storefileSizeMB values listed on the regionserver
> UI, they're all around 256mb, which makes me think the default value is
> still being used for whatever reason. I'm also setting other properties
> like the handler count which are getting used so I'm pretty sure I have the
> conf in the right place and it's getting read. I've also tried restarting
> hbase and doing a major compact on the table but the region sizes stay the
> same. Any ideas on what might be wrong? Is there some way to verify the
> max filesize is getting set correctly?
> >
> > Thanks,
> > Albert
> >
>
--
Thanks & Best regards
jiajun
Re: region max filesize
Posted by Ravi Veeramachaneni <ra...@gmail.com>.
Albert,
You doing partially right. The change will take into effect only to new tables created going forward. There are two options I'm aware of, 1. Use alter command on HBase shell to specify new region size. 2. Create new set of tables and create simple m/r job to migrate data to new tables and drop the old ones.
We followed the second approach when we bump the region size. The first approach leaves the current region count But fills up until region gets to 1GB. Our goal was to reduce number of regions to minimum, second approach worked well for us.
Thanks,
Ravi
Sent from my mobile phone.
On Jul 11, 2011, at 8:56 PM, Albert Shau <as...@yahoo-inc.com> wrote:
> Hi everyone,
>
> I'm using 0.90.3 and have the region max filesize set to 1gb in my hbase-site.xml like so:
>
> <property>
> <name>hbase.hregion.max.filesize</name>
> <value>1073741824</value>
> </property>
>
> But when I look at the storefileSizeMB values listed on the regionserver UI, they're all around 256mb, which makes me think the default value is still being used for whatever reason. I'm also setting other properties like the handler count which are getting used so I'm pretty sure I have the conf in the right place and it's getting read. I've also tried restarting hbase and doing a major compact on the table but the region sizes stay the same. Any ideas on what might be wrong? Is there some way to verify the max filesize is getting set correctly?
>
> Thanks,
> Albert
>