You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Nick Dimiduk <nd...@gmail.com> on 2017/03/22 18:28:39 UTC
Re: Why hbase doesn't remove the empty region
Sorry for the long delay. I've linked to our script over on HBASE-15712.
Hope folks find it useful.
On Sat, Nov 19, 2016 at 10:54 AM, Xi Yang <al...@gmail.com> wrote:
> Thanks Nick & Ted. Thank you for your help and your patience.
>
> "Tool for retiring empty regions" will be an useful feature!
>
> 2016-11-18 17:18 GMT-08:00 Nick Dimiduk <nd...@gmail.com>:
>
> > HBase will not remove empty regions. It assumes you know what you're
> doing.
> >
> > In 1.2 there's a new "Region Normalizer" [0] feature that runs in the
> > Master, which acts a bit like the Balancer, but for region sizes. I think
> > it's still considered experimental, so YMMV. We're still on 1.1, so I
> > haven't tried it in prod yet.
> >
> > I also have this problem in our cluster, because of a combination of TTL
> > and Phoenix's Rowkey Timestamp feature. This results in region boundaries
> > including the timestamp, and data that goes away after a period.
> Eventually
> > all regions will become empty. I have a script that runs periodically to
> > calculate region sizes and drop empty regions. I volunteered to
> contribute
> > it [1], but haven't pulled it out of our puppet infra in the form of a
> > patch. Let me revive that thread internally.
> >
> > -n
> >
> > [0]: https://issues.apache.org/jira/browse/HBASE-13103
> > [1]: https://issues.apache.org/jira/browse/HBASE-15712
> >
> > On Fri, Nov 18, 2016 at 4:59 PM, Xi Yang <al...@gmail.com> wrote:
> >
> > > Yes, we're still on old release of 0.98.x
> > >
> > > Many of empty region didn't be deleted. Take one for example:
> > >
> > >
> > > > "Namespace_default_table_java_app_logs_region_
> > > 29d4fa5482d17daf813e12732155e0ee_metric_storeCount"
> > > > : 1,
> > > >
> > > > "Namespace_default_table_java_app_logs_region_
> > > 29d4fa5482d17daf813e12732155e0ee_metric_storeFileCount"
> > > > : 1,
> > > >
> > > > "Namespace_default_table_java_app_logs_region_
> > > 29d4fa5482d17daf813e12732155e0ee_metric_memStoreSize"
> > > > : 424,
> > > >
> > > > "Namespace_default_table_java_app_logs_region_
> > > 29d4fa5482d17daf813e12732155e0ee_metric_storeFileSize"
> > > > : 549,
> > > >
> > > > "Namespace_default_table_java_app_logs_region_
> > > 29d4fa5482d17daf813e12732155e0ee_metric_compactionsCompletedCount"
> > > > : 4,
> > > >
> > > > "Namespace_default_table_java_app_logs_region_
> > > 29d4fa5482d17daf813e12732155e0ee_metric_numBytesCompactedCount"
> > > > : 8616506268,
> > > >
> > > > "Namespace_default_table_java_app_logs_region_
> > > 29d4fa5482d17daf813e12732155e0ee_metric_numFilesCompactedCount"
> > > > : 4,
> > >
> > >
> > > The log about last time major compaction of this region is:
> > >
> > > 2016-11-03 22:45:33,453 INFO org.apache.hadoop.hbase.
> > regionserver.HStore:
> > > > Completed major compaction of 1 (all) file(s) in log of
> > > > java_app_logs,CM-sjcmhpcapp01-1476076988799-135168375-
> > > 178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449.
> > > 29d4fa5482d17daf813e12732155e0ee.
> > > > into 47b8ee796fc74ff0b82222bf60318f99(size=549), total size for
> store
> > is
> > > > 549. This selection was in queue for 0sec, and took 0sec to execute.
> > > > 2016-11-03 22:45:33,454 INFO
> > > > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
> > > > compaction: Request =
> > > > regionName=java_app_logs,CM-sjcmhpcapp01-1476076988799-
> > > 135168375-178aebc1-4bfc-47c4-907f-952f50270f50-b,1476360804449.
> > > 29d4fa5482d17daf813e12732155e0ee.,
> > > > storeName=log, fileCount=1, fileSize=1.3 G, priority=9,
> > > > time=25652433393376299; duration=0sec
> > >
> > >
> > > You can find that its size is only 549 bytes at that time. But HBase
> > didn't
> > > remove it.
> > >
> > > Here is the result I run HFile tool
> > >
> > > $ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f
> > > > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/
> > > 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99
> > > > 16/11/18 16:48:25 INFO Configuration.deprecation: hadoop.native.lib
> is
> > > > deprecated. Instead, use io.native.lib.available
> > > > 16/11/18 16:48:29 INFO Configuration.deprecation: fs.default.name is
> > > > deprecated. Instead, use fs.defaultFS
> > > > Scanning ->
> > > > hdfs://xxxxxxx:8020/hbase/data/default/java_app_logs/
> > > 29d4fa5482d17daf813e12732155e0ee/log/47b8ee796fc74ff0b82222bf60318f99
> > > > 16/11/18 16:48:29 INFO hfile.CacheConfig: CacheConfig:disabled
> > > > Scanned kv count -> 0
> > >
> > >
> > >
> > > 2016-11-17 18:45 GMT-08:00 Ted Yu <yu...@gmail.com>:
> > >
> > > > You would be able to find in region server log(s).
> > > >
> > > > Can you do quick inspection of the hfiles using:
> > > >
> > > > http://hbase.apache.org/book.html#hfile_tool
> > > >
> > > > BTW you're still on old release of 0.98.x, right ?
> > > >
> > > > On Thu, Nov 17, 2016 at 6:40 PM, Xi Yang <al...@gmail.com>
> > wrote:
> > > >
> > > > > Thank you for reply.
> > > > >
> > > > > Yes. There are many regions have this problem. And those regions
> were
> > > > > created several years ago.
> > > > > hbase.hregion.majorcompaction of our cluster is 7 days.
> > > > > Where can I see the last major compaction time?
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > > 2016-11-17 18:22 GMT-08:00 Ted Yu <yu...@gmail.com>:
> > > > >
> > > > > > When was the last time major compaction was performed on this
> > region
> > > ?
> > > > > >
> > > > > > Were you referring to the store files by 'them' in your question
> ?
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Thu, Nov 17, 2016 at 6:18 PM, Xi Yang <alex.xi.yang@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > I have some regions are empty. But HBase doesn't remove them,
> > why?
> > > > > > >
> > > > > > > You can see the storeFileSize is only 549. But I use hbase
> shell
> > to
> > > > > scan
> > > > > > > this region, the result is empty
> > > > > > >
> > > > > > > "Namespace_default_table_java_app_logs_region_
> > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileCount"
> > > > > > > : 1,
> > > > > > > "Namespace_default_table_java_app_logs_region_
> > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_memStoreSize"
> > > > > > > : 424,
> > > > > > > "Namespace_default_table_java_app_logs_region_
> > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_storeFileSize"
> > > > > > > : 549,
> > > > > > > "Namespace_default_table_java_app_logs_region_
> > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_
> > compactionsCompletedCount"
> > > > > > > : 4,
> > > > > > > "Namespace_default_table_java_app_logs_region_
> > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_
> numBytesCompactedCount"
> > > > > > > : 8503271111,
> > > > > > > "Namespace_default_table_java_app_logs_region_
> > > > > > > 3854912733f1acaaa8a255abd6b7b1ec_metric_
> numFilesCompactedCount"
> > > > > > > : 4,
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>