You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Li Zhexi (Jira)" <ji...@apache.org> on 2026/06/17 04:05:00 UTC
[jira] [Updated] (HBASE-30231) LoadIncrementalHFiles failed for MetricsException: Metrics source RegionServer,Sub=IO already exists
[ https://issues.apache.org/jira/browse/HBASE-30231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Zhexi updated HBASE-30231:
-----------------------------
Description:
when called
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles#doBulkLoad(Path hfofDir, final Admin admin, Table table, RegionLocator regionLocator) to load HFiles, with hadoop jar command.
Showed the Error message:
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsConfig - Loaded properties from hadoop-metrics2.properties
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - HBase metrics system started
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.hbase.metrics.MetricRegistries - Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
ERROR tool.LoadIncrementalHFiles: Unexpected execution exception during splitting
java.util.concurrent.ExecutionException: org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub=IO already exists!
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:662)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:454)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:371)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:285)
at com.xx.xx.XXLoadFile.main(XXLoadFile.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:332)
at org.apache.hadoop.util.RunJar.main(RunJar.java:247)
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub-IO already exists!
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
at org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:44)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:39)
at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:99)
at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:36)
at org.apache.hadoop.hbase.io.MetricsIO.getInstance(MetricsIO.java:52)
at org.apache.hadoop.hbase.io.hfile.HFile.updateWriteLatency(HFile.java:204)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.finishBlockAndWriteHeaderAndData(HFileBlock.java:1047)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFileBlock.java:1032)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:1069)
at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:646)
at org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.close(StoreFileWriter.java:786)
at org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.access$1300(StoreFileWriter.java:493)
at org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:293)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.copyHFileHalf(LoadIncrementalHFiles.java:1209)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:1157)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:702)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:825)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:652)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:648)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
The probability of this issue occurring is very low, it only happened once, and it succeeded after a retry. It cannot be reproduced.
Seems HFile.updateWriteLatency 100% initilize MetricsIO, does it necessary for client work?
was:
when called
org.apache.hadoop.hbase.tool.LoadIncrementalHFiles#doBulkLoad(Path hfofDir, final Admin admin, Table table, RegionLocator regionLocator) to load HFiles, with hadoop jar command.
Showed the Error message:
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.snappy]
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsConfig - Loaded properties from hadoop-metrics2.properties
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - HBase metrics system started
[LoadIncrementalHFiles-19] INFO org.apache.hadoop.hbase.metrics.MetricRegistries - Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
ERROR tool.LoadIncrementalHFiles: Unexpected execution exception during splitting
java.util.concurrent.ExecutionException: org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub=IO already exists!
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:662)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:454)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:371)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:285)
at com.xx.xx.XXLoadFile.main(XXLoadFile.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:332)
at org.apache.hadoop.util.RunJar.main(RunJar.java:247)
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub-IO already exists!
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
at org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:44)
at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:39)
at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:99)
at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:36)
at org.apache.hadoop.hbase.io.MetricsIO.getInstance(MetricsIO.java:52)
at org.apache.hadoop.hbase.io.hfile.HFile.updateWriteLatency(HFile.java:204)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.finishBlockAndWriteHeaderAndData(HFileBlock.java:1047)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFileBlock.java:1032)
at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:1069)
at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:646)
at org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.close(StoreFileWriter.java:786)
at org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.access$1300(StoreFileWriter.java:493)
at org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:293)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.copyHFileHalf(LoadIncrementalHFiles.java:1209)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:1157)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:702)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:825)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:652)
at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:648)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
The probability of this issue occurring is very low, it only happened once, and it succeeded after a retry. It cannot be reproduced.
> LoadIncrementalHFiles failed for MetricsException: Metrics source RegionServer,Sub=IO already exists
> ----------------------------------------------------------------------------------------------------
>
> Key: HBASE-30231
> URL: https://issues.apache.org/jira/browse/HBASE-30231
> Project: HBase
> Issue Type: Bug
> Components: metrics
> Affects Versions: 2.6.1
> Reporter: Li Zhexi
> Assignee: Li Zhexi
> Priority: Major
>
> when called
> org.apache.hadoop.hbase.tool.LoadIncrementalHFiles#doBulkLoad(Path hfofDir, final Admin admin, Table table, RegionLocator regionLocator) to load HFiles, with hadoop jar command.
> Showed the Error message:
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.snappy]
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.snappy]
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsConfig - Loaded properties from hadoop-metrics2.properties
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - HBase metrics system started
> [LoadIncrementalHFiles-19] INFO org.apache.hadoop.hbase.metrics.MetricRegistries - Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> ERROR tool.LoadIncrementalHFiles: Unexpected execution exception during splitting
> java.util.concurrent.ExecutionException: org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub=IO already exists!
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:662)
> at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:454)
> at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:371)
> at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:285)
> at com.xx.xx.XXLoadFile.main(XXLoadFile.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:332)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:247)
> Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source RegionServer,sub-IO already exists!
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
> at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:44)
> at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(IOSourceImpl.java:39)
> at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:99)
> at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:36)
> at org.apache.hadoop.hbase.io.MetricsIO.getInstance(MetricsIO.java:52)
> at org.apache.hadoop.hbase.io.hfile.HFile.updateWriteLatency(HFile.java:204)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.finishBlockAndWriteHeaderAndData(HFileBlock.java:1047)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeHeaderAndData(HFileBlock.java:1032)
> at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:1069)
> at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:646)
> at org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.close(StoreFileWriter.java:786)
> at org.apache.hadoop.hbase.regionserver.StoreFileWriter$SingleStoreFileWriter.access$1300(StoreFileWriter.java:493)
> at org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:293)
> at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.copyHFileHalf(LoadIncrementalHFiles.java:1209)
> at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:1157)
> at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:702)
> at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:825)
> at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:652)
> at org.apache.hadoop.hbase.tools.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:648)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
>
> The probability of this issue occurring is very low, it only happened once, and it succeeded after a retry. It cannot be reproduced.
> Seems HFile.updateWriteLatency 100% initilize MetricsIO, does it necessary for client work?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)