You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "jonasjc (Jira)" <ji...@apache.org> on 2025/01/24 06:08:00 UTC
[jira] [Updated] (FLINK-37212) When hbase region move, throw "Unable to load exception received from server: XXX"
[ https://issues.apache.org/jira/browse/FLINK-37212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
jonasjc updated FLINK-37212:
----------------------------
Summary: When hbase region move, throw "Unable to load exception received from server: XXX" (was: Hbase connector "Unable to load exception received from server: XXX")
> When hbase region move, throw "Unable to load exception received from server: XXX"
> -----------------------------------------------------------------------------------
>
> Key: FLINK-37212
> URL: https://issues.apache.org/jira/browse/FLINK-37212
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
> Affects Versions: hbase-4.0.0
> Reporter: jonasjc
> Priority: Major
>
> *Problem:*
> when hbase region move,hbase connector throw the following exception:
> {code:java}
> Caused by: org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.DoNotRetryIOException: Unable to load exception received from server:org.apache.hadoop.hbase.NotServingRegionException at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:85) at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:282) at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:269) at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:129) at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107) at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.HTable.get(HTable.java:384) at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.HTable.get(HTable.java:358) at org.apache.flink.connector.hbase.source.HBaseRowDataLookupFunction.lookup(HBaseRowDataLookupFunction.java:98) ... 18 moreCaused by: org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException): org.apache.hadoop.hbase.NotServingRegionException: DIM:jichun_test_dim,,1736922383279.ebbf1db8150deec82662ff4087dbcc67. is not online on xxx at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3462) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3439) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1488) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2561) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45815) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:392) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:359) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:339)
> {code}
> and
> {code:java}
> 2025-01-24 11:11:41,930 [Legacy Source Thread - Source: TableSourceScan(table=[[default_catalog, default_database, datagen_test]], fields=[c1]) -> LookupJoin(table=[default_catalog.default_database.dim_kefu_ticket_detail], joinType=[LeftOuterJoin], lookup=[rowkey=c1], select=[c1, rowkey]) -> Sink: Sink(table=[default_catalog.default_database.sink_test], fields=[c1, rowkey]) (1/1)#0] [] ERROR org.apache.flink.connector.hbase.source.HBaseRowDataLookupFunction(105) - HBase lookup error, retry times = 0org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.DoNotRetryIOException: Unable to load exception received from server:org.apache.hadoop.hbase.exceptions.RegionMovedException at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:85) ~[blob_p-05917e94670b1c6e110fce2c94b45405cea035d1-9465ef6bc8090a3af4d68959bb598539:4.0-SNAPSHOT] at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:282) ~[blob_p-05917e94670b1c6e110fce2c94b45405cea035d1-9465ef6bc8090a3af4d68959bb598539:4.0-SNAPSHOT] at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:269) ~[blob_p-05917e94670b1c6e110fce2c94b45405cea035d1-9465ef6bc8090a3af4d68959bb598539:4.0-SNAPSHOT] at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:129) ~[blob_p-05917e94670b1c6e110fce2c94b45405cea035d1-9465ef6bc8090a3af4d68959bb598539:4.0-SNAPSHOT] at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107) ~[blob_p-05917e94670b1c6e110fce2c94b45405cea035d1-9465ef6bc8090a3af4d68959bb598539:4.0-SNAPSHOT] at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.HTable.get(HTable.java:384) ~[blob_p-05917e94670b1c6e110fce2c94b45405cea035d1-9465ef6bc8090a3af4d68959bb598539:4.0-SNAPSHOT] at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.HTable.get(HTable.java:358) ~[blob_p-05917e94670b1c6e110fce2c94b45405cea035d1-9465ef6bc8090a3af4d68959bb598539:4.0-SNAPSHOT] at org.apache.flink.connector.hbase.source.HBaseRowDataLookupFunction.lookup(HBaseRowDataLookupFunction.java:98) [blob_p-05917e94670b1c6e110fce2c94b45405cea035d1-9465ef6bc8090a3af4d68959bb598539:4.0-SNAPSHOT] at org.apache.flink.table.functions.LookupFunction.eval(LookupFunction.java:52) [flink-table-api-java-uber-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at LookupFunction$6.flatMap(Unknown Source) [flink-table-runtime-du-1.16-SNAPSHOT.jar:?] at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.doFetch(LookupJoinRunner.java:92) [flink-table-runtime-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:79) [flink-table-runtime-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.table.runtime.operators.join.lookup.LookupJoinRunner.processElement(LookupJoinRunner.java:34) [flink-table-runtime-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:99) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:80) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:39) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:418) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:513) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.api.operators.StreamSourceContexts$SwitchingOnClose.collect(StreamSourceContexts.java:103) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource.run(DataGeneratorSource.java:117) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT] at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:333) [flink-dist-du-1.16-SNAPSHOT.jar:du-1.16-SNAPSHOT]Caused by: org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: org.apache.hadoop.hbase.exceptions.RegionMovedException: Region moved to: hostname=xxx port=16020 startCode=1736822871845. As of locationSeqNum=54. at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3453) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3439) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1488) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2561) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45815) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:392) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:359) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:339)
> {code}
> *Cause:*
> it cause by : 'org.apache.hadoop.hbase.NotServingRegionException' had been relocate to
> 'org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.NotServingRegionException' and can not fond the class 'org.apache.hadoop.hbase.NotServingRegionException' from error message fo hbase sever. and the same as 'org.apache.hadoop.hbase.exceptions.RegionMovedException'
>
> *Fix:*
> don't relocate class 'org.apache.hadoop.hbase.NotServingRegionException' and 'org.apache.hadoop.hbase.exceptions.RegionMovedException'
--
This message was sent by Atlassian Jira
(v8.20.10#820010)